Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewritten Var/values implementation based around flexbuffers #371

Merged
merged 1 commit into from
Aug 30, 2024
Merged

Conversation

rdaum
Copy link
Owner

@rdaum rdaum commented Aug 30, 2024

This basically rewrites most of the values layer to be based around values and collections held in flexbuffers. (Flexbuffers are part of the flatbuffers project, and are a kind of zero copy dynamic flatbuffer.)

Part of a bigger effort to do the following:

  • Rip out custom serialization formats and replace with a combination of flatbuffers and flexbuffers
  • Same for all uses of bincode2
  • Clean up the Var interface more broadly

Along with this change came major cleanups generally to the interface in the values layer:

  • Make all index-based operations take an argument stating whether they should operate on one-indexed (like MOO) or zero-indexed (like most languages). This cleans up the ad-hoc way the conversion was happening before, and makes it easier to use languages other than MOO, but using the same values layer.
  • Move printing of the literal representation of a value out into the compiler crate, making it clear that the literal printing is of a MOO value, and leaving room for other languages to have other representations, or to add different pretty printing / decompilation options.
  • Makes the type system interface more regular in general, providing traits to identify operations for scalar/associative/sequence classes of type.

To be done still:

  • Rip out use of bincode2 and custom serialization code and replace with schema'd flatbuffer entities.
  • There are definitely lots of excessive copies still happening, and there's likely much more clever things that can be happening inside Var, including switching between a 'building' and 'reading' mode such that a given value doesn't obtain its flexbuffered pickled form until first written to the database. Or something.

This basically rewrites most of the values layer to be based around
values and collections held in flexbuffers. (Flexbuffers are part of
the flatbuffers project, and are a kind of zero copy dynamic
flatbuffer.)

Part of a bigger effort to do the following:

  * Rip out custom serialization formats and replace with a
    combination of flatbuffers and flexbuffers
  * Same for all uses of bincode2
  * Clean up the Var interface more broadly

Along with this change came major cleanups generally to the interface
in the values layer:

  * Make all index-based operations take an argument stating whether
    they should operate on one-indexed (like MOO) or
    zero-indexed (like most languages). This cleans up the ad-hoc way
    the conversion was happening before, and makes it easier to use
    languages other than MOO, but using the same values layer.
  * Move printing of the literal representation of a value out into
    the `compiler` crate, making it clear that the literal printing is
    of a MOO value, and leaving room for other languages to have other
    representations, or to add different pretty printing /
    decompilation options.
  * Makes the type system interface more regular in general, providing
    traits to identify operations for scalar/associative/sequence
    classes of type.

To be done still:

  * Rip out use of bincode2 and custom serialization code and replace
    with schema'd flatbuffer entities.
  * There are definitely lots of excessive copies still happening, and
    there's likely much more clever things that can be happening
    inside Var, including switching between a 'building' and 'reading'
    mode such that a given value doesn't obtain its flexbuffered
    pickled form until first written to the database. Or something.
@rdaum rdaum merged commit 5bf5d90 into main Aug 30, 2024
11 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant