Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make impl_writeable_tlv_based_enum* actually upgradable #3160

Conversation

TheBlueMatt
Copy link
Collaborator

In cc78b77 it was discovered that impl_writeable_tlv_based_enum_upgradable wasn't actually upgradable - tuple variants weren't written with length-prefixes, causing downgrades with new tuple variants to be unreadable by older clients as they wouldn't know where to stop reading.

This was fixed by simply assuming that any new variants will be non-tuple variants with a length prefix, but no code write-side changes were made, allowing new code to freely continue to use the broken tuple-variant serialization.

Here we address this be defining yet more serialization macros which aren't broken, and convert existing usage of the existing macros using non-length-prefixed tuple variants to renamed *_legacy macros.

Note that this changes the serialization format of impl_writeable_tlv_based_enum[_upgradable] when tuple fields are written, and as such deliberately changes the call semantics for such tuples.

Given this is a public/exported macro, we may wish to not do this and instead give it a new name to ensure users are forced to recon with the changes rather than just seeing a compilation failure due to new semantics and fixing those (without reading the release notes I guess?). Not sure if its worth it.

@codecov-commenter
Copy link

codecov-commenter commented Jul 8, 2024

Codecov Report

Attention: Patch coverage is 72.22222% with 20 lines in your changes missing coverage. Please review.

Project coverage is 89.76%. Comparing base (6035c83) to head (1d1f47c).
Report is 57 commits behind head on main.

Files Patch % Lines
lightning/src/util/ser_macros.rs 73.23% 8 Missing and 11 partials ⚠️
lightning/src/ln/channelmanager.rs 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3160      +/-   ##
==========================================
+ Coverage   89.74%   89.76%   +0.01%     
==========================================
  Files         121      121              
  Lines       99858   100730     +872     
  Branches    99858   100730     +872     
==========================================
+ Hits        89622    90421     +799     
- Misses       7561     7618      +57     
- Partials     2675     2691      +16     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@TheBlueMatt TheBlueMatt force-pushed the 2024-07-better-enum-upgradable-ser branch from 7fb8f80 to 008a291 Compare July 10, 2024 18:56
@TheBlueMatt
Copy link
Collaborator Author

Updated after the merge of #3085 to change the format of MessageContext.

@arik-so
Copy link
Contributor

arik-so commented Jul 16, 2024

Is there anywhere a canonical breakdown of when to use which of:

  • impl_writeable_tlv_based_enum
  • impl_writeable_tlv_based_enum_legacy
  • impl_writeable_tlv_based_enum_upgradable
  • impl_writeable_tlv_based_enum_upgradable_legacy

I wonder if it might be useful as the method names become chaotic.

@TheBlueMatt
Copy link
Collaborator Author

TheBlueMatt commented Jul 16, 2024

Is there anywhere a canonical breakdown of when to use which of:

Don't ever write _legacy for anything new, anything already using it (assuming it as something after the ;, ie it serializes tuple variants) has to stay. _legacy seems to imply that pretty heavily to me.

The upgradable-vs-not thing is a question of what you want to read - if you do upgradable, you get MaybeReadable, not-upgradable you get Readable. This is pretty clear when you try to use it cause the compiler will scream :)

Copy link
Contributor

@valentinewallace valentinewallace left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM I think

($st: ident, $(($variant_id: expr, $variant_name: ident) =>
{$(($type: expr, $field: ident, $fieldty: tt)),* $(,)*}
),* $(,)*;
$(($tuple_variant_id: expr, $tuple_variant_name: ident)),* $(,)*) => {
$(($tuple_variant_id: expr, $tuple_variant_name: ident)),+ $(,)?) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To check my understanding, this change ensures that we don't use this legacy macro unless there is a non-empty tuple variant present?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, the + here mandates that you actually use a tuple variant, cause if you're not you shouldn't be using legacy. The * -> ? change is just an MSRV cleanup.

lightning/src/util/ser_macros.rs Show resolved Hide resolved
lightning/src/util/ser_macros.rs Outdated Show resolved Hide resolved
@TheBlueMatt TheBlueMatt force-pushed the 2024-07-better-enum-upgradable-ser branch from 008a291 to 7fd9bfb Compare July 16, 2024 20:27
@arik-so
Copy link
Contributor

arik-so commented Jul 16, 2024

Is there anywhere a canonical breakdown of when to use which of:

Don't ever write _legacy for anything new, anything already using it (assuming it as something after the ;, ie it serializes tuple variants) has to stay. _legacy seems to imply that pretty heavily to me.

The upgradable-vs-not thing is a question of what you want to read - if you do upgradable, you get MaybeReadable, not-upgradable you get Readable. This is pretty clear when you try to use it cause the compiler will scream :)

Right, I meant in the code or the docs, as opposed to in the PR :)

@TheBlueMatt
Copy link
Collaborator Author

Yea, I figured "legacy" for serialization stuff was relatively clear :).

@valentinewallace
Copy link
Contributor

Feel free to squash!

In cc78b77 it was discovered that
`impl_writeable_tlv_based_enum_upgradable` wasn't actually
upgradable - tuple variants weren't written with length-prefixes,
causing downgrades with new tuple variants to be unreadable by
older clients as they wouldn't know where to stop reading.

This was fixed by simply assuming that any new variants will be
non-tuple variants with a length prefix, but no code write-side
changes were made, allowing new code to freely continue to use the
broken tuple-variant serialization.

Here we address this be defining yet more serialization macros
which aren't broken, and convert existing usage of the existing
macros using non-length-prefixed tuple variants to renamed
`*_legacy` macros.

Note that this changes the serialization format of
`impl_writeable_tlv_based_enum[_upgradable]` when tuple fields are
written, and as such deliberately changes the call semantics for
such tuples.

Only the serialization format of `MessageContext` is changed here
which is fine as it has not yet reached a release of LDK.
@TheBlueMatt
Copy link
Collaborator Author

Squashed without change.

@TheBlueMatt TheBlueMatt force-pushed the 2024-07-better-enum-upgradable-ser branch from 7fd9bfb to 1d1f47c Compare July 17, 2024 15:02
@TheBlueMatt TheBlueMatt merged commit 0cfe55c into lightningdevkit:main Jul 17, 2024
17 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants