CIP-???? | Efficient Proofs for Dynamic Sets

[ilap]

📄 Rendered proposal

[Crypto2099]

Overall a very, very interesting proposal in my mind and would love to see commentary from @Quantumplation @perturbing and any other subject matter experts.

Some outstanding questions for me before really proceeding forward would be:

1. How efficient (or expensive) is the proof in terms of Plutus/Aiken integration, particularly when we run into the tens of thousands of records (i.e. what is the theoretical maximum for a given tree size based on current network parameters)
2. How would the "off chain" portion of this work? Would it be a matter of indexing a special address that exists to keep the datum and combining the sequence of tree hashes w/ the minting or burning of tokens to recreate the entire tree?
3. Assuming yes, that the off-chain requires indexing and recording these data points, would it make sense to create a complimentary standard for "snapshot" backups of this data that could be stored similar to Mithril snapshots (IPFS or some other decentralized storage) that could contain the entire tree to shortcut the indexing the chain process?

[rphair]

@ilap interesting & looks very useful. Looking forward to introduction & initial reactions at next CIP meeting; I've added it to the agenda & you are welcome to attend the meeting: https://hackmd.io/@cip-editors/89

[ilap]

Hi @Crypto2099

1. How efficient (or expensive) is the proof in terms of Plutus/Aiken integration, particularly when we run into the tens of thousands of records (i.e. what is the theoretical maximum for a given tree size based on current network parameters)

With the current mainnet parameters it (the aiken implementation) could theoretically handle around 2^28 (~268M elements in the tree, almost 2^32 i.e. 4B elements) large tree , assuming max. 4KB redeemer size for storing proof, see append check test result below.

Note: the PlutusTx implementation was very inefficient, that's why I switched to aiken around end of Oct, begin of Nov.

    ┍━ ilap/integri_tree/tests/test_integri_tree ━━━━━━━━━━━━━━━━━━
    │ PASS [mem:    713128, cpu:   308048621] test_append_check_2_1
    │ PASS [mem:   3772914, cpu:  1700016553] test_append_check_2_10
    │ PASS [mem:   5877078, cpu:  2671638937] test_append_check_2_16
    │ PASS [mem:  10085406, cpu:  4614883705] test_append_check_2_28
    │ PASS [mem:  11488182, cpu:  5262631961] test_append_check_2_32
    │ PASS [mem:  23148230, cpu: 10739828797] test_append_check_2_64
    │ PASS [mem:    223141, cpu:    96424038] test_initial_added_state
    │ PASS [mem:      2203, cpu:     1871668] bytearray_compare_check
    ┕━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8 tests | 8 passed | 0 failed

2. How would the "off chain" portion of this work? Would it be a matter of indexing a special address that exists to keep the datum and combining the sequence of tree hashes w/ the minting or burning of tokens to recreate the entire tree?

Yes. Though, if the burning is not implemented then a normal Ogmios/Kupo like indexing would work for reconstructing the tree from on chain data by querying the chain using similar URL https://<kupo.url>/matches/<POLICYID>.*?order=oldest_first&created_after=<BOOTSTRAPSLOT>. What the adahandle like PoC implementation uses currently (no burning is implemented yet).

Otherwise, some sophisticated queries from some providers might be required for tracking the mint/burn sequences for reconstructing the tree.
As an example, as you assumed, tracking all transactions of the Authorisation NFT (state changes) and gathering the element added/removed from the datums (states).

3. Assuming yes, that the off-chain requires indexing and recording these data points, would it make sense to create a complimentary standard for "snapshot" backups of this data that could be stored similar to Mithril snapshots (IPFS or some other decentralized storage) that could contain the entire tree to shortcut the indexing the chain process?

The protocol does not assume any honest party, therefore any 3rd-party implementation of some service that offer to download the tree structure, or a backend that constantly updates its tree, or just offers proofs (as the validations are deterministic) for an element add/remove would do.
I do not think that any snapshot is required, but it could be handy for providers offer some similar services or when the dataset/tree has grown to some very large size.

But, TBH, I checked for rebuilding the tree for some millions of elements and it was quite quick (in minutes or max an hour, I cannot recall really), though I did not do any benchmark on these scenarios due to the limited time I had for working on this.

[perturbing]

Hi @ilap, nice work on this hash-based dynamic universal accumulator!

I can see you thought this through thoroughly. Did you have any references on the security properties of this scheme? Was this design taken from a paper, or is it original? If the former, can you reference it? If the latter, have you proven that this scheme is complete and sound?

Additionally, I wonder if your contribution needs a CIP? As you specify in your motivation:

This CIP aims to address this challenge by providing a solution for efficiently and securely tracking the dynamic minting and burning of NFTs on the Cardano blockchain.

This provides a very specific solution to a very specific problem. @rphair, is this within the scope of a CIP? I feel it would be a CIP if it included a general method for any project to prove their adherence to this design via a Plutus script (requiring cordination between community members). Your CIP does not describe this—is this something you had in mind?

Also, there are more ways of doing this. For example, see a pairing-based one here which allows for constant proof size and aggregation of (non)-membership proofs, though it is more CPU intensive. What would justify this particular design over the other possible designs?

[perturbing]

Can you explain what trusted means here?

[ilap]

Yeah, it's ambiguous and misleading. It simply means, that the deployed Plutus script is secure in the sense that no any adversarial can modify/change it after deployment, but it does not mean that Plutus script itself is secure (it requires some audit etc.) in any way.

[perturbing]

Ah, is this the script that everyone will use? In that, you aim to create a universal script that others can reference, so that you can onchain proof that projects adhere to this standard.

[perturbing]

Your solution is more aimed at the unconstrained case, the above is not true in the constrained case where the assets are completely specified beforehand. As an example, the Matrix berry drop used a merkle tree to dynamically mint and prove that the correct asset was minted and ensured that only one could be minted.

So I think this section or some other should specify more that the problem you are addressing is when assets are not completely know yet before deployment (correct me if my assessment is wrong).

A suggestion, perhaps replace dynamic with unconstrained as the dynamic but constrained case knows a less complex solution?

[ilap]

Yes, it does not assume any knowledge

So I think this section or some other should specify more that the problem you are addressing is when assets are not completely know yet before deployment (correct me if my assessment is wrong).

Yes, you're correct, this solution does not assume any knowledge of the mint-able assets.

[perturbing]

This is again not enough, you are really focussing on the unconstrained case.

The simpler merkle tree example that matrix berry used also achieves this goal in its current form, so I like to ask you, what does your solution add to that?

[ilap]

Yes, your assumption is correct. Though, that's why the title contains dynamic set, though the name can be misleading/confusing, as there is no proper definition what I meant by dynamic set (dynamically add/remove elements without any constraints).

[ilap]

I can see you thought this through thoroughly. Did you have any references on the security properties of this scheme? Was this design taken from a paper, or is it original? If the former, can you reference it? If the latter, have you proven that this scheme is complete and sound?

Yes, this design implemented was based on some paper but it has very different assumptions to achieve soundness/completeness.
So, this design is not proven to be complete and sound.

Also, there are more ways of doing this. For example, see a pairing-based one here which allows for constant proof size and aggregation of (non)-membership proofs, though it is more CPU intensive. What would justify this particular design over the other possible designs?

Yes, I was thinking of these other solutions too and might be - in the near future - the open-interval type of constraints could be used for simplifying the aggregations of large sets (using BLS).

I was trying to build a solution that can be used relative cheap and be able to implement.

[rphair]

@perturbing #816 (review): This provides a very specific solution to a very specific problem. @rphair, is this within the scope of a CIP? I feel it would be a CIP if it included a general method...

If the submitted content only describes an individual approach then generally it's not considerable as a CIP. But so far, several times in the editorial process, when such a method has looked sound then we've tried in the course of community review to broaden the proposal so it might be applicable to other implementors & applications.

So we'll carry it to the CIP meeting to see if the variety of attendees can either confirm a broader applicability or suggest how it can be broadened if necessary.

Also we have quite a few unimplemented CIPs which I believe still qualify as "improvement" proposals for Cardano itself (not just "design ideas"), even if only in an educational sense, and @ilap I believe this could be one of them even if that generality cannot be established.

@Crypto2099 has experience leagues beyond my own in matters of token minting & distribution so editorially he is the best person to settle the generality question & hopefully he can give a review soon with that in mind, and in any case we'll focus on that question in Tuesday's meeting. cc @Ryun1

[Ryun1]

Suggested change

	## Minimal Subtree (Proof)
	### Minimal Subtree (Proof)

[rphair]

@ilap at the just-finished CIP meeting we were attempting to qualify this is a CIP rather than a "best practices" document that could exist somewhere else (a private repo, a blog article, the Developer Portal, ...).

Mainly this was a confirmation of #816 (comment) but also suggesting the potential problem that Cardano implementors might read this as a "standard" document and assume that this is the "preferred way" of handling the problem: maybe even adjusting their implementations to make the CIP applicable.

It was suggested that this would be less likely to happen if (and/or):

• the CIP were made even more clear that this is a narrow use case, with a specific means of addressing it;
• a "container" CPS were drafted for dynamic sets — to include this CIP as a solution — and each "solving" CIP indicating its own strengths & weaknesses compared to other approaches.

Also I'm removing the term on Cardano from the title since this would generally be redundant for a CIP or CPS.

cc @perturbing @rjharmon @Ryun1 @Crypto2099

[ilap]

It was suggested that this would be less likely to happen if (and/or):

• the CIP were made even more clear that this is a narrow use case, with a specific means of addressing it;
• a "container" CPS were drafted for dynamic sets — to include this CIP as a solution — and each "solving" CIP indicating its own strengths & weaknesses compared to other approaches.

@rphair, thanks for that. I would say the CPS would be a better option, as there could be more than one solution to solve the problem. In fact, I have another proposal that is based on BLS12-381.

But, TBH I am not familiar with CIP/CPS, so I will follow your and others suggestions.

[Crypto2099]

It was suggested that this would be less likely to happen if (and/or):

• the CIP were made even more clear that this is a narrow use case, with a specific means of addressing it;
• a "container" CPS were drafted for dynamic sets — to include this CIP as a solution — and each "solving" CIP indicating its own strengths & weaknesses compared to other approaches.

@rphair, thanks for that. I would say the CPS would be a better option, as there could be more than one solution to solve the problem. In fact, I have another proposal that is based on BLS12-381.

But, TBH I am not familiar with CIP/CPS, so I will follow your and others suggestions.

I am happy to help with framing the CPS (Cardano Problem Statement) to describe the overall "branch" of solutions that this and other Merkle and similar tree-based hashing structures seek to solve.

[rphair]

thanks @ilap for producing the CPS. Surely it can stay in this branch while you are drafting content & balancing it with the linked CIP... but when it's ready for review please submit it in a separate PR since the editorial process & candidacy will generally be different for these two documents. (cc @Crypto2099)

[MicroProofs]

Could add 2 new solutions in the readme as options. Here they are
https://github.com/aiken-lang/sparse-merkle-tree
https://github.com/aiken-lang/merkle-patricia-forestry

[perturbing]

Could add 2 new solutions in the readme as options. Here they are

While you are at is, could you also add this one as well?

https://github.com/perturbing/plutus-accumulator

It is plutusTx, but the aiken variant is being made :)

[ilap]

@rphair I think it would be better that this CIP would be a placeholder/container for the existing solutions of the relevant CPS.
Any suggestion for reformat/restructure this CIP for adding these other solutions above?

[rphair]

@ilap the draft CPS you've provided in this branch is a good starting point... we can review & refine it as a CPS candidate when ready & after it's moved into a separate PR branch. Whatever "existing solutions" the community has for the problem of "ensuring the uniqueness of dynamically minted NFTs" can be mentioned by name & briefly summarised here.

According to your last suggestion, you'd then have separate Specification sections, for each "existing solution" you have, listed in sub- (H3 = ###) sections of your CIP. In general there would be (and have always had so far) one CIP for each of the methods that were solving your CPS... but we can follow your suggestion for now & see what kind of document it produces if it contains more than one specification:

• If there is too much disparity between multiple Specifications, editors would recommend it be broken up into separate CIPs.
• If there are significant common elements between them, it would make sense for those common elements to be included only once in a single CIP containing multiple specifications.

The solutions from @MicroProofs and @perturbing mentioned above have independent documentation and don't have to be detailed as separate specifications in your CIP: they only need to be mentioned in the CPS... though it's standard & welcome to compare your own specification to these in your CIP's Rationale section. It's up to the creators of those methods to submit CIPs about them: if & when that happens, they can update your CPS to link to them.

@Ryun1 @Crypto2099 please also let us know your editorial opinion before going ahead this way.

[rphair]

@ilap as it's 2½ months since answering your question about how to convert this into a CPS with corresponding CIP (#816 (comment)) I'm marking this Waiting for Author.

If you are hung up on any uncertainty about the best way of going about this, please post; we can also discuss at a CIP meeting if that would help & if so we'll put it on a near-term meeting agenda.

Recently we posted a CIP Wiki which contains some ideas (in Section 2.) about partitioning & relating CIPs and CPSs which may be of help, starting here: https://github.com/cardano-foundation/CIPs/wiki/2.-CIPs-for-Reviewers-&-Authors

[ilap]

@ilap as it's 2½ months since answering your question about how to convert this into a CPS with corresponding CIP (#816 (comment)) I'm marking this Waiting for Author.

Hi @rphair, I thought the ball was in @Ryun1 and @Crypto2099's court.

@Ryun1 @Crypto2099 please also let us know your editorial opinion before going ahead this way.

[rphair]

@ilap - that was a reasonable assumption. Without @Ryun1 @Crypto2099 commenting further, and given that all editors don't comment on all proposals, they were by default agreeing with the latest suggestions in #816 (comment). I'll tag them for review again to ensure they have nothing further to add & please you are welcome to break this up into CPS / CIP as suggested.

[rphair]

p.s. @perturbing that was a GitHub UI glitch that made me tag you for review (not intentional & can't be withdrawn) but you of course would be welcome to help answer the last question.

[rphair]

@ilap as per the last #816 (comment), here is a work which is partially a CIP and partially a CPS. We can't allow it to exist here in this indefinite state forever and so would have to close this submission unless we can work out those details.

In the meantime there has been some more written about how to draw the line between CIPs and CIPs, as well as review characteristics for both:

• https://github.com/cardano-foundation/CIPs/wiki/2.-CIPs-for-Reviewers-&-Authors
• https://github.com/cardano-foundation/CIPs/wiki/21.-Suggestions-for-CIP-review

I hope that's enough for you to work on partitioning this into a CIP + CPS if you are still interested in pursuing this. Please update us soon how how you intend to go forward.

[perturbing]

Do not want to speak for the author, but from my perspective this can be closed (see the comments above for the reason).

[ilap]

Do not want to speak for the author, but from my perspective this can be closed (see the comments above for the reason).

Hi @rphair,

I agree that this can be closed, and I'll likely submit a new PR with those suggestions.