Storage chains part 1 #7868

arkpar · 2021-01-11T10:59:35Z

Initial groundwork towards supporting storage chains in substrate. This includes transactions addressable in the database and block/transaction pruning.

Future PRs will include:

Serving transactions over IPFS bitswap
Ref-counted transactions and renewals.
Offchain worker API
Runtime module

polkadot companion: paritytech/polkadot#2253

burdges · 2021-01-11T11:30:46Z

Can you explain what "storage chain" means? Why is IPFS involved? What purpose it serves, aka what is the threat model?

arkpar · 2021-01-11T14:27:31Z

The idea is to maximize on-chain storage efficiency. Here's Gav's writeup on the feature:

v1.0: Finite history direct storage chain:

6 second blocks (10/min), 4 MB max extrinsics/block
IPFS-addressable extrinsics (new feature)
This chain doesn't have its own internal economy but rather uses a DOT derivative.
State only handles data market and maintains upcoming renewals. Each block has an array of sizes, one for each extrinsic, rounded up to 256 bytes. Assuming max 16MB extrinsics a 24-bit size), we can drop the lower 8-bits to get a Vec<u16>. Assuming 40KB average item size, then ~~100 items per block, =~~ 200 bytes state per block. 100,000 state entries for each block in 7 day period, and ~20 MB state.
Each block from the last 7 days also has a 32-byte Merkle root stored in state. The trie is formed (in-memory) from the 32-byte chunked extrinsics (or perhaps use some other chunking means), indexed by (extrinsic_index, chunk_index). This should be checked as part of block verification.
Creating a new block requires solving a short proof-of-work. The seed for the PoW is derived from several randomly selected bytes from the block 7 days prior (and which is now going out of scope); the proof of these bytes as well as the PoW solution is included on-chain. The risk of short-term forks may first be reduced through a proof-of-stake mechanism which applies different difficult requirements to each potential block author.
Block authors are paid out 50% of the fees immediately, and 50% is placed in a pot for them to split pro-rata weekly.
New data extrinsics charged per-byte (to pay for network storing a byte for 7 days); renewals charged per-byte on their original data (the size is in state). A single fee for the book-keeping of the size payable, too. Since everything is fixed-timespan, there's no issue with fee-charged (as opposed to deposit-taking).
Network maintainers (i.e. full nodes) are required to retain
Assume only 7-day block history (and all current state) needed for sync of full nodes.
All data (extrinsics) kept for minimum 7 days. Data can be renewed with a simple on-chain tx (no need to resend data).
Nodes throw out old (> 7 days) blocks and their data.
Total max block data needed to be stored by full-nodes: 7*24*60*10*4MB =~ 400GB
Needs fast-sync (warp sync not so important) along with only partial block history download. Syncing beyond 7 days of history is designed to be needless for running a block-authoring full-node.
An optimised publication of blocks allow extrinsics whose hash already appears in chain history to be ommitted. For normal sync-behaviour, the full block (all extrinsics) remains (to ensure that nodes which are not yet synced and don't have full history can still sync properly)

cheme

So, to be sure, the PR introduce:

an alternate block storage mode config (not switchable) that allows addressing extrinsics directly by their hash.
pruning for block body (in both mode).

Wondering a bit about addressing by something like 'BlockID ++ Extrinsic index' instead of extrinsic hash.
But I guess it doesn't make sense if extrinsic/transaction hash is going to be replaced with ipfs address and support ref counting?

'Transaction' usage in naming and description instead of block extrinsic confused me a little, but it is probably related to the targeted functionalities (which I am unaware of).

cheme · 2021-01-13T08:27:35Z

client/service/src/config.rs

+	/// State pruning settings.
+	pub state_pruning: PruningMode,
+	/// Block pruning settings.
+	pub keep_blocks: KeepBlocks,


block_pruning ? body_pruning ? for name consistency.

Unlike state pruning, the only setting here is the number of blocks to keep. So I went with more straightforward name.

cheme · 2021-01-13T08:32:05Z

client/db/src/lib.rs

+						let mut hashes = Vec::with_capacity(body.len());
+						for extrinsic in body {
+							let extrinsic = extrinsic.encode();
+							let hash = HashFor::<Block>::hash(&extrinsic);


Is there some collision possible here? two identically encoded extrinsic between two different blocks.
Edit: just viewed the 'ref counted' comment in the pr description, just not 100% sure it is related.

Collisions will be handled by reference counting indeed. ParityDb should use internal reference counting mechanism and for RocksDb there will be an explicit counter.

Where is this explicit counter?

Or do you mean this counter is added automatically by kvdb?

Not implemented yet. This will require a new extrinsic format. Will be added in a future PR.

Why does this require a new extrinsic format?

The native code will have to distinguish between extrinsics that actually contain new data and the ones that reference a previous extrinsic. This will probably require some special version of OpaqueExtrinsic.
Quoting Gav:

so one way would be introducing a new kind of extrinsic which is like a "by-reference" extrinsic.
and assumes that the node either already stores the preimage or that it can easily find it from other nodes.
this might also be useful for block propagation and alleviate the need to gossip the entire block if you know most of your peers already have the transactions in it

Also according to Gav, this should not be handled by the runtime, but rather by the database/networking layer.

Ahh okay, makes sense.

cheme · 2021-01-13T09:00:49Z

client/db/src/lib.rs

-	fn new(db: Arc<dyn Database<DbHash>>) -> ClientResult<Self> {
+	fn new(
+		db: Arc<dyn Database<DbHash>>,
+		transaction_storage: TransactionStorage


Suggested change

transaction_storage: TransactionStorage

transaction_storage: TransactionStorage,

'TransactionStorage' name confused me a little, my first thought was that it was a specific storage instance.

Maybe 'Mode' or 'Config' naming variant or 'StoreTransaction' (similar to 'KeepBlock') could be alternative naming.

cheme · 2021-01-13T09:27:13Z

client/db/src/lib.rs

+			if finalized >= keep.into() {
+				let number = finalized.saturating_sub(keep.into());
+				match read_db(&*self.storage.db, columns::KEY_LOOKUP, columns::BODY, BlockId::<Block>::number(number))? {
+					Some(body) => {


Is there a use case where we want to remove the transactions but not the body reference? (from description not really)

Don't think so

arkpar · 2021-01-13T10:15:34Z

Wondering a bit about addressing by something like 'BlockID ++ Extrinsic index' instead of extrinsic hash.
But I guess it doesn't make sense if extrinsic/transaction hash is going to be replaced with ipfs address and support ref counting?

Right, IPFS content is addressable by content hash. And the idea with renewal transactions is that that they put the same content on the chain by increasing the reference counter.

'Transaction' usage in naming and description instead of block extrinsic confused me a little, but it is probably related to the targeted functionalities (which I am unaware of).

My understanding that we use the term "transaction" for user generated pieces of block body. And "extrinsic" should not be user facing. So I've used "transaction" in the database layer to underline the fact that this is intended for storing blocks of user data, addressable by data hash.

cheme

My understanding that we use the term "transaction" for user generated pieces of block body. And "extrinsic" should not be user facing. So I've used "transaction" in the database layer to underline the fact that this is intended for storing blocks of user data, addressable by data hash.

Makes me a bit curious about this 'Finite history direct storage chain' purpose.

I am not really sold to using a different name (transaction) in the context of this storage mode.

Anyway, code looks good to me.

burdges · 2021-01-13T12:17:19Z

The idea is to maximize on-chain storage efficiency. Here's Gav's writeup on the feature:

What does it store? And why?

Ignoring parachains internal archetecture, there are three storage security models proposed for Polkadot:

Relay chain blocks and state known by all validators.
Availability store data guaranteed available by validators, but efficiently erasure coded across all validators. It expires data after 24 hours currently.
XCMP messages provided by a sending block's approval checkers.
Archival long-term storage where we make zero security guarantees and data could simply vanish forever, but it theoretically exists for curious people.

If I understand, you've designed an archival-ish storage system with zero security guarantees, meaning PVFs cannot ever depend upon data stored here, yes? In other words, anytime a PVF wants data stored here it must slurp said data back into its own block, which then makes the data available for validators. If so, why should this require Polkadot integration?

Also, you expire data after 7 days. Why archive such a short time?

You're doing so as a parachain. Why? As a parachain, this pushes its data into the availability store, meaning this costs a parachain exactly what placing data into its own blocks costs.

* IPFS-addressable extrinsics (new feature)

What does this mean?

* State only handles data market and maintains upcoming renewals.

So there is zero user data in these block? Only IPFS addresses for which storage nodes store the data? If so, yes this makes sense as a storage model. :)

Why would this require any special polkadot integration though? We cannot use this data from polkadot directly so why not do everything inside the parachain's runtime?

Is the 7 days just an accounting feature?

* Each block from the last 7 days also has a 32-byte Merkle root stored in state. The trie is formed (in-memory) from the 32-byte chunked extrinsics (or perhaps use some other chunking means), indexed by `(extrinsic_index, chunk_index)`. This should be checked as part of block verification.

What does this mean?

* Creating a new block requires solving a short proof-of-work. The seed for the PoW is derived from several randomly selected bytes from the block 7 days prior (and which is now going out of scope); the proof of these bytes as well as the PoW solution is included on-chain. The risk of short-term forks may first be reduced through a proof-of-stake mechanism which applies different difficult requirements to each potential block author.

I'm confused: A priori, I'd envision this being some parathread, so surely paying the parathread fee mostly suffices, no? Are we talking huge amounts of data here? Like what?

* Block authors are paid out 50% of the fees immediately, and 50% is placed in a pot for them to split pro-rata weekly.

Is the block author hosting all the user data? If not, they're not doing much work, so no reason to pay them. We've no mechanism to enforce they serve the data obviously, but that's fine since we'll never depend upon the data.

* Network maintainers (i.e. full nodes) are required to retain

Who are these? Not Polkadot full nodes obviously. It's whoever joins this parachain?

* Assume only 7-day block history (and all current state) needed for sync of full nodes.

* All data (extrinsics) kept for minimum 7 days. Data can be renewed with a simple on-chain tx (no need to resend data).

That's useful.

Sorry that got long..

In short, I'm unsure why this requires any polkadot integration, given that polkadot cannot depend upon archival-like guarantees.

Just fyi, we'll later expand availability store usage for required data: At present, anytime a parachain calls set_code we require all polkadot nodes see its code immediately. In theory, parachains could post_code with their own parachain candidate block, which then stores the new code into the erasure coded availability store. After we finalize the relay chain block including post_code parachain candidate, then the parachain could switch_code into the new code. At this point, we've perhaps only one polkadot validator who possesses the code. Yet, any who require this code could fetch it like approval checkers fetch the blocks they check, thanks to the availability store. We'll require this for hierarchical/multiple relay chain operation, and maybe it'd make polkadot more efficient even now, but currently we're hoping code churns slowly enough to make it worth storing parachain code on the relay chain.

arkpar · 2021-01-13T13:51:48Z

@burdges This PR does not mention polkadot or parachains anywhere. This is to add generic support for such chains in substrate. The idea is to store user data not in the state merkle tree, but rather as block bodies. Runtime only keeps track of data hashes but has no access to the data itself. This way storage would be way less expensive. @gavofyork can probably explain rationale better than myself.

Users send data as transaction payload, which is added to the block database (hosted by all full nodes) and may be retrieved by CID (data hash) over IPFS.
The payload is removed from the database after 7 days unless there's another renewal transaction that references existing data hash. The runtime only validates the merkle tree route of all the added/renewed data hashes in the block.

burdges · 2021-01-13T13:58:50Z

Runtime only keeps track of data hashes but has no access to the data itself.

Alright sounds fine then. I'm still unclear on why this needs special support, but anyways it breaks nothing afaik. :)

burdges · 2021-01-14T08:41:58Z

What purpose does the PoW serve here? Any block production constraint suffices, no?

arkpar · 2021-01-14T10:33:52Z

@burdges right. Initial implementation will use existing consensus at least. Not sure why @gavofyork mentioned PoW there, but it is out of the scope for now.

Alright sounds fine then. I'm still unclear on why this needs special support, but anyways it breaks nothing afaik. :)

Special support would be in the database/networking layer to enable querying and reference-counting individual transactions.

…orage

bkchr

Looks good to me. Mostly nitpicks.

We should make sure that this appears in the next polkadot release as information for the users that the db format changes.

client/db/src/lib.rs

client/db/src/upgrade.rs

client/db/src/utils.rs

bkchr · 2021-01-14T12:21:31Z

client/db/src/utils.rs

@@ -327,6 +327,23 @@ pub fn read_db<Block>(
 	})
 }

+/// Remove database column entry for the given block.
+pub fn remove_db<Block>(


I see that this naming follows the read_db function above, but it is really confusing to me IMHO.

Isn't this more a remove_block_from_db or similar?

If I understand this correctly, this will remove a block entry from the db?

It is generic and may remove anything that is referenced with col_index. E.g. justifications. I'll rename it to remove_from_db

bkchr · 2021-01-14T12:33:06Z

client/db/src/lib.rs

+						let mut hashes = Vec::with_capacity(body.len());
+						for extrinsic in body {
+							let extrinsic = extrinsic.encode();
+							let hash = HashFor::<Block>::hash(&extrinsic);


Or do you mean this counter is added automatically by kvdb?

bkchr · 2021-01-14T12:37:23Z

client/db/src/lib.rs

+		if let KeepBlocks::Some(keep_blocks) = self.keep_blocks {
+			// Always keep the last finalized block
+			let keep = std::cmp::max(keep_blocks, 1);
+			if finalized >= keep.into() {


Suggested change

if finalized >= keep.into() {

if finalized >= keep.into() {

return Ok(())

}

Than we don't require that much indentation :D

client/db/src/lib.rs

bkchr · 2021-01-14T12:39:44Z

client/db/src/lib.rs

+							columns::BODY,
+							BlockId::<Block>::number(number)
+						)?;
+						debug!(target: "db", "Removing block #{}", number);


Should be moved above the remove_db call, because that one could fail and we would not see this log message.

bkchr · 2021-01-14T12:41:07Z

client/db/src/lib.rs

+						match self.transaction_storage {
+							TransactionStorageMode::BlockBody => {},
+							TransactionStorageMode::StorageChain => {
+								match Vec::<Block::Hash>::decode(&mut &body[..]) {
+									Ok(hashes) => {
+										for h in hashes {
+											transaction.remove(columns::TRANSACTION, h.as_ref());
+										}
+									}
+									Err(err) => return Err(sp_blockchain::Error::Backend(
+										format!("Error decoding body list: {}", err)
+									)),
+								}
+							}
+						}


Suggested change

match self.transaction_storage {

TransactionStorageMode::BlockBody => {},

TransactionStorageMode::StorageChain => {

match Vec::<Block::Hash>::decode(&mut &body[..]) {

Ok(hashes) => {

for h in hashes {

transaction.remove(columns::TRANSACTION, h.as_ref());

}

}

Err(err) => return Err(sp_blockchain::Error::Backend(

format!("Error decoding body list: {}", err)

)),

}

}

}

if let TransactionStorageMode::StorageChain = self.transaction_storage {

match Vec::<Block::Hash>::decode(&mut &body[..]) {

Ok(hashes) => {

for h in hashes {

transaction.remove(columns::TRANSACTION, h.as_ref());

}

}

Err(err) => return Err(sp_blockchain::Error::Backend(

format!("Error decoding body list: {}", err)

)),

}

}

Would really prefer to keep match here for the same reason match is required to be exhaustive in rust. Code update will be required In case a new variant is added to the enum.

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

bkchr · 2021-01-14T14:56:15Z

client/db/src/lib.rs

+						let mut hashes = Vec::with_capacity(body.len());
+						for extrinsic in body {
+							let extrinsic = extrinsic.encode();
+							let hash = HashFor::<Block>::hash(&extrinsic);


Why does this require a new extrinsic format?

…orage

bkchr · 2021-01-14T18:55:37Z

bot merge

ghost · 2021-01-14T18:55:40Z

Trying merge.

h4x3rotab · 2021-01-15T12:06:54Z

Is there a general level tracking issue for the storage chain? I think it's very helpful for our use case: We need to store a series of "snapshot" for the encrypted confidential contract states and do re-encryption periodically. Since they are just snapshots, only the recent ones are actually useful, and therefore we would like to just prune the older snapshots by default for all the full node. It can make our blockchain much much lighter and more efficient.

arkpar added 4 commits January 10, 2021 20:24

CLI options and DB upgrade

4490333

Transaction storage

8abac6e

Block pruning

8648239

Block pruning test

2b97468

arkpar added the A3-in_progress Pull request is in progress. No review needed at this stage. label Jan 11, 2021

Style

65a2708

arkpar marked this pull request as ready for review January 12, 2021 11:43

arkpar added A0-please_review Pull request needs code review. B0-silent Changes should not be mentioned in any release notes C1-low PR touches the given topic and has a low impact on builders. and removed A3-in_progress Pull request is in progress. No review needed at this stage. labels Jan 12, 2021

github-actions bot added the A7-needspolkadotpr label Jan 12, 2021

arkpar mentioned this pull request Jan 12, 2021

Companion for substrate #7868 paritytech/polkadot#2253

Merged

arkpar requested a review from bkchr January 13, 2021 09:07

cheme reviewed Jan 13, 2021

View reviewed changes

Naming

fca526b

github-actions bot removed the A7-needspolkadotpr label Jan 13, 2021

cheme approved these changes Jan 13, 2021

View reviewed changes

Merge branch 'master' of github.com:paritytech/substrate into a-tx-st…

42740af

…orage

bkchr approved these changes Jan 14, 2021

View reviewed changes

bkchr added the B5-clientnoteworthy label Jan 14, 2021

Apply suggestions from code review

6e19ef4

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

arkpar and others added 2 commits January 14, 2021 17:38

Apply suggestions from code review

3e90eed

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Style

95ec11e

arkpar requested a review from bkchr January 14, 2021 14:46

bkchr approved these changes Jan 14, 2021

View reviewed changes

github-actions bot added the A7-needspolkadotpr label Jan 14, 2021

Merge branch 'master' of github.com:paritytech/substrate into a-tx-st…

678f828

…orage

ghost merged commit 8ee55dd into master Jan 14, 2021

ghost deleted the a-tx-storage branch January 14, 2021 18:55

arkpar mentioned this pull request Jan 23, 2021

Storage chains: Tracking issue #7962

Closed

7 tasks

insipx mentioned this pull request Jun 24, 2021

Support Storage Chains paritytech/substrate-archive#292

Merged

This pull request was closed.

	transaction_storage: TransactionStorage
	transaction_storage: TransactionStorage,

Storage chains part 1 #7868

Storage chains part 1 #7868

Conversation

arkpar commented Jan 11, 2021 • edited Loading

burdges commented Jan 11, 2021

arkpar commented Jan 11, 2021 • edited Loading

v1.0: Finite history direct storage chain:

cheme left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arkpar commented Jan 13, 2021 • edited Loading

cheme left a comment

Choose a reason for hiding this comment

burdges commented Jan 13, 2021 • edited Loading

arkpar commented Jan 13, 2021

burdges commented Jan 13, 2021

burdges commented Jan 14, 2021

arkpar commented Jan 14, 2021

bkchr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bkchr commented Jan 14, 2021

ghost commented Jan 14, 2021

h4x3rotab commented Jan 15, 2021

arkpar commented Jan 11, 2021 •

edited

Loading

arkpar commented Jan 11, 2021 •

edited

Loading

arkpar commented Jan 13, 2021 •

edited

Loading

burdges commented Jan 13, 2021 •

edited

Loading