Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved scheme for Conflicts attribute storing and pricing #2913

Merged
merged 15 commits into from
Nov 8, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/Neo/Ledger/Blockchain.cs
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ private void OnFillMemoryPool(IEnumerable<Transaction> transactions)
{
if (NativeContract.Ledger.ContainsTransaction(snapshot, tx.Hash))
continue;
if (NativeContract.Ledger.ContainsConflictHash(snapshot, tx.Hash, tx.Signers.Select(s => s.Account)))
if (NativeContract.Ledger.ContainsConflictHash(snapshot, tx.Hash, tx.Signers.Select(s => s.Account), system.Settings.MaxTraceableBlocks))
continue;
// First remove the tx if it is unverified in the pool.
system.MemPool.TryRemoveUnVerified(tx.Hash, out _);
Expand Down
2 changes: 1 addition & 1 deletion src/Neo/NeoSystem.cs
Original file line number Diff line number Diff line change
Expand Up @@ -287,7 +287,7 @@ public bool ContainsTransaction(UInt256 hash)
/// <returns><see langword="true"/> if the transaction conflicts with on-chain transaction; otherwise, <see langword="false"/>.</returns>
public bool ContainsConflictHash(UInt256 hash, IEnumerable<UInt160> signers)
{
return NativeContract.Ledger.ContainsConflictHash(StoreView, hash, signers);
return NativeContract.Ledger.ContainsConflictHash(StoreView, hash, signers, Settings.MaxTraceableBlocks);
}
}
}
33 changes: 27 additions & 6 deletions src/Neo/SmartContract/Native/LedgerContract.cs
Original file line number Diff line number Diff line change
Expand Up @@ -47,13 +47,21 @@ internal override ContractTask OnPersist(ApplicationEngine engine)
engine.Snapshot.Add(CreateStorageKey(Prefix_Block).Add(engine.PersistingBlock.Hash), new StorageItem(Trim(engine.PersistingBlock).ToArray()));
foreach (TransactionState tx in transactions)
{
// Remove possible previously saved malicious conflict records for the transaction (if any).
foreach (var (key, _) in engine.Snapshot.Find(CreateStorageKey(Prefix_Transaction).Add(tx.Transaction.Hash).ToArray()))
engine.Snapshot.Delete(key);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The discussion started by @roman-khimov and ported from AnnaShaleva@deb1470#r127677374:

Not sure about this. It's garbage collection effectively. Can slow down tx processing.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's true and everything will work without this line. This garbage collection may be safely run in a separate routine (or as a separate task), we don't need to include it into OnPersist, but I'm not sure what's the right place for it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd say that's a job for RemoveUntraceableBlocks option (C# node doesn't have it, but it can be implemented). This storage space is paid for at the same time with the new setting, so keeping these entries is OK.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that we should have a maximum allowed signers too

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having maximum allowed signers will bring us to the same problem that was described in #2907 (comment). So any kind of such restriction leads to this possible attack when malicious conflicts prevent some real (and valid) conflict from entering the chain.

That's why we build this new solution in such way so that all conflicts data are properly paid. And it's even harder to spam the DB with Conflicts-related data than via contract storage.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's not necessary to care about those "dust" records and they don't really hurt.

correct attr fee introduced by @shargon is able to avoid the abuse and restrict the storage cost.

an uncontrollable O(n) procedure here can be avoided

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

an uncontrollable O(n) procedure here can be avoided

I agree, let's then keep these garbage records. Implemented in a separate commit, see the 024a82a.


engine.Snapshot.Add(CreateStorageKey(Prefix_Transaction).Add(tx.Transaction.Hash), new StorageItem(tx));

// Store transaction's conflicits.
var conflictingSigners = tx.Transaction.Signers.Select(s => s.Account);
foreach (var attr in tx.Transaction.GetAttributes<Conflicts>())
{
var conflictRecord = engine.Snapshot.GetAndChange(CreateStorageKey(Prefix_Transaction).Add(attr.Hash),
() => new StorageItem(new TransactionState { ConflictingSigners = Array.Empty<UInt160>() })).GetInteroperable<TransactionState>();
conflictRecord.ConflictingSigners = conflictRecord.ConflictingSigners.Concat(conflictingSigners).Distinct().ToArray();
engine.Snapshot.Add(CreateStorageKey(Prefix_Transaction).Add(attr.Hash), new StorageItem(new TransactionState() { BlockIndex = engine.PersistingBlock.Index }));
Copy link
Member

@superboyiii superboyiii Nov 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to hack a private net by the code from #2907, when multi transactions want to cancel the same tx(the same conflict attribute), it will throw InvalidOperationException when they all change the same key's vale in the same block but it passed consensus. So finally it will be like:
image
1699261532080

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's my poor knowledge of NeoC# codebase, the intention here was to replace the existing conflict record (if it's in the DB), but engine.Snapshot.Add doesn't allow to do it. I'll fix it, thank you for testing!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@superboyiii, could you, please, run the same test? I've modified the code so that now it's possible to rewrite existing malicious conflict record with the proper transaction.

foreach (var signer in conflictingSigners)
{
engine.Snapshot.Add(CreateStorageKey(Prefix_Transaction).Add(attr.Hash).Add(signer), new StorageItem(new TransactionState() { BlockIndex = engine.PersistingBlock.Index }));
}
}
}
engine.SetState(transactions);
Expand Down Expand Up @@ -145,11 +153,24 @@ public bool ContainsTransaction(DataCache snapshot, UInt256 hash)
/// <param name="snapshot">The snapshot used to read data.</param>
/// <param name="hash">The hash of the conflicting transaction.</param>
/// <param name="signers">The list of signer accounts of the conflicting transaction.</param>
/// <param name="maxTraceableBlocks">MaxTraceableBlocks protocol setting.</param>
/// <returns><see langword="true"/> if the blockchain contains the hash of the conflicting transaction; otherwise, <see langword="false"/>.</returns>
public bool ContainsConflictHash(DataCache snapshot, UInt256 hash, IEnumerable<UInt160> signers)
public bool ContainsConflictHash(DataCache snapshot, UInt256 hash, IEnumerable<UInt160> signers, uint maxTraceableBlocks)
{
var state = snapshot.TryGet(CreateStorageKey(Prefix_Transaction).Add(hash))?.GetInteroperable<TransactionState>();
return state is not null && state.Transaction is null && (signers is null || state.ConflictingSigners.Intersect(signers).Any());
// Check the dummy stub firstly to define whether there's exist at least one conflict record.
var stub = snapshot.TryGet(CreateStorageKey(Prefix_Transaction).Add(hash))?.GetInteroperable<TransactionState>();
if (stub is null || stub.Transaction is not null || !IsTraceableBlock(snapshot, stub.BlockIndex, maxTraceableBlocks))
return false;

// At least one conflict record is found, then need to check signers intersection.
foreach (var signer in signers)
{
var state = snapshot.TryGet(CreateStorageKey(Prefix_Transaction).Add(hash).Add(signer))?.GetInteroperable<TransactionState>();
if (state is not null && IsTraceableBlock(snapshot, state.BlockIndex, maxTraceableBlocks))
return true;
}

return false;
}

/// <summary>
Expand Down
17 changes: 7 additions & 10 deletions src/Neo/SmartContract/Native/TransactionState.cs
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,6 @@ public class TransactionState : IInteroperable
/// </summary>
public Transaction Transaction;

public UInt160[] ConflictingSigners;

/// <summary>
/// The execution state
/// </summary>
Expand All @@ -47,7 +45,6 @@ IInteroperable IInteroperable.Clone()
{
BlockIndex = BlockIndex,
Transaction = Transaction,
ConflictingSigners = ConflictingSigners,
State = State,
_rawTransaction = _rawTransaction
};
Expand All @@ -58,7 +55,6 @@ void IInteroperable.FromReplica(IInteroperable replica)
TransactionState from = (TransactionState)replica;
BlockIndex = from.BlockIndex;
Transaction = from.Transaction;
ConflictingSigners = from.ConflictingSigners;
State = from.State;
if (_rawTransaction.IsEmpty)
_rawTransaction = from._rawTransaction;
Expand All @@ -67,20 +63,21 @@ void IInteroperable.FromReplica(IInteroperable replica)
void IInteroperable.FromStackItem(StackItem stackItem)
{
Struct @struct = (Struct)stackItem;
if (@struct.Count == 1)
{
ConflictingSigners = ((VM.Types.Array)@struct[0]).Select(u => new UInt160(u.GetSpan())).ToArray();
return;
}
BlockIndex = (uint)@struct[0].GetInteger();

// Conflict record.
if (@struct.Count == 1) return;

// Fully-qualified transaction.
_rawTransaction = ((ByteString)@struct[1]).Memory;
Transaction = _rawTransaction.AsSerializable<Transaction>();
State = (VMState)(byte)@struct[2].GetInteger();
}

StackItem IInteroperable.ToStackItem(ReferenceCounter referenceCounter)
{
if (Transaction is null) return new Struct(referenceCounter) { new VM.Types.Array(referenceCounter, ConflictingSigners.Select(u => new ByteString(u.ToArray())).ToArray()) };
if (Transaction is null)
return new Struct(referenceCounter) { BlockIndex };
if (_rawTransaction.IsEmpty)
_rawTransaction = Transaction.ToArray();
return new Struct(referenceCounter) { BlockIndex, _rawTransaction, (byte)State };
Expand Down
21 changes: 17 additions & 4 deletions tests/Neo.UnitTests/Ledger/UT_Blockchain.cs
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ public void TestMaliciousOnChainConflict()
{
Header = new Header()
{
Index = 10000,
Index = 5, // allow tx1, tx2 and tx3 to fit into MaxValidUntilBlockIncrement.
MerkleRoot = UInt256.Zero,
NextConsensus = UInt160.Zero,
PrevHash = UInt256.Zero,
Expand All @@ -149,13 +149,26 @@ public void TestMaliciousOnChainConflict()
sb.EmitSysCall(ApplicationEngine.System_Contract_NativeOnPersist);
onPersistScript = sb.ToArray();
}
TransactionState[] transactionStates;
using (ApplicationEngine engine2 = ApplicationEngine.Create(TriggerType.OnPersist, null, snapshot, block, TestBlockchain.TheNeoSystem.Settings, 0))
{
engine2.LoadScript(onPersistScript);
if (engine2.Execute() != VMState.HALT) throw new InvalidOperationException();
Blockchain.ApplicationExecuted application_executed = new(engine2);
transactionStates = engine2.GetState<TransactionState[]>();
engine2.Snapshot.Commit();
}
snapshot.Commit();

// Run PostPersist to update current block index in native Ledger.
// Relevant current block index is needed for conflict records checks.
byte[] postPersistScript;
using (ScriptBuilder sb = new())
{
sb.EmitSysCall(ApplicationEngine.System_Contract_NativePostPersist);
postPersistScript = sb.ToArray();
}
using (ApplicationEngine engine2 = ApplicationEngine.Create(TriggerType.PostPersist, null, snapshot, block, TestBlockchain.TheNeoSystem.Settings, 0))
{
engine2.LoadScript(postPersistScript);
if (engine2.Execute() != VMState.HALT) throw new InvalidOperationException();
engine2.Snapshot.Commit();
}
snapshot.Commit();
Expand Down
9 changes: 2 additions & 7 deletions tests/Neo.UnitTests/Ledger/UT_TransactionState.cs
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,7 @@ public void Initialize()
};
originTrimmed = new TransactionState
{
ConflictingSigners = new UInt160[]
{
new UInt160(Crypto.Hash160(new byte[] { 1, 2, 3 })),
new UInt160(Crypto.Hash160(new byte[] { 4, 5, 6 }))
}
BlockIndex = 1,
};
}

Expand All @@ -67,10 +63,9 @@ public void TestDeserializeTrimmed()
TransactionState dest = new();
((IInteroperable)dest).FromStackItem(BinarySerializer.Deserialize(ref reader, ExecutionEngineLimits.Default, null));

dest.BlockIndex.Should().Be(0);
dest.BlockIndex.Should().Be(originTrimmed.BlockIndex);
dest.Transaction.Should().Be(null);
dest.Transaction.Should().BeNull();
CollectionAssert.AreEqual(originTrimmed.ConflictingSigners, dest.ConflictingSigners);
}
}
}