feat(katana): add executor metrics #1791
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1791 +/- ##
==========================================
+ Coverage 68.56% 68.61% +0.05%
==========================================
Files 307 308 +1
Lines 33865 33900 +35
==========================================
+ Hits 23219 23261 +42
+ Misses 10646 10639 -7 ☔ View full report in Codecov by Sentry. |
The (1) approach is definitely the more reactive one. Since Katana should usually have low block-time and potentially a looooot of transactions, shouldn't be better performance wise to go with (2)? For sure, a 20s block time would totally drop the refresh rate. :P But it still a very long block time for our use-cases I guess. |
Good point! I asked a Voyager guy and it seems like they evaluate the TPS and SPS below based on previous block. Which is how it'd be like if we go with (2)
|
|
Just realised that we wouldn't be able to see the updates in realtime anyway if the grafana refresh rate is set to more than the node's block time. By default the lowest refresh rate is 5s and i assume setting it too low could generally harm the node's performance. Doing (2) on instant mining would pretty much appear similar to (1) so this decision mostly depends on whether or not we want to receive instant updates during the pending block period in interval mining. EDIT: will change to (2) |
kariy
force-pushed
the
executor-metrics
branch
2 times, most recently
from
b144326
to
f1d95f0
Compare
This stack of pull requests is managed by Graphite. Learn more about stacking. |
ref #1791 #1369 <img width="1420" alt="Screenshot 2024-04-12 at 2 36 56 AM" src="https://github.com/dojoengine/dojo/assets/26515232/b97b49df-c5fc-429a-9cb0-bf66138a00b6"> showing total gas and steps using a simple line charts, tracking its growth over time
ref #1369 Add metrics on katana executor; tracking the total L1 **gas** and Cairo **steps** used. There were two approaches that i thought of; 1. record the metrics on every tx execution, or 2. on every block ~Decided to go with (1) as it would allow to measure it in realtime (as the tx is being executed), instead of having to wait until the block is finished being processed.~ Thought im not exactly sure which one is the ideal one. Doing (1) might be less performant bcs we have to acquire the lock to the metrics recorder more frequently (ie every tx), as opposed to only updating the metrics once every block. another thing to note, currently doing (1) would require all executor implementations to define the metrics in their own implmentations, meaning have to duplicate code. If do (2) can just define it under `block_producer` scope and be executor agnostic. EDIT: doing (2). metrics are collected upon completion of block production --- some changes are made to gather the value after block production: - simplify params on `backend::do_mine_block`, now only accept two args; `BlockEnv` and `ExecutionOutput` - add a new type `ExecutionStats` under `katana-executor`, this is where executor would store the gas and steps value
ref #1791 #1369 <img width="1420" alt="Screenshot 2024-04-12 at 2 36 56 AM" src="https://github.com/dojoengine/dojo/assets/26515232/b97b49df-c5fc-429a-9cb0-bf66138a00b6"> showing total gas and steps using a simple line charts, tracking its growth over time
ref #1369 Add metrics on katana executor; tracking the total L1 **gas** and Cairo **steps** used. There were two approaches that i thought of; 1. record the metrics on every tx execution, or 2. on every block ~Decided to go with (1) as it would allow to measure it in realtime (as the tx is being executed), instead of having to wait until the block is finished being processed.~ Thought im not exactly sure which one is the ideal one. Doing (1) might be less performant bcs we have to acquire the lock to the metrics recorder more frequently (ie every tx), as opposed to only updating the metrics once every block. another thing to note, currently doing (1) would require all executor implementations to define the metrics in their own implmentations, meaning have to duplicate code. If do (2) can just define it under `block_producer` scope and be executor agnostic. EDIT: doing (2). metrics are collected upon completion of block production --- some changes are made to gather the value after block production: - simplify params on `backend::do_mine_block`, now only accept two args; `BlockEnv` and `ExecutionOutput` - add a new type `ExecutionStats` under `katana-executor`, this is where executor would store the gas and steps value
ref #1791 #1369 <img width="1420" alt="Screenshot 2024-04-12 at 2 36 56 AM" src="https://github.com/dojoengine/dojo/assets/26515232/b97b49df-c5fc-429a-9cb0-bf66138a00b6"> showing total gas and steps using a simple line charts, tracking its growth over time
ref #1369
Add metrics on katana executor; tracking the total L1 gas and Cairo steps used.
There were two approaches that i thought of;
Decided to go with (1) as it would allow to measure it in realtime (as the tx is being executed), instead of having to wait until the block is finished being processed.Thought im not exactly sure which one is the ideal one.
Doing (1) might be less performant bcs we have to acquire the lock to the metrics recorder more frequently (ie every tx), as opposed to only updating the metrics once every block.
another thing to note, currently doing (1) would require all executor implementations to define the metrics in their own implmentations, meaning have to duplicate code. If do (2) can just define it under
block_producerscope and be executor agnostic.EDIT: doing (2). metrics are collected upon completion of block production
some changes are made to gather the value after block production:
backend::do_mine_block, now only accept two args;BlockEnvandExecutionOutputExecutionStatsunderkatana-executor, this is where executor would store the gas and steps value