Analysis stage

[slundqui]

This PR reworks the chainsync data to dashboard pipeline to move computation away from the dashboard and into a separate process.

Major Changes:

• New run_data_analysis.py process to run analysis after acquire_data that focuses on computation.
  • This stage currently computes spot price, fixed rate, base buffer, wallet positions, pnl, and the ticker.
• Adding new tables (and interface) to support output of data analysis.
• Simplifying dashboard frontend to query output of data analysis.

Minor Changes:

• Adding drop argument to sqlalchemy initialize_session to drop existing tables if set to True for debugging.
• Adding in create table retries due to race condition between acquire data and data analysis both trying to create tables.
• Removing obsolete agent positions data class.
• Renames WalletInfo schema to be WalletInfoFromChain.
• Removing chainsync scripts for calculating leaderboards from previous trading competition.
• Renaming table names to be snake case.
• Adding in a rerun of bots + acquire data + data analysis in system test to test data pipeline restarts.
• Splitting out system tests to different files.

Breaking Change:

• postgres db interface getter functions no longer return blockNumber as an index. This is to keep the getter functions consistent, as not every table stores the block number.

Bug Fixes:

• smart_contract_preview_transaction now takes in a block number argument to query the mock trade at that exact block. This helps solve a race condition where the preview could be mocking a position that's no longer there.

TODO:

• More plots in dashboard.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
elf-simulations	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 22, 2023 9:16pm

[dpaiton]

Overall it looks great to me! I agree with the proposed formatting & data flow. I am assuming that your tests ensure the trading pipeline still works, or that you tested it locally.

The change from using *fn_args to fn_args is fine with me, but I think this should be reflected everywhere (e.g. in ethpy we use the splats for smart_contract_read & smart_contract_transact). We don't want to have an inconsistent interface for the ethpy API.

We will be turning on the docstring linting soon (once we are done with these major tasks for the next competition), so you might want to just add the full docs to your new functions now to save a headache later. Your call.