e2e-tests: Add filtered indexer e2e tests #1503
Conversation
| tempdir = tempfile.mkdtemp() | ||
| if not args.keep_temps: | ||
| atexit.register(shutil.rmtree, tempdir, onerror=logger.error) | ||
| else: | ||
| logger.info("leaving temp dir %r", tempdir) | ||
| # Sane default for lastblock--used for read_only runs | ||
| lastblock = 93 |
There was a problem hiding this comment.
The e2e test data currently contains 93 rounds. Usually we get this via reading the e2e data to see how many rounds we should expect to process. But in this case our read-only indexer will be in a separate container from the algod/e2e test data so we have no source of truth.
To resolve that I've just defaulted this to 93--it still gets overwritten by the lastblock value in regular e2elive.py runs.
e2e_tests/src/e2e_indexer/e2elive.py
Outdated
| algoddir = None | ||
| if not args.read_only: | ||
| tempnet, lastblock = setup_algod(tempdir, args.source_net, args.s3_source_net) | ||
| algoddir = os.path.join(tempnet, "Node") |
There was a problem hiding this comment.
can these be in the else block starting at line 106?
There was a problem hiding this comment.
Probably, let me try to move it around.
| RUN mkdir -p /opt/go/indexer | ||
| RUN mkdir -p /opt/conduit-dir | ||
| RUN mkdir -p /opt/algod-dir | ||
| COPY . /opt/go/indexer |
There was a problem hiding this comment.
we'll be removing conduit from this repo? should we download the conduit repo instead?
There was a problem hiding this comment.
Yes, but when I wrote this the conduit repo didn't exist/didn't have a buildable binary.
We can either wait until everything is ready (which maybe it is now?) and update this or just do this in a followup commit.
I think I prefer following up on this in the future. For example, this PR #1505 is failing on this same issues and would force me to make this change as well.
Summary
Adds an e2e test which uses a conduit writer pipeline w/ a filter processor.
Uses docker-compose w/ a conduit writer, an e2e reader w/ queries, and a pgsql db.
Followup steps: working on separating these tests into conduit and indexer.
Test Plan
make e2e-filter-test-nightly