Enhancement: remove import validator utility and obsolete ledger for evaluator #1146

tzaffi · 2022-07-26T14:07:53Z

Removing

Continuation of #1095:

Test Plan

CI

* integrate block processor

* add simple local ledger migration

* add fast catchup

* return empty lists from fetchApplications and fetchAppLocalStates (#1010) * Update model to converge with algod (#1005) * New Feature: Adds Data Directory Support (#1012) - Updates the go-algorand submodule hash to point to rel/beta - Moves the cpu profiling file, pid file and indexer configuration file to be options of only the daemon sub-command - Changes os.Exit() to be a panic with a special handler. This is so that defer's are handled instead of being ignored. - Detects auto-loading configuration files in the data directory and issues errors if equivalent command line arguments are supplied. - Updates the README with instructions on how to use the auto-loading configuration files and the data directory. * Update mockery version Co-authored-by: erer1243 <1377477+erer1243@users.noreply.github.com> Co-authored-by: AlgoStephenAkiki <85183435+AlgoStephenAkiki@users.noreply.github.com>

* handle ledger recovery scenario

* refactor create genesis block

* Adds Local Ledger Readme Resolves #4109 Starts Readme docs * Update docs/LocalLedger.md Co-authored-by: Will Winder <wwinder.unh@gmail.com> * Update docs/LocalLedger.md Co-authored-by: Will Winder <wwinder.unh@gmail.com> * Update docs/LocalLedger.md Co-authored-by: Will Winder <wwinder.unh@gmail.com> * Removed troubleshooting section Co-authored-by: Will Winder <wwinder.unh@gmail.com>

Part 1 cleanup genesis file access. put node catchup into a function that can be swapped out with the catchup service. pass the indexer logger into the block processor. move open ledger into a util function, and move the initial state util function into a new ledger util file. add initial catchupservice implementation. move ledger init from daemon.go to constructor. Merge multiple read genesis functions. Part 2 Merge local_ledger migration package into blockprocessor. Rename Migration to Initialize Use logger in catchup service catchup Part 3 Update submodule and use NewWrappedLogger. Make util.CreateInitState private

…1075)

Refactors daemon cmd into separate, testable pieces.

Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com>

fix TestWriteReadResources

codecov · 2022-07-26T14:16:44Z

Codecov Report

Merging #1146 (9f657d2) into develop (ed2a991) will decrease coverage by 0.18%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop    #1146      +/-   ##
===========================================
- Coverage    60.07%   59.89%   -0.19%     
===========================================
  Files           52       51       -1     
  Lines         8752     8281     -471     
===========================================
- Hits          5258     4960     -298     
+ Misses        3012     2862     -150     
+ Partials       482      459      -23

Impacted Files	Coverage Δ
idb/postgres/postgres.go	`63.92% <ø> (+2.74%)`	⬆️
fetcher/fetcher.go	`50.87% <0.00%> (-1.32%)`	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us.

idb/postgres/postgres_rand_test.go

winder

LGTM

Co-authored-by: Will Winder <wwinder.unh@gmail.com>

* Bump version to 2.13.0-rc1 * Bump version to 2.13.0 * Documentation for data directory. (#1125) Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com> * Don't lookup big foreign assets. (#1141) * Revert "Bump version to 2.13.0" This reverts commit 0a8af61. * Bump version to 2.13.0 * Fix import performance test runner. (#1133) * Start on round 1 since round 0 is now computed from the genesis file. * Wait for indexer processor to exit. * Better logging for metric collection errors. * Proper support for data directory. * New test script for future release automation. * Revert "Bump version to 2.13.0" This reverts commit 7915890. * Bump version to 2.13.0 * test fixes: Submodule updates (#1144) * Update go-algorand submodule * Fix test failure due to duplicate txns * Add new ledger interface method * Enhancement: remove import validator utility and obsolete ledger for evaluator (#1146) removing a bunch of code and make the random test pass with the new ledger for evaluator * Docs: Readme update (#1149) * Update README header * Testing: Use tempdir instead of /tmp for e2elive test (#1152) * Format misc/*.py with `black` (#1153) * apply black to e2elive.py as well (#1154) * Enhancement: More information about S3 keys searched for and Dockerfile that uses submodule instead of channel (#1151) * Eric's Dockerfile improvements * Update misc/e2elive.py Co-authored-by: DevOps Service <devops-service@algorand.com> Co-authored-by: Will Winder <wwinder.unh@gmail.com> Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com> Co-authored-by: Barbara Poon <barbara.poon@algorand.com> Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com>

* Bump version to 2.13.0-rc1 * Bump version to 2.13.0 * Documentation for data directory. (#1125) Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com> * Don't lookup big foreign assets. (#1141) * Revert "Bump version to 2.13.0" This reverts commit 0a8af61. * Bump version to 2.13.0 * Fix import performance test runner. (#1133) * Start on round 1 since round 0 is now computed from the genesis file. * Wait for indexer processor to exit. * Better logging for metric collection errors. * Proper support for data directory. * New test script for future release automation. * Revert "Bump version to 2.13.0" This reverts commit 7915890. * Bump version to 2.13.0 * test fixes: Submodule updates (#1144) * Update go-algorand submodule * Fix test failure due to duplicate txns * Add new ledger interface method * Enhancement: remove import validator utility and obsolete ledger for evaluator (#1146) removing a bunch of code and make the random test pass with the new ledger for evaluator * Docs: Readme update (#1149) * Update README header * Testing: Use tempdir instead of /tmp for e2elive test (#1152) * Format misc/*.py with `black` (#1153) * apply black to e2elive.py as well (#1154) * Enhancement: More information about S3 keys searched for and Dockerfile that uses submodule instead of channel (#1151) * Eric's Dockerfile improvements * Update misc/e2elive.py * Bug-Fix: Implement BlockHdrCached + miscellany (#1162) * Enhancement: add max int64 checks (#1166) * state proofs: Indexer Support for State Proofs (#1002) Adds API support to the Indexer for State Proof Transactions and header fields. Co-authored-by: Will Winder <wwinder.unh@gmail.com> Co-authored-by: Will Winder <wwinder.unh@gmail.com> * Bump version to 2.14.0-rc1 * Stop Panics if no config is supplied (#1180) Give a default config if not supplied to stop panics. * Fix spec name collisions. (#1182) * Update go-algorand submodule to v3.9.1-beta (#1185) * Bump version to 2.14.0-rc2 * Disable deadlock detection (#1186) * Add support for new block header: TxnRoot SHA256 (#989) * Accept yaml and yml configuration files. (#1181) * Fix bug in reveals lookup (#1198) * Fix bug in reveals lookup (#1198) * Bump version to 2.14.0-rc3 * add state proof example with high reveal index - from betanet (#1199) * Devops: Bump go-algorand submodule to v3.9.2-beta (#1203) * Bump version to 2.14.0-rc4 * enhancement: Clarify REST query parameters for accounts search (#1201) * update description for /v2/accounts * cicd: add darwin arm64 support to release script (#1169) * Bump version to 2.14.0 * Downgrade mockery to prevent incorrect deprecation warning. (#1211) * Enhancement: update e2e test policy (#1197) *update e2e test policy * Fix release 2.14.0 (#1214) * Accept yaml and yml configuration files. (#1181) * Fix bug in reveals lookup (#1198) * add state proof example with high reveal index - from betanet (#1199) * enhancement: Clarify REST query parameters for accounts search (#1201) * update description for /v2/accounts * cicd: add darwin arm64 support to release script (#1169) * Downgrade mockery to prevent incorrect deprecation warning. (#1211) * Enhancement: update e2e test policy (#1197) *update e2e test policy * Update test expected value: transaction root sha256 Co-authored-by: AlgoStephenAkiki <85183435+AlgoStephenAkiki@users.noreply.github.com> Co-authored-by: Michael Diamant <michaeldiamant@users.noreply.github.com> Co-authored-by: algoidan <79864820+algoidan@users.noreply.github.com> Co-authored-by: shiqizng <80276844+shiqizng@users.noreply.github.com> Co-authored-by: algolucky <lucky.baar@algorand.com> Co-authored-by: Will Winder <wwinder.unh@gmail.com> Co-authored-by: DevOps Service <devops-service@algorand.com> Co-authored-by: Will Winder <wwinder.unh@gmail.com> Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com> Co-authored-by: Barbara Poon <barbara.poon@algorand.com> Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com> Co-authored-by: shiqizng <80276844+shiqizng@users.noreply.github.com> Co-authored-by: AlgoStephenAkiki <85183435+AlgoStephenAkiki@users.noreply.github.com> Co-authored-by: John Lee <john.lee@algorand.com> Co-authored-by: Or Aharonee <17099688+Aharonee@users.noreply.github.com> Co-authored-by: Michael Diamant <michaeldiamant@users.noreply.github.com> Co-authored-by: algoidan <79864820+algoidan@users.noreply.github.com> Co-authored-by: algolucky <lucky.baar@algorand.com>

shiqizng and others added 28 commits June 8, 2022 09:59

Local Ledger (#1011)

12f8c2f

* integrate block processor

Local Ledger Deployment (#1013)

45b9dbe

* add simple local ledger migration

add deleted opts

d40c76b

fast catchup (#1023)

eff4e6f

* add fast catchup

recovery scenario (#1024)

5a03dc8

* handle ledger recovery scenario

refactor create genesis block (#1026)

621c135

* refactor create genesis block

update ledger file path and migration (#1042)

5f5c7fe

build: merge develop into localledger/integration (#1062)

409d725

Ledger init status (#1058)

a1a08b2

Generate an error if the catchpoint is not valid for initialization. (#…

f40e112

…1075)

Use main logger in handler and fetcher. (#1077)

932e0cf

Switch from fullNode catchup to catchpoint catchup service. (#1076)

997719a

Refactor daemon, add more tests (#1039)

f3f1177

Refactors daemon cmd into separate, testable pieces.

Merge develop into localledger/integration (#1083)

6b24968

Merge branch 'develop' into localledger/integration

f977817

Misc Local Ledger cleanup (#1086)

1b89a6b

Update processor/blockprocessor/initialize.go

0290cc3

Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com>

Remove old ledger for evaluator and import validator utility.

ec9c9e0

Fix first test.

782ae20

WIP

d909191

fix TestWriteReadResources

dc59c1d

lint

1ce9049

Merge pull request #1 from tzaffi/amend-remove-import-validator

e324793

fix TestWriteReadResources

Merge branch 'develop' into tzaffi/remove-import-validator

b56bf97

fix import to non-existing package

48fdfc9

tzaffi changed the title ~~Tzaffi/remove import validator utility and obsolete ledger for evaluator~~ Enhancement: remove import validator utility and obsolete ledger for evaluator Jul 26, 2022

tzaffi added the Enhancement New feature or request label Jul 26, 2022

tzaffi requested review from winder, shiqizng and Eric-Warehime July 26, 2022 14:13

winder mentioned this pull request Jul 26, 2022

Remove old ledger for evaluator and import validator utility. #1095

Closed

winder reviewed Jul 26, 2022

View reviewed changes

idb/postgres/postgres_rand_test.go Outdated Show resolved Hide resolved

winder approved these changes Jul 26, 2022

View reviewed changes

Update idb/postgres/postgres_rand_test.go

9f657d2

Co-authored-by: Will Winder <wwinder.unh@gmail.com>

tzaffi mentioned this pull request Jul 26, 2022

New Feature: Indexer Application Box Ingestion #1067

Merged

13 tasks

tzaffi merged commit 646ca85 into develop Jul 26, 2022

tzaffi deleted the tzaffi/remove-import-validator branch July 26, 2022 14:51

This was referenced Aug 10, 2022

FOR REVIEW ONLY: indexer 2.14.0 into master #1170

Merged

FOR REVIEW ONLY: indexer 2.14.0 into develop #1171

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhancement: remove import validator utility and obsolete ledger for evaluator #1146

Enhancement: remove import validator utility and obsolete ledger for evaluator #1146

tzaffi commented Jul 26, 2022

codecov bot commented Jul 26, 2022 •

edited

Loading

winder left a comment

Enhancement: remove import validator utility and obsolete ledger for evaluator #1146

Enhancement: remove import validator utility and obsolete ledger for evaluator #1146

Conversation

tzaffi commented Jul 26, 2022

Removing

Test Plan

codecov bot commented Jul 26, 2022 • edited Loading

Codecov Report

winder left a comment

Choose a reason for hiding this comment

codecov bot commented Jul 26, 2022 •

edited

Loading