[Merged by Bors] - Add `validator-manager` #3502

paulhauner · 2022-08-24T07:58:45Z

Issue Addressed

Addresses #2557

Proposed Changes

Adds the lighthouse validator-manager command, which provides:

lighthouse validator-manager create
- Creates a validators.json file and a deposits.json (same format as https://github.com/ethereum/staking-deposit-cli)
lighthouse validator-manager import
- Imports validators from a validators.json file to the VC via the HTTP API.
lighthouse validator-manager move
- Moves validators from one VC to the other, utilizing only the VC API.

Additional Info

In 98bcb94 I've reduced some VC ERRO and CRIT warnings to WARN or DEBG for the case where a pubkey is missing from the validator store. These were being triggered when we removed a validator but still had it in caches. It seems to me that UnknownPubkey will only happen in the case where we've removed a validator, so downgrading the logs is prudent. All the logs are DEBG apart from attestations and blocks which are WARN. I thought having some logging about this condition might help us down the track.

In 856cd7e I've made the VC delete the corresponding password file when it's deleting a keystore. This seemed like nice hygiene. Notably, it'll only delete that password file after it scans the validator definitions and finds that no other validator is also using that password file.

This reverts commit 0e335bd.

Co-authored-by: chonghe <44791194+chong-he@users.noreply.github.com>

paulhauner · 2023-08-07T06:15:27Z

When using create with the flag --network goerli, the deposits.json file created will have the field:
Network_name: “prater”
I am not sure if we use this deposits.json to send the deposit, will it go through? I see that staking-deposit-cli no longer has “prater” as an option in the network selection.

I just checked this and the launchpad will accept "prater" for the Goerli network. I suspect it'll need to keep doing so to ensure it supports old versions of CLI tooling.

jimmygchen · 2023-08-07T06:32:05Z

I just checked this and the launchpad will accept "prater" for the Goerli network. I suspect it'll need to keep doing so to ensure it supports old versions of CLI tooling.

Just dug up some old notes on this that may be relevant to the above, if it hasn't changed since I last read the launchpad code, I suspect it could even be any value 🙈 :

Launchpad doesn't actually use the fork_version, eth2_network_name, deposit_cli_version fields for computing signing root (for signature verification), these fields just need to be present in the file.

paulhauner · 2023-08-07T06:33:24Z

I suspect it could even be any value

I did try (by accident) uploading a mainnet deposit JSON when it was in Goerli mode and I was rejected. So it's doing some verification at least!

paulhauner · 2023-08-07T06:38:26Z

With this validator-manager feature, the feature account validator create and the info in: https://lighthouse-book.sigmaprime.io/key-management.html now appears to be obsolete. We have put a bold font on this page to encourage users to use staking-deposit-cli, which I think is mainly because the deposit file created using account validator create is not compatible with the staking launchpad.

Great suggestion @chong-he, thanks 🙏 I've added a deprecation notice in 5654966.

book/src/key-management.md

Co-authored-by: chonghe <44791194+chong-he@users.noreply.github.com>

paulhauner · 2023-08-07T23:44:07Z

This is ready to merge! I believe the test failures are unrelated and should be helped by #4574.

paulhauner · 2023-08-07T23:44:58Z

bors r+

## Issue Addressed Addresses #2557 ## Proposed Changes Adds the `lighthouse validator-manager` command, which provides: - `lighthouse validator-manager create` - Creates a `validators.json` file and a `deposits.json` (same format as https://github.com/ethereum/staking-deposit-cli) - `lighthouse validator-manager import` - Imports validators from a `validators.json` file to the VC via the HTTP API. - `lighthouse validator-manager move` - Moves validators from one VC to the other, utilizing only the VC API. ## Additional Info In 98bcb94 I've reduced some VC `ERRO` and `CRIT` warnings to `WARN` or `DEBG` for the case where a pubkey is missing from the validator store. These were being triggered when we removed a validator but still had it in caches. It seems to me that `UnknownPubkey` will only happen in the case where we've removed a validator, so downgrading the logs is prudent. All the logs are `DEBG` apart from attestations and blocks which are `WARN`. I thought having *some* logging about this condition might help us down the track. In 856cd7e I've made the VC delete the corresponding password file when it's deleting a keystore. This seemed like nice hygiene. Notably, it'll only delete that password file after it scans the validator definitions and finds that no other validator is also using that password file.

bors · 2023-08-07T23:46:54Z

Build failed:

ef-tests-ubuntu

paulhauner · 2023-08-07T23:47:51Z

bors retry

## Issue Addressed Addresses #2557 ## Proposed Changes Adds the `lighthouse validator-manager` command, which provides: - `lighthouse validator-manager create` - Creates a `validators.json` file and a `deposits.json` (same format as https://github.com/ethereum/staking-deposit-cli) - `lighthouse validator-manager import` - Imports validators from a `validators.json` file to the VC via the HTTP API. - `lighthouse validator-manager move` - Moves validators from one VC to the other, utilizing only the VC API. ## Additional Info In 98bcb94 I've reduced some VC `ERRO` and `CRIT` warnings to `WARN` or `DEBG` for the case where a pubkey is missing from the validator store. These were being triggered when we removed a validator but still had it in caches. It seems to me that `UnknownPubkey` will only happen in the case where we've removed a validator, so downgrading the logs is prudent. All the logs are `DEBG` apart from attestations and blocks which are `WARN`. I thought having *some* logging about this condition might help us down the track. In 856cd7e I've made the VC delete the corresponding password file when it's deleting a keystore. This seemed like nice hygiene. Notably, it'll only delete that password file after it scans the validator definitions and finds that no other validator is also using that password file.

bors · 2023-08-08T00:01:44Z

Build failed:

no-eth1-simulator-ubuntu

paulhauner · 2023-08-08T00:03:03Z

Seems like CI is having trouble making calls to HTTP servers 😕

bors retry

## Issue Addressed Addresses #2557 ## Proposed Changes Adds the `lighthouse validator-manager` command, which provides: - `lighthouse validator-manager create` - Creates a `validators.json` file and a `deposits.json` (same format as https://github.com/ethereum/staking-deposit-cli) - `lighthouse validator-manager import` - Imports validators from a `validators.json` file to the VC via the HTTP API. - `lighthouse validator-manager move` - Moves validators from one VC to the other, utilizing only the VC API. ## Additional Info In 98bcb94 I've reduced some VC `ERRO` and `CRIT` warnings to `WARN` or `DEBG` for the case where a pubkey is missing from the validator store. These were being triggered when we removed a validator but still had it in caches. It seems to me that `UnknownPubkey` will only happen in the case where we've removed a validator, so downgrading the logs is prudent. All the logs are `DEBG` apart from attestations and blocks which are `WARN`. I thought having *some* logging about this condition might help us down the track. In 856cd7e I've made the VC delete the corresponding password file when it's deleting a keystore. This seemed like nice hygiene. Notably, it'll only delete that password file after it scans the validator definitions and finds that no other validator is also using that password file.

bors · 2023-08-08T00:55:14Z

Pull request successfully merged into unstable.

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

Addresses sigp#2557 Adds the `lighthouse validator-manager` command, which provides: - `lighthouse validator-manager create` - Creates a `validators.json` file and a `deposits.json` (same format as https://github.com/ethereum/staking-deposit-cli) - `lighthouse validator-manager import` - Imports validators from a `validators.json` file to the VC via the HTTP API. - `lighthouse validator-manager move` - Moves validators from one VC to the other, utilizing only the VC API. In 98bcb94 I've reduced some VC `ERRO` and `CRIT` warnings to `WARN` or `DEBG` for the case where a pubkey is missing from the validator store. These were being triggered when we removed a validator but still had it in caches. It seems to me that `UnknownPubkey` will only happen in the case where we've removed a validator, so downgrading the logs is prudent. All the logs are `DEBG` apart from attestations and blocks which are `WARN`. I thought having *some* logging about this condition might help us down the track. In sigp@856cd7e I've made the VC delete the corresponding password file when it's deleting a keystore. This seemed like nice hygiene. Notably, it'll only delete that password file after it scans the validator definitions and finds that no other validator is also using that password file.

paulhauner added the work-in-progress PR is a work-in-progress label Aug 24, 2022

paulhauner changed the base branch from stable to unstable August 24, 2022 07:59

paulhauner added the blocked label Aug 24, 2022

paulhauner force-pushed the validator-manager branch 3 times, most recently from 04c7e1a to 25b6313 Compare August 31, 2022 00:50

paulhauner added 24 commits September 22, 2022 16:45

Add JSON deposit data to create

9503e33

Add deposit data to recover function

5af83be

Revert "Add deposit data to recover function"

e3b2020

This reverts commit 0e335bd.

Add incomplete test script

ec33951

Add changes to test.py

66e24e2

Add initial progress

1984c6b

Add progress

ff3a025

Add progress

e25526e

Start refactoring into separate commands

e463518

Remove old command

fec2969

Add function to check against beacon node

85d0843

Appease clippy

ce7f29d

Link validator_manager into Lighthouse binary

709598f

Fix mistake in command name

0cc72bc

Avoid pulling env into blocking context

b025185

Tidy logging

3b324fc

Fix deposit log creation

7b62064

Refactor to use Config struct

c27d3dd

Add testing for create command

61e0347

Start working on test vectors

5edcfb7

Add working test vector generator

79b5c6f

Pass test vectors

3bab10b

Remove keystore json

ed0f354

Add test vectors

516e4ee

Apply suggestions from code review

e86c0e6

Co-authored-by: chonghe <44791194+chong-he@users.noreply.github.com>

paulhauner added 3 commits August 7, 2023 16:37

Fix --validators help text

c07a554

Update "no validators present" log

88db492

Add deprecation notice to "Key Management"

5654966

paulhauner added v4.4.1 ETA August 2023 ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Aug 7, 2023

chong-he reviewed Aug 7, 2023

View reviewed changes

book/src/key-management.md Outdated Show resolved Hide resolved

Update book/src/key-management.md

a8865f9

Co-authored-by: chonghe <44791194+chong-he@users.noreply.github.com>

paulhauner added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Aug 7, 2023

bors bot changed the title ~~Add validator-manager~~ [Merged by Bors] - Add validator-manager Aug 8, 2023

bors bot closed this Aug 8, 2023

chong-he mentioned this pull request Aug 31, 2023

[Merged by Bors] - Feat: siren faq update #4685

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Merged by Bors] - Add `validator-manager` #3502

[Merged by Bors] - Add `validator-manager` #3502

paulhauner commented Aug 24, 2022 •

edited

Loading

paulhauner commented Aug 7, 2023

jimmygchen commented Aug 7, 2023

paulhauner commented Aug 7, 2023

paulhauner commented Aug 7, 2023

paulhauner commented Aug 7, 2023 •

edited

Loading

paulhauner commented Aug 7, 2023

bors bot commented Aug 7, 2023

paulhauner commented Aug 7, 2023

bors bot commented Aug 8, 2023

paulhauner commented Aug 8, 2023

bors bot commented Aug 8, 2023

[Merged by Bors] - Add validator-manager #3502

[Merged by Bors] - Add validator-manager #3502

Conversation

paulhauner commented Aug 24, 2022 • edited Loading

Issue Addressed

Proposed Changes

Additional Info

paulhauner commented Aug 7, 2023

jimmygchen commented Aug 7, 2023

paulhauner commented Aug 7, 2023

paulhauner commented Aug 7, 2023

paulhauner commented Aug 7, 2023 • edited Loading

paulhauner commented Aug 7, 2023

bors bot commented Aug 7, 2023

paulhauner commented Aug 7, 2023

bors bot commented Aug 8, 2023

paulhauner commented Aug 8, 2023

bors bot commented Aug 8, 2023

[Merged by Bors] - Add `validator-manager` #3502

[Merged by Bors] - Add `validator-manager` #3502

paulhauner commented Aug 24, 2022 •

edited

Loading

paulhauner commented Aug 7, 2023 •

edited

Loading