Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

physical/spanner: use separate client for updating locks #9423

Merged
merged 3 commits into from
Jul 9, 2020
Merged

physical/spanner: use separate client for updating locks #9423

merged 3 commits into from
Jul 9, 2020

Conversation

sethvargo
Copy link
Contributor

We believe this mitigates an issue where a large influx of requests cause the leader to be unable to update the lock table (since it cannot grab a client from the pool or the client has no more open connections), which causes cascading failure.

@sethvargo sethvargo mentioned this pull request Jul 8, 2020
Merged
@sethvargo
Copy link
Contributor Author

Test output:

$ GOOGLE_SPANNER_DATABASE=projects/vs-test-282716/instances/test/databases/test GOOGLE_SPANNER_TABLE=Vault GOOGLE_SPANNER_HA_TABLE=VaultHA go test -v -count=1 ./physical/spanner/...
=== RUN   TestHABackend
2020-07-08T13:02:52.553-0400 [DEBUG] configuring backend
2020-07-08T13:02:52.553-0400 [DEBUG] creating HA client
2020-07-08T13:02:52.554-0400 [DEBUG] configuration: database=projects/vs-test-282716/instances/test/databases/test table=Vault haEnabled=true haTable=VaultHA maxParallel=0
2020-07-08T13:02:52.554-0400 [DEBUG] creating client
2020-07-08T13:02:52.554-0400 [DEBUG] configuring backend
2020-07-08T13:02:52.554-0400 [DEBUG] creating HA client
2020-07-08T13:02:52.555-0400 [DEBUG] configuration: database=projects/vs-test-282716/instances/test/databases/test table=Vault haEnabled=true haTable=VaultHA maxParallel=0
2020-07-08T13:02:52.555-0400 [DEBUG] creating client
--- PASS: TestHABackend (11.07s)
=== RUN   TestBackend
2020-07-08T13:03:03.113-0400 [DEBUG] configuring backend
2020-07-08T13:03:03.113-0400 [DEBUG] configuration: database=projects/vs-test-282716/instances/test/databases/test table=Vault haEnabled=false haTable=VaultHA maxParallel=0
2020-07-08T13:03:03.113-0400 [DEBUG] creating client
--- PASS: TestBackend (1.54s)
PASS
ok  	github.com/hashicorp/vault/physical/spanner	12.819s

@kalafut kalafut requested a review from calvn July 9, 2020 00:28
calvn
calvn approved these changes Jul 9, 2020
View reviewed changes
Copy link
Contributor

@calvn calvn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small comments, otherwise 👍!

physical/spanner/spanner.go Outdated Show resolved Hide resolved
physical/spanner/spanner.go Outdated Show resolved Hide resolved
@sethvargo
Copy link
Contributor Author

@calvn updated. PTAL

sethvargo and others added 3 commits July 9, 2020 14:47
@sethvargo
physical/spanner: use separate client for updating locks
76f6475 
We believe this mitigates an issue where a large influx of requests
cause the leader to be unable to update the lock table (since it cannot
grab a client from the pool or the client has no more open connections),
which causes cascading failure.
@sethvargo
Review feedback
23e083a
@calvn
Merge branch 'master' into sethvargo/google_locks
551a52a
@calvn calvn merged commit 29d1f21 into hashicorp:master Jul 9, 2020
calvn pushed a commit that referenced this pull request Jul 9, 2020
@sethvargo @calvn
physical/spanner: use separate client for updating locks (#9423)
d19f9f7 
* physical/spanner: use separate client for updating locks

We believe this mitigates an issue where a large influx of requests
cause the leader to be unable to update the lock table (since it cannot
grab a client from the pool or the client has no more open connections),
which causes cascading failure.
calvn added a commit that referenced this pull request Jul 9, 2020
@calvn @sethvargo
physical/spanner: use separate client for updating locks (#9423) (#9440)
cd69a30 
* physical/spanner: use separate client for updating locks

We believe this mitigates an issue where a large influx of requests
cause the leader to be unable to update the lock table (since it cannot
grab a client from the pool or the client has no more open connections),
which causes cascading failure.

Co-authored-by: Seth Vargo <seth@sethvargo.com>
andaley pushed a commit that referenced this pull request Jul 17, 2020
@sethvargo @andaley
physical/spanner: use separate client for updating locks (#9423)
fd8d839 
* physical/spanner: use separate client for updating locks

We believe this mitigates an issue where a large influx of requests
cause the leader to be unable to update the lock table (since it cannot
grab a client from the pool or the client has no more open connections),
which causes cascading failure.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@calvn calvn calvn approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sethvargo @calvn