Shuffle-sharding support in Ruler. (grafana#3235)

* Reconfigure ruler in docker compose.

Two rulers are started now, they use S3 for storing rules, and use
sharding.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Extracted different sharding strategies into methods.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Added support for shuffle sharding to ruler.go

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Sharding tests.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Add tests for shuffle sharding strategy.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Removed configstore.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Return error if "common prefix" without supplied prefix is returned.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Changed replication strategy used by ruler.

Ruler must be active to receive any rules now.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Another test.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Load rules for users concurrently when using shuffle sharding.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Removed extra method LoadRuleGroupsForUser, and replaced with LoadRuleGroupsForUserAndNamespace.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Comment.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Fix comments after rename.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Lint comments.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Added ruler sync on ring change.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Move ShuffleShardSeed to util package.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Added log message.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Fix compilation of tests.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Create user-manager even if files even haven't changed.

During resharding, user can easily "jump" between rulers.
When it "jumps" back, previously manager wouldn't be second time.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Unregister user registry when removing ruler manager.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Added short paragraph about ruler shuffle sharding.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Review feedback.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Modified test, it now verifies expected rules for each ruler.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Added test for bugfix in DefaultTenantManagerFactory.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Added CHANGELOG.md

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Added test for Collect and DeleteUserRegistry.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Clean white noise.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Move comment about replication factor.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Added comment about tokens setup.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Fix compilation problem after master merge.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

storage_client.go

@@ -58,7 +58,7 @@ type ReadBatchIterator interface {
 	Value() []byte
 }
 
-// ObjectClient is used to store arbitrary data in Object Store (S3/GCS/Azure/Etc)
+// ObjectClient is used to store arbitrary data in Object Store (S3/GCS/Azure/...)
 type ObjectClient interface {
 	PutObject(ctx context.Context, objectKey string, object io.ReadSeeker) error
 	GetObject(ctx context.Context, objectKey string) (io.ReadCloser, error)
@@ -85,4 +85,5 @@ type StorageObject struct {
 }
 
 // StorageCommonPrefix represents a common prefix aka a synthetic directory in Object Store.
+// It is guaranteed to always end with delimiter passed to List method.
 type StorageCommonPrefix string