Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Indexes in s3 are not deleted #2944

Closed
shokada opened this issue Nov 18, 2020 · 3 comments
Closed

Indexes in s3 are not deleted #2944

shokada opened this issue Nov 18, 2020 · 3 comments

Comments

@shokada
Copy link
Contributor

shokada commented Nov 18, 2020

Describe the bug
Indexes stored in s3 are not deleted with the following error output to the table manager log.

level=error ts=2020-11-17T07:15:19.846302Z caller=table_client.go:57 msg="unexpected directories in index_18577 folder, not touching them" directories=[index_18577/]
level=error ts=2020-11-17T07:15:19.867115Z caller=table_client.go:57 msg="unexpected directories in index_18578 folder, not touching them" directories=[index_18578/]
level=error ts=2020-11-17T07:15:19.8829Z   caller=table_client.go:57 msg="unexpected directories in index_18579 folder, not touching them" directories=[index_18579/]

To Reproduce
Probably occurs when the index store to s3.
My configuration is below. I am using s3 compatible storage.

schema_config:
    configs:
        - from: "2020-10-24"
          index:
            period: 24h
            prefix: index_
          object_store: s3
          schema: v11
          store: boltdb-shipper
storage_config:
    aws:
        bucketnames: loki
        endpoint: https://s3-compatible-storage.com
        s3: s3://*:*@https://s3-compatible-storage.com/loki
        s3forcepathstyle: false
    boltdb_shipper:
        active_index_directory: /loki/index
        cache_location: /loki/index_cache
        shared_store: s3
    index_queries_cache_config:
        memcached:
            batch_size: 100
            parallelism: 100
        memcached_client:
            consistent_hash: true
            host: memcached-index-queries.loki.svc.cluster.local
            service: memcached-client

Expected behavior
The table manager should delete indexes in s3.

Environment:

  • Infrastructure: [Kubernetes]
  • Deployment tool: [jsonnet]

What seems to be the cause
I think the value of the name variable needs to be changed from index_18577 to index_18577/.
https://github.com/grafana/loki/blob/v2.0.0/pkg/storage/stores/shipper/table_client.go#L51

Without the addition of /, output.Contents would be empty.
https://github.com/cortexproject/cortex/blob/v1.5.0/pkg/chunk/aws/s3_storage_client.go#L304-L314

@gcotone
Copy link

gcotone commented Nov 24, 2020

I'm also observing this, e.g:

msg="table has exceeded the retention period" table=loki_index_18585
msg="deleting table" table=loki_index_18585
msg="unexpected directories in loki_index_18585 folder, not touching them" directories=[loki_index_18585/]

@eatsoup
Copy link

eatsoup commented Nov 28, 2020

Should be fixed by: #2960
Thanks to @sandeepsukhani

@shokada
Copy link
Contributor Author

shokada commented Nov 30, 2020

@eatsoup Thank you for letting me know.
I have confirmed that indexes are deleted.

@shokada shokada closed this as completed Nov 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants