Retention doc

docs/sources/configuration/_index.md

@@ -1631,8 +1631,17 @@ compacts index shards to more performant forms.
 # Prefix should never start with a separator but should always end with it.
 [shared_store_key_prefix: <string> | default = "index/"]
 
-# Interval at which to re-run the compaction operation.
-[compaction_interval: <duration> | default = 2h]
+# Interval at which to re-run the compaction operation (or retention if enabled).
+[compaction_interval: <duration> | default = 10m]
+
+# (Experimental) Activate custom (per-stream,per-tenant) retention.
+[retention_enabled: <bool> | default = false]
+
+# Delay after which chunks will be fully deleted during retention.
+[retention_delete_delay: <duration> | default = 2h]
+
+# The total amount of worker to use to delete chunks.
+[retention_delete_worker_count: <int> | default = 150]
 ```
 
 ## limits_config
@@ -1753,6 +1762,24 @@ logs in Loki.
 # CLI flag: -ruler.max-rule-groups-per-tenant
 [ruler_max_rule_groups_per_tenant: <int> | default = 0]
 
+# Retention to apply for the store, if the retention is enable on the compactor side.
+# CLI flag: -store.retention
+[retention_period: <duration> | default = 744h]
+
+# Per-stream retention to apply, if the retention is enable on the compactor side.
+# Example:
+# retention_stream:
+# - selector: '{namespace="dev"}'
+#   priority: 1
+#   period: 24h
+# - selector: '{container="nginx"}'
+#   priority: 1
+#   period: 744h
+# Selector is a Prometheus labels matchers that will apply the `period` retention only if
+# the stream is matching. In case multiple stream are matching, the highest priority will be picked.
+# If no rule is matched the `retention_period` is used.
+[retention_stream: <array> | default = none]
+
 # Feature renamed to 'runtime configuration', flag deprecated in favor of -runtime-config.file (runtime_config.file in YAML).
 # CLI flag: -limits.per-user-override-config
 [per_tenant_override_config: <string>]

docs/sources/operations/storage/retention.md

@@ -3,7 +3,167 @@ title: Retention
 ---
 # Loki Storage Retention
 
-Retention in Loki is achieved through the [Table Manager](../table-manager/).
+Retention in Loki is achieved either through the [Table Manager](#table-manager) or the [Compactor](#Compactor).
+
+
+Retention through the [Table Manager](../table-manager/) is achieved by relying on the object store TTL feature, and will work for both [boltdb-shipper](../boltdb-shipper) store and chunk/index store. However retention through the [Compactor](../boltdb-shipper#compactor) is supported only with the [boltdb-shipper](../boltdb-shipper) store.
+
+The [Compactor](#Compactor) retention will become the default and have long term support. While this retention is still **experimental**, it supports more granular retention policies on per tenant and per stream use cases.
+
+## Compactor
+
+The [Compactor](../boltdb-shipper#compactor) can deduplicate index entries. It can also apply granular retention. When applying retention with the Compactor, the [Table Manager](../table-manager/) is unnecessary.
+
+> Run the compactor as a singleton (a single instance).
+
+Compaction and retention are idempotent. If the compactor restarts, it will continue from where it left off.
+
+The Compactor loops to apply compaction and retention at every `compaction_interval`, or as soon as possible if running behind.
+
+The compactor's algorithm to update the index:
+
+- For each table within each day:
+  - Compact the table into a single index file.
+  - Traverse the entire index. Use the tenant configuration to identify and mark chunks that need to be removed.
+  - Remove marked chunks from the index and save their reference in a file on disk.
+  - Upload the new modified index files.
+
+The retention algorithm is applied to the index. Chunks are not deleted while applying the retention algorithm. The chunks will be deleted by the compactor asynchronously when swept.
+
+Marked chunks will only  be deleted after `retention_delete_delay` configured is expired because:
+
+1. boltdb-shipper indexes are refreshed from the shared store on components using it (querier and ruler) at a specific interval. This means deleting chunks instantly could lead to components still having reference to old chunks and so they could fails to execute queries. Having a delay allows for components to refresh their store and so remove gracefully their reference of those chunks.
+
+2. It gives you a short period to cancel chunks deletion in case of mistakes.
+
+Marked chunks will only  be deleted after `retention_delete_delay` configured is expired because
+
+- boltdb-shipper indexes are refreshed from the shared store on components using them (querier and ruler) at a specific interval. Deleting chunks instantly could lead to components still having a reference to old chunks. This could, in turn, cause query execution failure. Having a delay allows components to refresh their store and gracefully remove their reference to those chunks.
+
+- It provides a short window of time in which to cancel chunk deletion in the case of a configuration mistake.
+
+Marker files (containing chunks to delete) should be stored on a persistent disk, since the disk will be the sole reference to them.
+
+### Retention Configuration
+
+This compactor configuration example activates retention.
+
+```yaml
+compactor:
+  working_directory: /data/retention
+  shared_store: gcs
+  compaction_interval: 10m
+  retention_enabled: true
+  retention_delete_delay: 2h
+  retention_delete_worker_count: 150
+schema_config:
+    configs:
+      - from: "2020-07-31"
+        index:
+            period: 24h
+            prefix: loki_index_
+        object_store: gcs
+        schema: v11
+        store: boltdb-shipper
+storage_config:
+    boltdb_shipper:
+        active_index_directory: /data/index
+        cache_location: /data/boltdb-cache
+        shared_store: gcs
+    gcs:
+        bucket_name: loki
+```
+
+> Note that retention is only available if the index period is 24h.
+
+Set `retention_enabled` to true. Without this, the Compactor will only compact tables.
+
+Define `schema_config` and `storage_config` to access the storage.
+
+The index period must be 24h.
+
+`working_directory` is the directory where marked chunks and temporary tables will be saved.
+
+`compaction_interval` dictates how often compaction and/or retention is applied. If the Compactor falls behind, compaction and/or retention occur as soon as possible.
+
+`retention_delete_delay` is the delay after which the compactor will delete marked chunks.
+
+`retention_delete_worker_count` specifies the maximum quantity of goroutine workers instantiated to delete chunks.
+
+#### Configuring the retention period
+
+Retention period is configured within the [`limits_config`](./../../../configuration/#limits_config) configuration section.
+
+There are two ways of setting retention policies:
+
+- `retention_period` which is applied globally.
+- `retention_stream` which is only applied to chunks matching the selector
+
+> The minimum retention period is 24h.
+
+This example configures global retention:
+
+```yaml
+...
+limits_config:
+  retention_period: 744h
+  retention_stream:
+  - selector: '{namespace="dev"}'
+    priority: 1
+    period: 24h
+  per_tenant_override_config: /etc/overrides.yaml
+...
+```
+
+Per tenant retention can be defined using the `/etc/overrides.yaml` files. For example:
+
+```yaml
+overrides:
+    "29":
+        retention_period: 168h
+        retention_stream:
+        - selector: '{namespace="prod"}'
+          priority: 2
+          period: 336h
+        - selector: '{container="loki"}'
+          priority: 1
+          period: 72h
+    "30":
+        retention_stream:
+        - selector: '{container="nginx"}'
+          priority: 1
+          period: 24h
+```
+
+A rule to apply is selected by choosing the first in this list that matches:
+
+1. If a per-tenant `retention_stream` matches the current stream, the highest priority is picked.
+2. If a global `retention_stream` matches the current stream, the highest priority is picked.
+3. If a per-tenant `retention_period` is specified, it will be applied.
+4. The global `retention_period` will be selected if nothing else matched.
+5. If no global `retention_period` is specified, the default value of `744h` (30days) retention is used.
+
+Stream matching uses the same syntax as Prometheus label matching:
+
+- `=`: Select labels that are exactly equal to the provided string.
+- `!=`: Select labels that are not equal to the provided string.
+- `=~`: Select labels that regex-match the provided string.
+- `!~`: Select labels that do not regex-match the provided string.
+
+The example configurations will set these rules:
+
+- All tenants except `29` and `30` in the `dev` namespace will have a retention period of `24h` hours.
+- All tenants except `29` and `30` that are not in the `dev` namespace will have the retention period of `744h`.
+- For tenant `29`:
+  - All streams except those in the container `loki` or in the namespace `prod` will have retention period of `168h` (1 week).
+  - All streams in the `prod` namespace will have a retention period of `336h` (2 weeks), even if the container label is `loki`, since the priority of the `prod` rule is higher.
+  - Streams that have the container label `loki` but are not in the namespace `prod` will have a `72h` retention period.
+- For tenant `30`:
+  - All streams except those having the container label `nginx` will have the global retention period of `744h`, since there is no override specified.
+  - Streams that have the label `nginx` will have a retention period of `24h`.
+
+## Table Manager
+
 In order to enable the retention support, the Table Manager needs to be
 configured to enable deletions and a retention period. Please refer to the
 [`table_manager_config`](../../../configuration#table_manager_config)