-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
promtail: add multi-tenant support #1135
Merged
cyriltovena
merged 4 commits into
grafana:master
from
pracucci:add-multi-tenant-support-to-promtail
Nov 7, 2019
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
7ef1b53
promtail: added tenant_id support to promtail client
pracucci 4fbcb25
promtail: added tenant stage to dinamically override the tenant ID
pracucci 34e55d2
promtail: documented client's batch struct and improved tenant stage …
pracucci 83a2128
Added static value support to tenant stage
pracucci File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,80 @@ | ||
# `tenant` stage | ||
|
||
The tenant stage is an action stage that sets the tenant ID for the log entry | ||
picking it from a field in the extracted data map. If the field is missing, the | ||
default promtail client [`tenant_id`](../configuration.md#client_config) will | ||
be used. | ||
|
||
|
||
## Schema | ||
|
||
```yaml | ||
tenant: | ||
# Name from extracted data to whose value should be set as tenant ID. | ||
# Either source or value config option is required, but not both (they | ||
# are mutually exclusive). | ||
[ source: <string> ] | ||
|
||
# Value to use to set the tenant ID when this stage is executed. Useful | ||
# when this stage is included within a conditional pipeline with "match". | ||
[ value: <string> ] | ||
``` | ||
|
||
### Example: extract the tenant ID from a structured log | ||
|
||
For the given pipeline: | ||
|
||
```yaml | ||
pipeline_stages: | ||
- json: | ||
expressions: | ||
customer_id: customer_id | ||
- tenant: | ||
source: customer_id | ||
``` | ||
|
||
Given the following log line: | ||
|
||
```json | ||
{"customer_id":"1","log":"log message\n","stream":"stderr","time":"2019-04-30T02:12:41.8443515Z"} | ||
``` | ||
|
||
The first stage would extract `customer_id` into the extracted map with a value of | ||
`1`. The tenant stage would set the `X-Scope-OrgID` request header (used by Loki to | ||
identify the tenant) to the value of the `customer_id` extracted data, which is `1`. | ||
|
||
|
||
### Example: override the tenant ID with the configured value | ||
|
||
For the given pipeline: | ||
|
||
```yaml | ||
pipeline_stages: | ||
- json: | ||
expressions: | ||
app: | ||
message: | ||
- labels: | ||
app: | ||
- match: | ||
selector: '{app="api"}' | ||
stages: | ||
- tenant: | ||
value: "team-api" | ||
- output: | ||
source: message | ||
``` | ||
|
||
Given the following log line: | ||
|
||
```json | ||
{"app":"api","log":"log message\n","stream":"stderr","time":"2019-04-30T02:12:41.8443515Z"} | ||
``` | ||
|
||
The pipeline would: | ||
|
||
1. Decode the JSON log | ||
2. Set the label `app="api"` | ||
3. Process the `match` stage checking if the `{app="api"}` selector matches | ||
and - whenever it matches - run the sub stages. The `tenant` sub stage | ||
would override the tenant with the value `"team-api"`. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,106 @@ | ||
package stages | ||
|
||
import ( | ||
"reflect" | ||
"time" | ||
|
||
"github.com/go-kit/kit/log" | ||
"github.com/go-kit/kit/log/level" | ||
"github.com/grafana/loki/pkg/promtail/constants" | ||
"github.com/mitchellh/mapstructure" | ||
"github.com/pkg/errors" | ||
"github.com/prometheus/common/model" | ||
) | ||
|
||
const ( | ||
ErrTenantStageEmptySourceOrValue = "source or value config are required" | ||
ErrTenantStageConflictingSourceAndValue = "source and value are mutually exclusive: you should set source or value but not both" | ||
) | ||
|
||
type tenantStage struct { | ||
cfg TenantConfig | ||
logger log.Logger | ||
} | ||
|
||
type TenantConfig struct { | ||
Source string `mapstructure:"source"` | ||
Value string `mapstructure:"value"` | ||
} | ||
|
||
// validateTenantConfig validates the tenant stage configuration | ||
func validateTenantConfig(c TenantConfig) error { | ||
if c.Source == "" && c.Value == "" { | ||
return errors.New(ErrTenantStageEmptySourceOrValue) | ||
} | ||
|
||
if c.Source != "" && c.Value != "" { | ||
return errors.New(ErrTenantStageConflictingSourceAndValue) | ||
} | ||
|
||
return nil | ||
} | ||
|
||
// newTenantStage creates a new tenant stage to override the tenant ID from extracted data | ||
func newTenantStage(logger log.Logger, configs interface{}) (*tenantStage, error) { | ||
cfg := TenantConfig{} | ||
err := mapstructure.Decode(configs, &cfg) | ||
if err != nil { | ||
return nil, err | ||
} | ||
|
||
err = validateTenantConfig(cfg) | ||
if err != nil { | ||
return nil, err | ||
} | ||
|
||
return &tenantStage{ | ||
cfg: cfg, | ||
logger: logger, | ||
}, nil | ||
} | ||
|
||
// Process implements Stage | ||
func (s *tenantStage) Process(labels model.LabelSet, extracted map[string]interface{}, t *time.Time, entry *string) { | ||
var tenantID string | ||
|
||
// Get tenant ID from source or configured value | ||
if s.cfg.Source != "" { | ||
tenantID = s.getTenantFromSourceField(extracted) | ||
} else { | ||
tenantID = s.cfg.Value | ||
} | ||
|
||
// Skip an empty tenant ID (ie. failed to get the tenant from the source) | ||
if tenantID == "" { | ||
return | ||
} | ||
|
||
labels[constants.ReservedLabelTenantID] = model.LabelValue(tenantID) | ||
} | ||
|
||
// Name implements Stage | ||
func (s *tenantStage) Name() string { | ||
return StageTypeTenant | ||
} | ||
|
||
func (s *tenantStage) getTenantFromSourceField(extracted map[string]interface{}) string { | ||
// Get the tenant ID from the source data | ||
value, ok := extracted[s.cfg.Source] | ||
if !ok { | ||
if Debug { | ||
level.Debug(s.logger).Log("msg", "the tenant source does not exist in the extracted data", "source", s.cfg.Source) | ||
} | ||
return "" | ||
} | ||
|
||
// Convert the value to string | ||
tenantID, err := getString(value) | ||
if err != nil { | ||
if Debug { | ||
level.Debug(s.logger).Log("msg", "failed to convert value to string", "err", err, "type", reflect.TypeOf(value).String()) | ||
} | ||
return "" | ||
} | ||
|
||
return tenantID | ||
} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is convoluted example with constant tenant ID. If that is a useful use case, maybe tenant stage can support it directly?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This originates from a comment made by @cyriltovena, where he was suggesting to add this example too (at least this is what I understood). Cyril, is this example what you meant?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes because I think that's mainly how it will be used. I don't think someone has a tenant id in its log, or at least this is not the most common.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can the tenant stage support also a const tenant ID ? I think that's a valid use case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It does not, but it can. What do you think if we add a
value
field, which is mutually exclusive withsource
? For example:There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cyriltovena Done! What's your take?