Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update promtail to support duration string formats #5290

Merged
merged 2 commits into from
Feb 3, 2022
Merged

Update promtail to support duration string formats #5290

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2923bcc
Update promtail to support duration string formats
ssncferreira Feb 1, 2022
626650d
Address comments
ssncferreira Feb 2, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
## Main

* [5302](https://github.com/grafana/loki/pull/5302) **MasslessParticle** Update azure blobstore client to use new sdk.
* [5243](https://github.com/grafana/loki/pull/5290) **ssncferreira**: Update Promtail to support duration string formats.
* [5266](https://github.com/grafana/loki/pull/5266) **jeschkies**: Write Promtail position file atomically on Unix.
* [5280](https://github.com/grafana/loki/pull/5280) **jeschkies**: Fix Docker target connection loss.
* [5243](https://github.com/grafana/loki/pull/5243) **owen-d**: moves `querier.split-queries-by-interval` to limits code only.
Expand Down
21 changes: 20 additions & 1 deletion clients/pkg/logentry/stages/metrics.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -311,7 +311,7 @@ func getFloat(unk interface{}) (float64, error) {
case uint:
return float64(i), nil
case string:
return strconv.ParseFloat(i, 64)
return getFloatFromString(i)
case bool:
if i {
return float64(1), nil
Expand All @@ -321,3 +321,22 @@ func getFloat(unk interface{}) (float64, error) {
return math.NaN(), fmt.Errorf("can't convert %v to float64", unk)
}
}

// getFloatFromString converts string into float64
// Two types of string formats are supported:
// * strings that represent floating point numbers, e.g., "0.804"
// * duration format strings, e.g., "0.5ms", "10h".
// Valid time units are "ns", "us", "ms", "s", "m", "h".
// Values in this format are converted as a floating point number of seconds.
// E.g., "0.5ms" is converted to 0.0005
func getFloatFromString(str string) (float64, error) {
dur, err := strconv.ParseFloat(str, 64)
if err != nil {
dur, err := time.ParseDuration(str)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dannykopping you suggested using ParseDuration from github.com/prometheus/common/model instead of the one from time package. They support different time units:

I would say that the one from time package has the most suitable set for this use case. Wdyt?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tend to agree; I think the smaller units are probably more valuable here. Good call

if err != nil {
return 0, err
}
return dur.Seconds(), nil
}
return dur, nil
}
37 changes: 19 additions & 18 deletions clients/pkg/logentry/stages/metrics_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -321,8 +321,9 @@ func TestMetricStage_Process(t *testing.T) {
"contains_false": "contains(keys(@),'nope')",
},
}
regexHTTPFixture := `11.11.11.11 - frank [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" 200 932ms"`
regexConfig := map[string]interface{}{
"expression": "(?P<get>\"GET).*HTTP/1.1\" (?P<status>\\d*) (?P<time>\\d*) ",
"expression": "(?P<get>\"GET).*HTTP/1.1\" (?P<status>\\d*) (?P<time>\\d*ms)",
}
timeSource := "time"
true := "true"
Expand Down Expand Up @@ -386,12 +387,12 @@ func TestMetricStage_Process(t *testing.T) {
Action: metric.CounterInc,
},
},
"response_time_ms": MetricConfig{
"response_time_seconds": MetricConfig{
MetricType: "Histogram",
Source: &timeSource,
Description: "response time in ms",
Config: metric.HistogramConfig{
Buckets: []float64{1, 2, 3},
Buckets: []float64{0.5, 1, 2},
},
},
}
Expand All @@ -410,7 +411,7 @@ func TestMetricStage_Process(t *testing.T) {
t.Fatalf("failed to create stage with metrics: %v", err)
}
out := processEntries(jsonStage, newEntry(nil, labelFoo, logFixture, time.Now()))
out[0].Line = regexLogFixture
out[0].Line = regexHTTPFixture
out = processEntries(regexStage, out...)
out = processEntries(metricStage, out...)
out[0].Labels = labelFu
Expand Down Expand Up @@ -473,20 +474,20 @@ promtail_custom_numeric_integer{baz="fu",fu="baz"} 123.0
# TYPE promtail_custom_numeric_string gauge
promtail_custom_numeric_string{bar="foo",foo="bar"} 123.0
promtail_custom_numeric_string{baz="fu",fu="baz"} 123.0
# HELP promtail_custom_response_time_ms response time in ms
# TYPE promtail_custom_response_time_ms histogram
promtail_custom_response_time_ms_bucket{bar="foo",foo="bar",le="1.0"} 0.0
promtail_custom_response_time_ms_bucket{bar="foo",foo="bar",le="2.0"} 0.0
promtail_custom_response_time_ms_bucket{bar="foo",foo="bar",le="3.0"} 0.0
promtail_custom_response_time_ms_bucket{bar="foo",foo="bar",le="+Inf"} 1.0
promtail_custom_response_time_ms_sum{bar="foo",foo="bar"} 932.0
promtail_custom_response_time_ms_count{bar="foo",foo="bar"} 1.0
promtail_custom_response_time_ms_bucket{baz="fu",fu="baz",le="1.0"} 0.0
promtail_custom_response_time_ms_bucket{baz="fu",fu="baz",le="2.0"} 0.0
promtail_custom_response_time_ms_bucket{baz="fu",fu="baz",le="3.0"} 0.0
promtail_custom_response_time_ms_bucket{baz="fu",fu="baz",le="+Inf"} 1.0
promtail_custom_response_time_ms_sum{baz="fu",fu="baz"} 932.0
promtail_custom_response_time_ms_count{baz="fu",fu="baz"} 1.0
# HELP promtail_custom_response_time_seconds response time in ms
# TYPE promtail_custom_response_time_seconds histogram
promtail_custom_response_time_seconds_bucket{bar="foo",foo="bar",le="0.5"} 0
promtail_custom_response_time_seconds_bucket{bar="foo",foo="bar",le="1"} 1
promtail_custom_response_time_seconds_bucket{bar="foo",foo="bar",le="2"} 1
promtail_custom_response_time_seconds_bucket{bar="foo",foo="bar",le="+Inf"} 1
promtail_custom_response_time_seconds_sum{bar="foo",foo="bar"} 0.932
promtail_custom_response_time_seconds_count{bar="foo",foo="bar"} 1
promtail_custom_response_time_seconds_bucket{baz="fu",fu="baz",le="0.5"} 0
promtail_custom_response_time_seconds_bucket{baz="fu",fu="baz",le="1"} 1
promtail_custom_response_time_seconds_bucket{baz="fu",fu="baz",le="2"} 1
promtail_custom_response_time_seconds_bucket{baz="fu",fu="baz",le="+Inf"} 1
promtail_custom_response_time_seconds_sum{baz="fu",fu="baz"} 0.932
promtail_custom_response_time_seconds_count{baz="fu",fu="baz"} 1.0
# HELP promtail_custom_total_keys the total keys per doc
# TYPE promtail_custom_total_keys counter
promtail_custom_total_keys{bar="foo",foo="bar"} 8.0
Expand Down
13 changes: 13 additions & 0 deletions docs/sources/clients/promtail/stages/metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -274,3 +274,16 @@ number in the `retries` field from the extracted map.
This pipeline creates a histogram that reads `response_time` from the extracted
map and places it into a bucket, both increasing the count of the bucket and the
sum for that particular bucket.

## Supported values

The metric values extracted from the log data are internally converted to floating points.
The supported values are the following:
* integers, floating point numbers
* string - two types of string formats are supported:
* strings that represent floating point numbers: e.g., `"0.804"` is converted to `0.804`.
* duration format strings. Valid time units are "ns", "us", "ms", "s", "m", "h".
Values in this format are converted as a floating point number of seconds. E.g., `"0.5ms"` is converted to `0.0005`.
* boolean:
* `true` is converted to `1`
* `false` is converted to `0`