dannykopping/autodeploy key #1
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Removing copyright notice per https://raintank-corp.slack.com/archives/CH34H23HD/p1643999482151029
* call mark phase started/finished only when applying retention * correctly pass userID to IntervalMayHaveExpiredChunks call on retention and dletion expiry check
The expected values were built from Cortex' `cortexpb.Sample` and `cortexpb.LabelPair`. They prove that `logproto.LegacyLabelPair` and `logproto.LegacySample` are not compatible with `logproto.LabelPair` and `logproto.Sample`, requiring us to keep both implementations for now.
**What this PR does / why we need it**: This patch replace the underlying heap of the `EntrySortIterator`. Instead of using a min or max heap the entry iterators are sorted at the beginning. Once the iteration starts only the first entry iterator is re-positioned depending on its current value. This will yield the same runtime complexity as the current heap based solution. However, since the new algorithm accesses the first element it can be cached. Thus the number of access calls to `Entry().Timestamp` are halved. ``` › benchstat before.txt after.txt name old time/op new time/op delta SortIterator/merge_sort-16 4.40ms ± 4% 4.38ms ± 3% ~ (p=0.400 n=9+10) SortIterator/sort-16 3.65ms ± 2% 2.80ms ± 3% -23.25% (p=0.000 n=9+10) ``` **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Add an entry in the `CHANGELOG.md` about the changes.
Current implementation of RoundToMilliseconds is broken due to improper handling of floating point operation in go. Fixing it by using simple division and modulo operators
* add a ms instead of ns to end time of the rebound chunk interval conversion between ms and ns loses ns precision which causes us to not include all the logs * improve error when failing to rebound chunk
* Ruler: Rule group not found API message We copied over the ruler code as part of #5089, and it seems that at the same time we stopped depending on the object storage implementation of Cortex. Turns out, there are (slight) differences between the clients which meant that response returned by the API when we try to get a rule group that is not found had changed. This ensures that is consistent with the error assertion that we have in the code for the ruler. * Use the method from the interfaace to determine object storage * also include DeleteObject * appease the linter
* Adds leader election process Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fluke Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes the kv typecheck * wire up the http client * Hooking into loki services, hit a bug * Add stats variable. * re-vendor dskit and improve to never fail service * Intrument Loki with the package * Add changelog entry Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes compactor test Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add configuration documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/usagestats/reporter.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> * Add boundary check Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add log for success report. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/usagestats/reporter.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>
* Improve report usage to never fail Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wait until cancelled for the service to not fail Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
* Fixes memberlist usage report Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the linter and improve comment Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
* documented custom k6 metrics that xk6-loki extension reports for instant and range queries * Update docs/sources/clients/k6/query-scenario.md Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * changed description * documented new metrics Co-authored-by: Christian Haudum <christian.haudum@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
* add a component key to our metrics.go line Signed-off-by: Edward Welch <edward.welch@grafana.com> * new computer... who dis Signed-off-by: Edward Welch <edward.welch@grafana.com> * one more lint Signed-off-by: Edward Welch <edward.welch@grafana.com>
NOTE: When client is tailing, Querier **has** to create grpc connections to **all** the ingesters. Sometimes when ingester is rolloing out and Querier unable to connect to ingester when client is tailing, then error message reported by Querier is confusing. This PR add some context to it. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
* Correct grouping type in pkg/logql/expr.y * Use loki-build-image to build loki instead of golang image * Include goyacc and other required tools to allow Makefile to run * Remove dependencies not relevant to yacc in cmd/loki/Dockerfile * Use go install instead of go get to install goyacc in cmd/loki/Dockerfile * Add goyacc to logcli and loki-canary Dockerfiles * Rerun CI * Rerun CI * Revert "Add goyacc to logcli and loki-canary Dockerfiles" This reverts commit 0ee887c. * Remove goyacc from cmd/loki/Dockerfile * Formatting of cmd/loki/Dockerfile
… spec is nil (#5383) * deploy gateway when both flag, specs provided * run in degraded when invalid config provided * return nil after applying degraded condition
* Add subqueries count Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
* moves encoding to own pkg * decwrap impl * forks tsdb index * renames tsdb_enc * wraps all tsdb decbufs * use loki encbuf * custom chunk meta replacing ref-id for uint32 checksum * handle overlapping chunks in tsdb encoding * tsdb builder impl * tsdb querier beginnings to use modified tsdb * fixes nonexistent reference * lint/remove unused * adds tsdb testware * lint
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
* Update dskit version Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * update changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add debug printing to see interfaces in CI. This will be reverted Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add comment to remember to remove Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use netutil.PrivateNetworkInterfacesWithFallback for the other rings Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Move import to place indicated by linter Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove debug printing Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Determine local interfaces in tests Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rearrange import to satify linter Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update build image in CircleCI Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Change changelegog message to show customer visible changes Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add doc tag and use private network interfaces in common config Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add comment about filtered out non private IP addresses Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update the documentation of the interface names settings Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
…ls from an environment variable value (#5359) * updated with ability to add extra tags * updated CHANGELOG.md * simplified extra label env variable to a simple comma seperated list in the form of "name1,value1,name2,value2,..." * refactored parsing of extra labels into its own method. Added unit tests. * removed invalid test, exit early on parseExtraLabels, refactored apply extra labels code to utility method, replaced hardcoded string with const global var in module. * updated docs, terraform template and cloud formation template * updated fmt.Println to fmt.Printf with line termination * updated to validate extra labels and added unit test for extra label validation behaviour * removed unused ExtraLabel struct * updated to use the new applyExtraLabels * pull latest * updated lambda promtail docs with instructions and example on how to use extra labels.
* set match_max_concurrent true by default Signed-off-by: Edward Welch <edward.welch@grafana.com> * add note to upgrade guide, fix up some of the existing notes Signed-off-by: Edward Welch <edward.welch@grafana.com>
* Add support for Prometheus like duration units in logql * Remove redundant type conversion * deduplicate call to sb.String Add a comment to explain why both promql and time's ParseDuration methods are used * extract duration parsing to separate function. * Add tests to ensure promql duration units are supported * Sort duration rune comment and list of runes in ascending order of length.
* run the make targets regardless of file timestamps and changes, just run them every time. Signed-off-by: Edward Welch <edward.welch@grafana.com> * attempt #2 Signed-off-by: Edward Welch <edward.welch@grafana.com> * don't need GO_FILES anymore Signed-off-by: Edward Welch <edward.welch@grafana.com> * remove all the mod=vendor flags required as we transitioned through go versions 13/14/15 Signed-off-by: Edward Welch <edward.welch@grafana.com> * changing drone deps because check-generated-files removes protos and I guess this all happens in the same place at the same time in drone. Signed-off-by: Edward Welch <edward.welch@grafana.com> * fix rebase Signed-off-by: Edward Welch <edward.welch@grafana.com>
* exports NewChunkIndexIterator * starts building tsdb-bench * adds query benchmark for tsdb prototyping * lint
…queriers/index-gateways (#5446) * add a metric for tracking the duration it took to sync the tables in queriers/index-gateways * changes suggested from PR review
This option can be used to add a set of labels to all alerts emitted by the ruler. Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
* builder chunks uses heap * idempotent index builder * remove early return * pr suggestions * more thorough equality check on ChunkMetas.finalize
…play Loki metrics for Memcache and Consul (#5462) Signed-off-by: Jordan Rushing <jordan.rushing@grafana.com>
…ryPages function from IndexClient interface (#5463)
…ir (#5454) * Add deprecated annotation to LegacySample and LegacyLabelPair Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Replace annotation with more generic docstring Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com>
* Fix two remote_timeout configs in ingester_client block * Add default value of remote_timeout in document
Index Gateway query performance improvements by bounding the number of concurrent grpc calls we make from the client.
* Fixes ingester sample deduping * Call get hash only once * Update pkg/iter/entry_iterator.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
* stream query response from boltdb index client instead of sending one row per batch * add benchmark for testing index gateway query performance * lint and fix broken test * wrap error message while failing to send back response * sort imports
…es (#5409) * promtail/targets/syslog: enable best effort parsing * promtail/targets/syslog: fmt
* add support for configuring query readiness for user index with limits overrides config * improve tracking of duration for query time downloading of index * update changelog * use the existing context.Context we have for background operations while ensuring query readiness
* Fix potential deadlock in the table manager * Ensure second lock is released on return * use t.TempDir instead of os.TempDir * fix broken test Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>
…modify the secret Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Pull Request Test Coverage Report for Build 1910810270
💛 - Coveralls |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
LegacySampleandLegacyLabelPaircompatibility. (Add tests that ensureLegacySampleandLegacyLabelPaircompatibility grafana/loki#5313)EntrySortIteratorby 20%. (Speed upEntrySortIteratorby 20%. grafana/loki#5318)make droneto use the new build image (Runmake droneto use the new build image grafana/loki#5368)