Logql stdin support only log queries (#4606)

* Hack stdin client into LogCLI

Add flag to choose what kind of client to make request to.

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

* Label filter working

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

* Demo checkpoint1

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

* Metrics query working.

Issue was with default value of `step` (1 nanosecond).
StepEvaluator try to go through every nanosecond to apply aggregate on sample data
during the metric queries.

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

* Basic tests for fileClient

* Tests for logqueries direction

* Use HeapIterator for Entries.

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

* Remove some debug statements

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

* Remove support for metric queries. Stick with only log queries

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

* Small rough usage doc

* Remove filesampleiterator

* Fix some typos and tests

* Fix breaking test cases

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

* Make linter happy

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

* PR remarks.

1. Add `--stdin` in examples of the usage
2. Add `--stdin` in all the command help output

* PR remarks
- Use parsed labels correctly
- Fix indendation with --stdin flag

* Fix issue with direction

* Fix linter

* MaxInt64 -> MaxInt (to support even arm32 images)

* Add note on calculating `step` value on the client side

cmd/logcli/main.go

@@ -2,9 +2,11 @@ package main
 
 import (
 	"log"
+	"math"
 	"net/url"
 	"os"
 	"runtime/pprof"
+	"strings"
 	"time"
 
 	"github.com/prometheus/common/config"
@@ -27,6 +29,7 @@ var (
 	timezone   = app.Flag("timezone", "Specify the timezone to use when formatting output timestamps [Local, UTC]").Default("Local").Short('z').Enum("Local", "UTC")
 	cpuProfile = app.Flag("cpuprofile", "Specify the location for writing a CPU profile.").Default("").String()
 	memProfile = app.Flag("memprofile", "Specify the location for writing a memory profile.").Default("").String()
+	stdin      = app.Flag("stdin", "Take input logs from stdin").Bool()
 
 	queryClient = newQueryClient(app)
 
@@ -138,6 +141,33 @@ func main() {
 		}()
 	}
 
+	if *stdin {
+		queryClient = client.NewFileClient(os.Stdin)
+		if rangeQuery.Step.Seconds() == 0 {
+			// Set default value for `step` based on `start` and `end`.
+			// In non-stdin case, this is set on Loki server side.
+			// If this is not set, then `step` will have default value of 1 nanosecond and `STepEvaluator` will go through every nanosecond when applying aggregation during metric queries.
+			rangeQuery.Step = defaultQueryRangeStep(rangeQuery.Start, rangeQuery.End)
+		}
+
+		// When `--stdin` flag is set, stream selector is optional in the query.
+		// But logQL package throw parser error if stream selector is not provided.
+		// So we inject "dummy" stream selector if not provided by user already.
+		// Which brings down to two ways of using LogQL query under `--stdin`.
+		// 1. Query with stream selector(e.g: `{foo="bar"}|="error"`)
+		// 2. Query without stream selector (e.g: `|="error"`)
+
+		qs := rangeQuery.QueryString
+		if strings.HasPrefix(strings.TrimSpace(qs), "|") {
+			// inject the dummy stream selector
+			qs = `{source="logcli"}` + qs
+			rangeQuery.QueryString = qs
+		}
+
+		// `--limit` doesn't make sense when using `--stdin` flag.
+		rangeQuery.Limit = math.MaxInt // TODO(kavi): is it a good idea?
+	}
+
 	switch cmd {
 	case queryCmd.FullCommand():
 		location, err := time.LoadLocation(*timezone)
@@ -307,6 +337,7 @@ func newQuery(instant bool, cmd *kingpin.CmdClause) *query.Query {
 		cmd.Flag("step", "Query resolution step width, for metric queries. Evaluate the query at the specified step over the time range.").DurationVar(&q.Step)
 		cmd.Flag("interval", "Query interval, for log queries. Return entries at the specified interval, ignoring those between. **This parameter is experimental, please see Issue 1779**").DurationVar(&q.Interval)
 		cmd.Flag("batch", "Query batch size to use until 'limit' is reached").Default("1000").IntVar(&q.BatchSize)
+
 	}
 
 	cmd.Flag("forward", "Scan forwards through logs.").Default("false").BoolVar(&q.Forward)
@@ -333,3 +364,11 @@ func mustParse(t string, defaultTime time.Time) time.Time {
 
 	return ret
 }
+
+// This method is to duplicate the same logic of `step` value from `start` and `end`
+// done on the loki server side.
+// https://github.com/grafana/loki/blob/main/pkg/loghttp/params.go
+func defaultQueryRangeStep(start, end time.Time) time.Duration {
+	step := int(math.Max(math.Floor(end.Sub(start).Seconds()/250), 1))
+	return time.Duration(step) * time.Second
+}

docs/sources/getting-started/logcli.md

@@ -74,13 +74,13 @@ When not set, `--limit` defaults to 30.
 The limit protects the user from overwhelming the system
 for cases in which the specified query would have returned a large quantity
 of log lines.
-The limit also protects the user from unexpectedly large responses. 
+The limit also protects the user from unexpectedly large responses.
 
 The quantity of log line results that arrive in each batch
 is set by the `--batch` option in a `logcli query` command.
 When not set, `--batch` defaults to 1000.
 
-Setting a `--limit` value larger than the `--batch` value causes the 
+Setting a `--limit` value larger than the `--batch` value causes the
 requests from LogCLI to Loki to be batched.
 Loki has a server-side limit that defaults to 5000 for the maximum quantity
 of lines returned for a single query.
@@ -120,7 +120,8 @@ Flags:
                          timestamps [Local, UTC]
       --cpuprofile=""    Specify the location for writing a CPU profile.
       --memprofile=""    Specify the location for writing a memory profile.
-      --addr="http://localhost:3100"  
+      --stdin            Take input logs from stdin
+      --addr="http://localhost:3100"
                          Server address. Can also be set using LOKI_ADDR env
                          var.
       --username=""      Username for HTTP basic auth. Can also be set using
@@ -275,7 +276,8 @@ Flags:
                            timestamps [Local, UTC]
       --cpuprofile=""      Specify the location for writing a CPU profile.
       --memprofile=""      Specify the location for writing a memory profile.
-      --addr="http://localhost:3100"  
+      --stdin              Take input logs from stdin
+      --addr="http://localhost:3100"
                            Server address. Can also be set using LOKI_ADDR env
                            var.
       --username=""        Username for HTTP basic auth. Can also be set using
@@ -308,9 +310,9 @@ Flags:
       --batch=1000         Query batch size to use until 'limit' is reached
       --forward            Scan forwards through logs.
       --no-labels          Do not print any labels
-      --exclude-label=EXCLUDE-LABEL ...  
+      --exclude-label=EXCLUDE-LABEL ...
                            Exclude labels given the provided key during output.
-      --include-label=INCLUDE-LABEL ...  
+      --include-label=INCLUDE-LABEL ...
                            Include labels given the provided key during output.
       --labels-length=0    Set a fixed padding to labels
       --store-config=""    Execute the current query using a configured storage
@@ -346,7 +348,8 @@ Flags:
                          timestamps [Local, UTC]
       --cpuprofile=""    Specify the location for writing a CPU profile.
       --memprofile=""    Specify the location for writing a memory profile.
-      --addr="http://localhost:3100"  
+      --stdin            Take input logs from stdin
+      --addr="http://localhost:3100"
                          Server address. Can also be set using LOKI_ADDR env
                          var.
       --username=""      Username for HTTP basic auth. Can also be set using
@@ -402,7 +405,8 @@ Flags:
                          timestamps [Local, UTC]
       --cpuprofile=""    Specify the location for writing a CPU profile.
       --memprofile=""    Specify the location for writing a memory profile.
-      --addr="http://localhost:3100"  
+      --stdin            Take input logs from stdin
+      --addr="http://localhost:3100"
                          Server address. Can also be set using LOKI_ADDR env
                          var.
       --username=""      Username for HTTP basic auth. Can also be set using
@@ -430,3 +434,27 @@ Flags:
 Args:
   <matcher>  eg '{foo="bar",baz=~".*blip"}'
 ```
+
+### LogCLI `--stdin` usage
+
+You can consume log lines from your `stdin` instead of Loki servers.
+
+Say you have log files in your local, and just want to do run some LogQL queries for that, `--stdin` flag can help.
+
+**NOTE: Currently it doesn't support any type of metric queries**
+
+You may have to use `stdin` flag for several reasons
+1. Quick way to check and validate a LogQL expressions.
+2. Learn basics of LogQL with just Log files and `LogCLI`tool ( without needing set up Loki servers, Grafana etc.)
+3. Easy discussion on public forums. Like Q&A, Share the LogQL expressions.
+
+**NOTES on Usage**
+1. `--limits` flag doesn't have any meaning when using `--stdin` (use pager like `less` for that)
+1. Be aware there are no **labels** when using `--stdin`
+   - So stream selector in the query is optional e.g just `|="timeout"|logfmt|level="error"` is same as `{foo="bar"}|="timeout|logfmt|level="error"`
+
+**Examples**
+1. Line filter - `cat mylog.log | logcli --stdin query '|="too many open connections"'`
+2. Label matcher - `echo 'msg="timeout happened" level="warning"' | logcli --stdin query '|logfmt|level="warning"'`
+3. Different parsers (logfmt, json, pattern, regexp) - `cat mylog.log | logcli --stdin query '|pattern <ip> - - <_> "<method> <uri> <_>" <status> <size> <_> "<agent>" <_>'`
+4. Line formatters - `cat mylog.log | logcli --stdin query '|logfmt|line_format "{{.query}} {{.duration}}"'`