Prometheus Input Plugin: Ability to gather metrics from all Kubernetes pods behind a service #3236
Conversation
| @@ -153,6 +210,9 @@ func (p *Prometheus) gatherURL(url string, acc telegraf.Accumulator) error { | |||
| for _, metric := range metrics { | |||
| tags := metric.Tags() | |||
| tags["url"] = url | |||
| if p.AddHostTag { | |||
| tags["host"] = req.Host | |||
| } | |||
There was a problem hiding this comment.
Use a different value for the tagkey, since Telegraf's default config adds the host tag to all points as the hostname Telegraf is running on. I propose calling it address. We should also rename the config option to go along with this.
.gitignore
Outdated
| @@ -5,3 +5,4 @@ tivan | |||
| .idea | |||
| *~ | |||
| *# | |||
| *.swp | |||
There was a problem hiding this comment.
Don't add this here, since *.swp are not created by the project but your text editor. (I should actually remove most of this file) Here is how text editor files should be handle IMO:
$ cat ~/.gitconfig
... snip ...
[core]
excludesfile = /home/dbn/.gitignore
... snip ...
$ cat ~/.gitignore
tags
*.swp
*.pyc
.noseids
There was a problem hiding this comment.
Awesome, did not know that, makes perfect sense!
| @@ -63,6 +79,28 @@ func (p *Prometheus) Description() string { | |||
|
|
|||
| var ErrProtocolError = errors.New("prometheus protocol error") | |||
|
|
|||
| func (p *Prometheus) SplitHostAndPort(host string) (string, string) { | |||
There was a problem hiding this comment.
You can use URL.Hostname() and URL.Port() for this.
| return host, hostPort | ||
| } | ||
|
|
||
| func (p *Prometheus) AddressToURL(u *url.URL, address string) string { |
There was a problem hiding this comment.
This can be done with URL and its String() function.
plugins/inputs/prometheus/README.md
Outdated
|
|
||
| # this will look up all the IP addresses behind the host and | ||
| # query every one of them, using the same schema, port and path | ||
| do_dns_lookup = true |
There was a problem hiding this comment.
I think we should have a separate list of records for dns lookup instead of a boolean, it could be called kubernetes_services since I believe this method is slightly different from standard DNS-SD. I think it would make sense to have these items automatically have the "host" tag.
This will allow combination setups more easily and also I am trying to use less boolean options since they tend to get confusing if you have too many of them.
|
Don't forget to run |
A Kubernetes service can represent an interface to one or many pods that would serve the functionality needed (for instance, a web API). Unfortunately, you cannot force communication with any pod in particular and thus, would run into problems collecting Prometheus metrics from all the pods behind a service.
Kubernetes has the ability to create headless services which allow more direct access to the pods behind it without knowing too much about the pods beforehand.
I added functionality to the Prometheus input plugin that allows doing a DNS query on the URLs provided and then querying metrics from all of the associated hosts. Thus, you are able to collect all metrics from all pods behind a Kubernetes service.
Assume, you have the following headless service at
my-headless-service.my-namespace:Specifying this:
Will gather metrics from
https://10.1.2.11:8080/metricsandhttps://10.1.4.35:8080/metricsand will even tag them with a label calledaddress.Required for all PRs: