Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fluentd: Refactor label_keys and and add extract_kubernetes_labels configuration #1186

Merged
merged 2 commits into from
Oct 25, 2019
Merged

fluentd: Refactor label_keys and and add extract_kubernetes_labels configuration #1186

merged 2 commits into from
Oct 25, 2019

Conversation

tarokkk
Copy link
Contributor

@tarokkk tarokkk commented Oct 20, 2019

What this PR does / why we need it:

  • Refactoring label_keys to <label> block allows to use record_accessor to define your labels.
    It is a flexible way to extract record or nested record information
  • Refactor remove_keys to allow record_accessor syntax.
  • Add extract_kubernetes_labels to easily extract Kubernetes metadata as Loki labels.

Checklist

  • Documentation added
  • Tests updated

@CLAassistant
Copy link

CLAassistant commented Oct 20, 2019 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@tarokkk tarokkk mentioned this pull request Oct 20, 2019
Closed
@dawidmalina
Copy link
Contributor

This is very useful improvement. @cyriltovena, @briangann can you please take a look on this?

@sagikazarmark
Copy link

We submitted this PR as per the request of @tomwilkie . We would love to drop our version and use it from upstream. Can someone take a look at it please? cc @cyriltovena, @briangann (?)

cyriltovena
cyriltovena approved these changes Oct 25, 2019
View reviewed changes
Copy link
Contributor

@cyriltovena cyriltovena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I think next step would be to integrate this into our helm charts. A bit like fluent-bit ^^.

I'm also wondering if we should align fluent-bit configuration for labels with this. Currently it is using a labelmap.

@cyriltovena
Copy link
Contributor

Can you bump the gem version ?

@tarokkk
Copy link
Contributor Author

tarokkk commented Oct 25, 2019

Can you bump the gem version ?

I updated the spec.version = '1.1.0' am I missing something?

@cyriltovena
Copy link
Contributor

No all good I did not see it. I'll merge it soon.

@cyriltovena cyriltovena merged commit 51105f6 into grafana:master Oct 25, 2019
@takanabe takanabe mentioned this pull request Feb 5, 2020
Merged
1 task
takanabe added a commit to takanabe/loki that referenced this pull request Feb 5, 2020
@takanabe
Use original README.md for fluentd plugin explanations
b7a1b29 
`doc/client/fluentd.m` includes obsolete explanations and it confuses new users.
For example, label_keys option was removed at grafana#1186
but `fluentd.md` has the explanation.

It's simple to maitain `fluentd/fluent-plugin-grafana-loki/README.md` for
the explantion of the output plugin and use the link when we share how to use it.
cyriltovena pushed a commit that referenced this pull request Feb 7, 2020
@takanabe
Unite docs for fluentd plugin (#1634)
973b96d 
* Use original README.md for fluentd plugin explanations

`doc/client/fluentd.m` includes obsolete explanations and it confuses new users.
For example, label_keys option was removed at #1186
but `fluentd.md` has the explanation.

It's simple to maitain `fluentd/fluent-plugin-grafana-loki/README.md` for
the explantion of the output plugin and use the link when we share how to use it.

* Remove `label_keys` option and simplify usage for `drop_single_key`

* Fix environment variable: FLUENT_CONFIG -> FLUENTD_CONFIG

`docker-compose up` fails due to the wrong environment variable
specifying the fluentd config. As a result this causes the following error

```
docker-compose up --build
WARNING: The LOKI_USERNAME variable is not set. Defaulting to a blank
string.
WARNING: The LOKI_PASSWORD variable is not set. Defaulting to a blank
string.
Building fluentd
Step 1/17 : FROM fluent/fluentd:v1.3.2-debian
 ---> 4790aaf4d1e5
Step 2/17 : USER root
 ---> Using cache
 ---> 7b54da6cec9e
Step 3/17 : WORKDIR /home/fluent
 ---> Using cache
 ---> a161d2fcf64b
Step 4/17 : ENV PATH /fluentd/vendor/bundle/ruby/2.3.0/bin:$PATH
 ---> Using cache
 ---> b67529aad0b5
Step 5/17 : ENV GEM_PATH /fluentd/vendor/bundle/ruby/2.3.0
 ---> Using cache
 ---> 3a8b05beb185
Step 6/17 : ENV GEM_HOME /fluentd/vendor/bundle/ruby/2.3.0
 ---> Using cache
 ---> d68a94345cb4
Step 7/17 : ENV FLUENTD_DISABLE_BUNDLER_INJECTION 1
 ---> Using cache
 ---> 70f9bbf646e8
Step 8/17 : COPY docker/Gemfile* /fluentd/
 ---> Using cache
 ---> 5f8b2f6ffe1a
Step 9/17 : RUN buildDeps="sudo make gcc g++ libc-dev ruby-dev"
&& apt-get update        && apt-get install -y --no-install-recommends
$buildDeps libsystemd0 net-tools libjemalloc1        && gem install
bundler --version 1.16.2        && bundle config silence_root_warning
true        && bundle install --gemfile=/fluentd/Gemfile
--path=/fluentd/vendor/bundle        && sudo gem sources --clear-all
&& SUDO_FORCE_REMOVE=yes        apt-get purge -y --auto-remove        -o
APT::AutoRemove::RecommendsImportant=false        $buildDeps        &&
rm -rf /var/lib/apt/lists/*
/home/fluent/.gem/ruby/2.3.0/cache/*.gem        /tmp/* /var/tmp/*
/usr/lib/ruby/gems/*/cache/*.gem
 ---> Using cache
 ---> 36c960a53c2c
Step 10/17 : COPY docker/entrypoint.sh /fluentd/entrypoint.sh
 ---> Using cache
 ---> 7f8d72ae63ca
Step 11/17 : COPY lib/fluent/plugin/out_loki.rb
/fluentd/plugins/out_loki.rb
 ---> Using cache
 ---> d09473ee4f25
Step 12/17 : COPY docker/conf/ /fluentd/etc/loki/
 ---> Using cache
 ---> 00d0439ef3c8
Step 13/17 : ENV FLUENTD_CONF="/fluentd/etc/loki/fluentd.conf"
 ---> Using cache
 ---> 2d33a002baaf
Step 14/17 : ENV FLUENTD_OPT=""
 ---> Using cache
 ---> c314df064756
Step 15/17 : ENV LOKI_URL "https://logs-us-west1.grafana.net"
 ---> Using cache
 ---> df78c829d416
Step 16/17 : ENV LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so.1"
 ---> Using cache
 ---> 3d7d14cf37f5
Step 17/17 : ENTRYPOINT ["/fluentd/entrypoint.sh"]
 ---> Using cache
 ---> 82c70c2354ac
Successfully built 82c70c2354ac
Successfully tagged fluentd:loki
Starting docker_fluentd_1 ... done
Attaching to docker_fluentd_1
fluentd_1  |
/fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.6.2/lib/fluent/supervisor.rb:769:in
`initialize': No such file or directory @ rb_sysopen -
/fluentd/etc/loki/fluentd.conf (Errno::ENOENT)
fluentd_1  |    from
/fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.6.2/lib/fluent/supervisor.rb:769:in
`open'
fluentd_1  |    from
/fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.6.2/lib/fluent/supervisor.rb:769:in
`read_config'
fluentd_1  |    from
/fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.6.2/lib/fluent/supervisor.rb:479:in
`run_supervisor'
fluentd_1  |    from
/fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.6.2/lib/fluent/command/fluentd.rb:314:in
`<top (required)>'
fluentd_1  |    from
/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
fluentd_1  |    from
/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
fluentd_1  |    from
/fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.6.2/bin/fluentd:8:in
`<top (required)>'
fluentd_1  |    from /fluentd/vendor/bundle/ruby/2.3.0/bin/fluentd:22:in
`load'
fluentd_1  |    from /fluentd/vendor/bundle/ruby/2.3.0/bin/fluentd:22:in
`<main>'
docker_fluentd_1 exited with code 1
```

* Use docs/clients/fluentd/README.md for fluent-plugin-grafana-loki explanation

* Arrange list order based on number

* Revert "Fix environment variable: FLUENT_CONFIG -> FLUENTD_CONFIG"

This reverts commit 8ed58ac97a82d5536321fb9a76120fed766ae506.

* Elaborate the way to use docker image

* Fix grammar: remove unnecessary with

* Add disclaimer for multi-worker usage
@slim-bean slim-bean mentioned this pull request Jun 25, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cyriltovena cyriltovena cyriltovena approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@tarokkk @CLAassistant @dawidmalina @sagikazarmark @cyriltovena