Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[telegraf] Error parsing response: parsing time "null" #165

Open
zinuzoid opened this issue Dec 13, 2016 · 6 comments
Open

[telegraf] Error parsing response: parsing time "null" #165

zinuzoid opened this issue Dec 13, 2016 · 6 comments
Labels
bug

Comments

@zinuzoid
Copy link

Metric in Grafana seems doesn't work, so I check telegraf and there're some error logs from pods.

I'm running cluster in GKE k8s version 1.4.6
Images: quay.io/deis/telegraf:v2.7.0

2016-12-13T09:01:07.656026655Z Node Name set (gke-deis-default-pool-6ba9e8cc-v3kd)
2016-12-13T09:01:07.824924771Z Node IP set (10.240.0.19)
2016-12-13T09:01:07.824989798Z Creating topic with URL: http://10.103.253.156:4151/topic/create?topic=metrics
2016-12-13T09:01:07.836194361Z Setting KUBERNETES_URL: http://10.240.0.19:10255
2016-12-13T09:01:07.836252337Z Building config.toml!
2016-12-13T09:01:07.849382935Z Finished building toml...
2016-12-13T09:01:07.849422582Z ###########################################
2016-12-13T09:01:07.849431084Z ###########################################
2016-12-13T09:01:07.850343415Z # Set Tag Configuration
2016-12-13T09:01:07.850365550Z [tags]
2016-12-13T09:01:07.850373997Z # Set Agent Configuration
2016-12-13T09:01:07.850381146Z [agent]
2016-12-13T09:01:07.850387919Z   interval = "10s"
2016-12-13T09:01:07.850394825Z   round_interval = true
2016-12-13T09:01:07.850400548Z   metric_buffer_limit = 100000
2016-12-13T09:01:07.850406425Z   collection_jitter = "1s"
2016-12-13T09:01:07.850412938Z   flush_interval = "1s"
2016-12-13T09:01:07.850417002Z   flush_jitter = "0s"
2016-12-13T09:01:07.850420935Z   debug = true
2016-12-13T09:01:07.850424675Z   quiet = false
2016-12-13T09:01:07.850428540Z   flush_buffer_when_full = true
2016-12-13T09:01:07.850433117Z   hostname = "gke-deis-default-pool-6ba9e8cc-v3kd" 
2016-12-13T09:01:07.850437102Z # Set output configuration
2016-12-13T09:01:07.850441009Z [[outputs.influxdb]]
2016-12-13T09:01:07.850444630Z   urls = ["http://10.103.253.43:80"]
2016-12-13T09:01:07.850448397Z   database = "kubernetes"
2016-12-13T09:01:07.850455322Z   precision = "ns"
2016-12-13T09:01:07.850459065Z   timeout = "5s"
2016-12-13T09:01:07.850462690Z   
2016-12-13T09:01:07.850466214Z   
2016-12-13T09:01:07.850469755Z   
2016-12-13T09:01:07.850473390Z   
2016-12-13T09:01:07.850476925Z # Set Input Configuration
2016-12-13T09:01:07.850480562Z [[inputs.netstat]]
2016-12-13T09:01:07.850484788Z [[inputs.swap]]
2016-12-13T09:01:07.850490907Z [[inputs.system]]
2016-12-13T09:01:07.850496625Z [[inputs.mem]]
2016-12-13T09:01:07.850502727Z [[inputs.cpu]]
2016-12-13T09:01:07.850508447Z   percpu = true
2016-12-13T09:01:07.850513963Z   totalcpu = true
2016-12-13T09:01:07.850519322Z [[inputs.disk]]
2016-12-13T09:01:07.850524783Z   
2016-12-13T09:01:07.850530315Z [[inputs.diskio]]
2016-12-13T09:01:07.850536095Z   
2016-12-13T09:01:07.850553141Z   
2016-12-13T09:01:07.850559799Z [[inputs.net]]
2016-12-13T09:01:07.850566049Z   
2016-12-13T09:01:07.850571555Z [[inputs.influxdb]]
2016-12-13T09:01:07.850576800Z   urls = ["http://10.103.253.43:80/debug/vars"]
2016-12-13T09:01:07.850582587Z [[inputs.kubernetes]]
2016-12-13T09:01:07.850588418Z   url = "http://10.240.0.19:10255"
2016-12-13T09:01:07.850594063Z   bearer_token = "/var/run/secrets/kubernetes.io/serviceaccount/token"
2016-12-13T09:01:07.850600021Z   
2016-12-13T09:01:07.850605122Z   
2016-12-13T09:01:07.850610547Z   
2016-12-13T09:01:07.850615908Z   insecure_skip_verify = true
2016-12-13T09:01:07.850621341Z [[inputs.nsq]]
2016-12-13T09:01:07.850627237Z   endpoints = ["http://10.103.253.156:4151"]
2016-12-13T09:01:07.850632803Z   interval = "1s"
2016-12-13T09:01:07.850638463Z [[inputs.redis]]
2016-12-13T09:01:07.850643955Z   servers = ["tcp://:foo@10.103.243.126:6379"]
2016-12-13T09:01:07.850649618Z # Set Service Input Configuration
2016-12-13T09:01:07.850655265Z [[inputs.nsq_consumer]]
2016-12-13T09:01:07.850660967Z   server = "10.103.253.156:4150"
2016-12-13T09:01:07.850666620Z   topic = "metrics"
2016-12-13T09:01:07.850672023Z   channel = "consumer"
2016-12-13T09:01:07.850678743Z   max_in_flight = 100
2016-12-13T09:01:07.850684593Z   data_format = "influx"
2016-12-13T09:01:07.850692843Z ###########################################
2016-12-13T09:01:07.850700329Z ###########################################
2016-12-13T09:01:07.910949043Z 2016/12/13 09:01:07 INF    1 [metrics/consumer] (10.103.253.156:4150) connecting to nsqd
2016-12-13T09:01:10.587872549Z 2016/12/13 09:01:10 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:01:20.813784155Z 2016/12/13 09:01:20 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:01:30.488890095Z 2016/12/13 09:01:30 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:01:41.219745287Z 2016/12/13 09:01:41 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:01:50.722231932Z 2016/12/13 09:01:50 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:02:01.229486206Z 2016/12/13 09:02:01 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:02:11.007993912Z 2016/12/13 09:02:11 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:02:20.600314059Z 2016/12/13 09:02:20 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:02:30.652158165Z 2016/12/13 09:02:30 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:02:40.198493854Z 2016/12/13 09:02:40 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:02:50.223258238Z 2016/12/13 09:02:50 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:03:00.858215308Z 2016/12/13 09:03:00 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:03:10.705881319Z 2016/12/13 09:03:10 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:03:20.771301571Z 2016/12/13 09:03:20 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:03:30.437874621Z 2016/12/13 09:03:30 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:03:40.603927629Z 2016/12/13 09:03:40 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:03:51.060918864Z 2016/12/13 09:03:51 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:04:00.266918214Z 2016/12/13 09:04:00 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:04:10.435277890Z 2016/12/13 09:04:10 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:04:20.816313126Z 2016/12/13 09:04:20 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:04:30.839003676Z 2016/12/13 09:04:30 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]

Is there a way to dig it deeper. I did try to add AGENT_DEBUG:true, AGENT_QUITE:false but doesn't help much from log.
@kmala kmala added the bug label Dec 13, 2016
@mboersma mboersma added this to the v2.11 milestone Jan 10, 2017
@vdice
Copy link
Member

vdice commented Jan 11, 2017

Hmmm, I'm not able to reproduce this issue w/ same telegraf image (also GKE, k8s server 1.4.7)

Is there a way to dig it deeper. I did try to add AGENT_DEBUG:true, AGENT_QUITE:false but doesn't help much from log.

@jchauncey any suggestions on how to debug?

@jchauncey
Copy link
Member

jchauncey commented Jan 11, 2017 via email

@zinuzoid
Copy link
Author

@jchauncey yes, it's still happen.

2017-01-12T11:00:40.151027731Z 2017/01/12 11:00:40 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2017-01-12T11:00:50.990826710Z 2017/01/12 11:00:50 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]

@vdice vdice removed this from the v2.11 milestone Jan 30, 2017
@stuszynski
Copy link

@zinuzoid I encounter a similar issue. Telegraf is using url = "http://10.240.0.19:10255" for connecting to kubelet, but your kubelet is most likely listen on HTTPS only.

@stuszynski
Copy link

stuszynski commented Feb 7, 2017 •
edited
Loading

Sorry, my bad. The http://<node-ip>:10255 refer to the unsecured read-only port available on each kubelet.

     --read-only-port int32                                    The read-only port for the Kubelet to serve on with no authentication/authorization, and for localhost healthz endpoint (set to 0 to disable) (default 10255)

I check it manually and I'm sure it provide data on this URL from a telegraf container. We still, however, experience above issue.

@vdice vdice removed their assignment Feb 10, 2017
@Cryptophobia Cryptophobia mentioned this issue Mar 20, 2018
Open
@Cryptophobia
Copy link

This issue was moved to teamhephy/monitor#9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@mboersma @jchauncey @zinuzoid @stuszynski @vdice @kmala @Cryptophobia