Inconsistency in metrics' naming

[mlowicki]

Some metrics have hostname (which is 10d74e37a755 is above case) as a part of their names but some don't:

[2016-09-16 07:46:40 +0000 UTC][G] 'vault.10d74e37a755.runtime.free_count': 13329169.000
[2016-09-16 07:46:40 +0000 UTC][G] 'vault.10d74e37a755.runtime.heap_objects': 23287.000
[2016-09-16 07:46:40 +0000 UTC][G] 'vault.10d74e37a755.runtime.total_gc_pause_ns': 727764608.000
[2016-09-16 07:46:40 +0000 UTC][G] 'vault.10d74e37a755.expire.num_leases': 0.000
[2016-09-16 07:46:40 +0000 UTC][G] 'vault.10d74e37a755.runtime.malloc_count': 13352456.000
[2016-09-16 07:46:40 +0000 UTC][G] 'vault.10d74e37a755.runtime.alloc_bytes': 7225488.000
[2016-09-16 07:46:40 +0000 UTC][G] 'vault.10d74e37a755.runtime.sys_bytes': 26097912.000
[2016-09-16 07:46:40 +0000 UTC][G] 'vault.10d74e37a755.runtime.total_gc_runs': 1261.000
[2016-09-16 07:46:40 +0000 UTC][G] 'vault.10d74e37a755.runtime.num_goroutines': 18.000
[2016-09-16 07:46:50 +0000 UTC][G] 'vault.10d74e37a755.runtime.alloc_bytes': 7262992.000
[2016-09-16 07:46:50 +0000 UTC][G] 'vault.10d74e37a755.runtime.malloc_count': 13353345.000
[2016-09-16 07:46:50 +0000 UTC][G] 'vault.10d74e37a755.expire.num_leases': 0.000
[2016-09-16 07:46:50 +0000 UTC][G] 'vault.10d74e37a755.runtime.num_goroutines': 18.000
[2016-09-16 07:46:50 +0000 UTC][G] 'vault.10d74e37a755.runtime.free_count': 13329233.000
[2016-09-16 07:46:50 +0000 UTC][G] 'vault.10d74e37a755.runtime.heap_objects': 24112.000
[2016-09-16 07:46:50 +0000 UTC][G] 'vault.10d74e37a755.runtime.total_gc_pause_ns': 727764608.000
[2016-09-16 07:46:50 +0000 UTC][G] 'vault.10d74e37a755.runtime.total_gc_runs': 1261.000
[2016-09-16 07:46:50 +0000 UTC][G] 'vault.10d74e37a755.runtime.sys_bytes': 26097912.000
[2016-09-16 07:47:00 +0000 UTC][G] 'vault.10d74e37a755.runtime.total_gc_runs': 1261.000
[2016-09-16 07:47:00 +0000 UTC][G] 'vault.10d74e37a755.runtime.alloc_bytes': 7300576.000
[2016-09-16 07:47:00 +0000 UTC][G] 'vault.10d74e37a755.runtime.free_count': 13329298.000
[2016-09-16 07:47:00 +0000 UTC][G] 'vault.10d74e37a755.runtime.heap_objects': 24935.000
[2016-09-16 07:47:00 +0000 UTC][G] 'vault.10d74e37a755.runtime.total_gc_pause_ns': 727764608.000
[2016-09-16 07:47:00 +0000 UTC][G] 'vault.10d74e37a755.runtime.num_goroutines': 18.000
[2016-09-16 07:47:00 +0000 UTC][G] 'vault.10d74e37a755.runtime.sys_bytes': 26097912.000
[2016-09-16 07:47:00 +0000 UTC][G] 'vault.10d74e37a755.runtime.malloc_count': 13354233.000
[2016-09-16 07:47:00 +0000 UTC][G] 'vault.10d74e37a755.expire.num_leases': 0.000
[2016-09-16 07:47:10 +0000 UTC][G] 'vault.10d74e37a755.runtime.alloc_bytes': 7343944.000
[2016-09-16 07:47:10 +0000 UTC][G] 'vault.10d74e37a755.runtime.sys_bytes': 26097912.000
[2016-09-16 07:47:10 +0000 UTC][G] 'vault.10d74e37a755.runtime.malloc_count': 13355251.000
[2016-09-16 07:47:10 +0000 UTC][G] 'vault.10d74e37a755.runtime.free_count': 13329373.000
[2016-09-16 07:47:10 +0000 UTC][G] 'vault.10d74e37a755.runtime.total_gc_pause_ns': 727764608.000
[2016-09-16 07:47:10 +0000 UTC][G] 'vault.10d74e37a755.runtime.total_gc_runs': 1261.000
[2016-09-16 07:47:10 +0000 UTC][G] 'vault.10d74e37a755.runtime.num_goroutines': 18.000
[2016-09-16 07:47:10 +0000 UTC][G] 'vault.10d74e37a755.expire.num_leases': 0.000
[2016-09-16 07:47:10 +0000 UTC][G] 'vault.10d74e37a755.runtime.heap_objects': 25878.000
[2016-09-16 07:47:10 +0000 UTC][S] 'vault.route.rollback.cubbyhole-': Count: 1 Sum: 0.017 LastUpdated: 2016-09-16 07:47:18.177427883 +0000 UTC
[2016-09-16 07:47:10 +0000 UTC][S] 'vault.rollback.attempt.cubbyhole-': Count: 1 Sum: 0.112 LastUpdated: 2016-09-16 07:47:18.177443381 +0000 UTC
[2016-09-16 07:47:10 +0000 UTC][S] 'vault.route.rollback.auth-token-': Count: 1 Sum: 0.033 LastUpdated: 2016-09-16 07:47:18.177450095 +0000 UTC
[2016-09-16 07:47:10 +0000 UTC][S] 'vault.rollback.attempt.auth-token-': Count: 1 Sum: 0.157 LastUpdated: 2016-09-16 07:47:18.177459046 +0000 UTC
[2016-09-16 07:47:10 +0000 UTC][S] 'vault.route.rollback.secret-': Count: 1 Sum: 0.036 LastUpdated: 2016-09-16 07:47:18.177387184 +0000 UTC
[2016-09-16 07:47:10 +0000 UTC][S] 'vault.rollback.attempt.secret-': Count: 1 Sum: 0.092 LastUpdated: 2016-09-16 07:47:18.177397342 +0000 UTC
[2016-09-16 07:47:10 +0000 UTC][S] 'vault.route.rollback.sys-': Count: 1 Sum: 0.029 LastUpdated: 2016-09-16 07:47:18.177407881 +0000 UTC
[2016-09-16 07:47:10 +0000 UTC][S] 'vault.rollback.attempt.sys-': Count: 1 Sum: 0.074 LastUpdated: 2016-09-16 07:47:18.177416719 +0000 UTC
[2016-09-16 07:47:20 +0000 UTC][G] 'vault.10d74e37a755.runtime.num_goroutines': 18.000
[2016-09-16 07:47:20 +0000 UTC][G] 'vault.10d74e37a755.runtime.sys_bytes': 26097912.000
[2016-09-16 07:47:20 +0000 UTC][G] 'vault.10d74e37a755.runtime.heap_objects': 26713.000
[2016-09-16 07:47:20 +0000 UTC][G] 'vault.10d74e37a755.runtime.total_gc_pause_ns': 727764608.000
[2016-09-16 07:47:20 +0000 UTC][G] 'vault.10d74e37a755.expire.num_leases': 0.000
[2016-09-16 07:47:20 +0000 UTC][G] 'vault.10d74e37a755.runtime.alloc_bytes': 7382368.000
[2016-09-16 07:47:20 +0000 UTC][G] 'vault.10d74e37a755.runtime.malloc_count': 13356157.000
[2016-09-16 07:47:20 +0000 UTC][G] 'vault.10d74e37a755.runtime.free_count': 13329444.000
[2016-09-16 07:47:20 +0000 UTC][G] 'vault.10d74e37a755.runtime.total_gc_runs': 1261.000

/ # vault --version
Vault v0.6.1

My config is defined as follows:

{"listener": {"tcp": {"address": "0.0.0.0:8201", "tls_disable": 1}}, "backend": {"zookeeper": {"path": "/vault", "address": "m1.mesos.services.ams.osa:2181,m2.mesos.services.ams.osa:2181,m3.mesos.services.ams.osa:2181"}}, "telemetry": {"statsd_address": "nixy-internal-test-loadbalancer.marathon.mesos:10059"}}

[jefferai]

This appears to be an issue in go-metrics; EnableHostname is set in the Vault config unless you disable it. Can you report upstream?

[mlowicki]

Metrics without hostname part seems Vault-specific:

• 'vault.route.rollback.cubbyhole-'
• 'vault.rollback.attempt.cubbyhole-'

I can try to file a bug upstream but no idea how this library is used to prove it's really bug in go-metrics.

[mbjelac]

Same thing on 0.6.0... It appears gauges have IP while timers don't.