stdout support for file backend via logger

[tallpauley]

Here is a working request based on @jefferai's input on #2195. For this one I just took the file backend and forked behavior if file_path is STDOUT.

What I did:

• passed the logger into the audit backend
• made a special log level LevelRaw (-2) that will dynamically select formatRaw.
• fork audit file backend when file_path == "STDOUT", I first tried - like @jefferai suggested, but this behaved weirdly in the CLI-- even in quotes I think it must get interpreted as a flag, wasn't sure how to escape it properly. STDOUT is easier-- no escaping

Questions:

• Should it be it's own backend?
• Is there any better way to do raw logging? It seems like it would be cleaner to make a special logger just for raw logging, but then the semaphore wouldn't be shared w/ the main logger
• Can my amateurish go code be improved anywhere?
• Is -2 an ok log-level? Since you have to mount it as an audit backend, I think it shouldn't ever be silenced by less verbose log-levels.

Still to be done:

• I need to update the docs, just figured I'd wait to get feedback first

[tallpauley]

I think it might be actually a lot cleaner to just make it a separate audit backend, since it uses almost none of the file backend. What do you think @jefferai?

[jefferai]

Logging goes to stderr after startup, not stdout, so this isn't necessary.

[tallpauley]

@jefferai I'm confused then by your comments on #2195. What were you referring to when you said "I worry about interleaving issues here given that Vault also writes things to stdout at various points."? I guess I wrongly assumed you were concerned about interleaving between a stdout audit backend and vault's main logging.

[jefferai]

When Vault starts it dumps a bunch of output to stdout. Then the server logs go to stderr. However, making stdout a separate audit backend doesn't make any difference. You'd basically just need to ignore any line that isn't JSON.

[jefferai]

You don't need any of this -- just use os.Stdout as the writer (https://golang.org/pkg/os/#pkg-variables)

[tallpauley]

Right, that's what I did initially, but then I was reading your comments on #2195 and thought that output from the audit-to-stdout and the primary vault logger would need to be synchronized (thinking vault logged to stdout). What needs to be changed for in #2195 for it to be merged?

[jefferai]

2195 will never be merged because it's a separate backend.

[tallpauley]

@jefferai Ok, made it super simple, can you take another look? Reused fileLock to prevent interleaving between request and response logs.

[jefferai]

Feels like this should be sanitized to lower case since a lot of people might assume it should be valid.

[jefferai]

Other than that, looking good!

[tallpauley]

@jefferai Ok, lowercased!

[jefferai]

Same problem, in reverse. To be clear I meant using a sanitizing function, like strings.ToLower.

[tallpauley]

@jefferai Got it, I should have understood by the fact you used the word sanitized. It is now sanitized! I kept the docs to lowercase, feels more natural. Technically this is a breaking change for anyone who was outputting to a file named stdout. I think it should be reported as a breaking change even though outputting to a file named stdout is questionable.

[jefferai]

I wouldn't worry about it, we don't have defined behavior on relative paths currently. People should be using absolute paths!

[jefferai]

Looks good!

[jefferai]

Merged, thanks!

[lyndon160]

This is great.

Is this documented anywhere in the vault documentation? It feel like we need a note on this page https://www.vaultproject.io/docs/audit/index.html