proposal: log: change Logger to be an interface

[srinathh]

Motivation

Go stdlib provides a log.Logger that is widely used in both Go programs and Go libraries (eg. most notably in http.Server) as a standardized logging component.

However, the current implementation of the log.Logger enforces a specific logging format with limited flexibility which requires text parsing to allow it to be connected to any other logging system implementation through the swappable io.Writer interface. This adds complexity and overhead. The other alternative of directly using third party logging libraries is not very scalable both because existing libraries including components like http.Server in the stdlib expect log.Logger and because of network effects of stdlib.

Proposal

This proposes a minor change to the log package to make log.Logger more extensible in a fully backwards compatible by allowing Logger.Output() to write logs using custom logging implementations.

We do this by defining a function type called OutputFn with the same function signature as the current Logger.Output() and introducing an unexported field outputfn in Logger to hold the function that should be used for output much like we use the out field to hold the io.Writer currently. By default, log.New() will set the outputfn to the current implementation which is moved to Logger.DefOutputFn(). This can be swapped to any other OutputFn implementation by calling Logger.SetOutputFn() in the same way as we currently can swap io.Writer by calling Logger.SetOutput()

For the top level log package, we provide log.SetOutputFn() and log.SetDefOutputFn() to set the OutputFn of the log.std logger and reset it to default.

Proof of Concept

I have implemented a proof of concept for this proposed change and added an example in the test suite that demonstrates logging to a Tab Separated Variable format. While in the example i am merely writing to a Writer, the Output could be to anything including over the network etc.

If this approach looks ok, I can package this into a CL incorporating any feedback.

Rationale for the Approach

Most of the actual work of logging happens in Logger.Output() (and in the unexported Logger.formatHeader() called by it) which makes it a prime candidate for flexibility. We cannot change log.Logger into an interface due to Go1 compatibility promise and this approach therefore allows a lot of flexibility with a very minor change.

[srinathh]

Could I request a review of the proposal? Should I submit a CL?

[adg]

I don't see the advantage that this provides over the default logger. Instead of parsing colons you parse tabs?

[srinathh]

No no.... the idea is to define a minimal function type mimicking the current Logger.Output() function that lets anyone use the standard library log package and log.Logger to do logging in any format and using any communication mechanism (eg. a logging to an API service exposed over the network) without any parsing.

This would allow for instance to pass a log.Logger to http.Server that can plug into any other logging service merely by implementing an OutputFn() and passing it to the Logger.

Purely as an example here, I have implemented a version of OutputFn() that logs into a Tab separated variable format file but the actual implementation can be anything at all.

[srinathh]

Hi - could I pls. request another evaluation of the proposal as I believe it may have been mis-understood originally. The objective is NOT to write logs with tabs formatting or whatever. The objective is to introduce flexibility in the standard Logger to allow alternative logging implementations (which could include writing to files, over network, databases etc.) to plug into it with a minimal interface while retaining full backwards compatibility.

[rsc]

Like @adg, I don't believe this proposal has enough benefit to warrant the complexity suggested for the API. The log package already has a way to divert output, namely SetOutput. Instead of adding a second, why not use the first?

Note that the log package guarantees to deliver each message in a different Write call. So if you really need to tweak the format, it's possible to do today with no changes to the standard library: tweak the log message itself. Quoting the docs:

A Logger represents an active logging object that generates lines of output
to an io.Writer. Each logging operation makes a single call to the Writer's
Write method. A Logger can be used simultaneously from multiple goroutines;
it guarantees to serialize access to the Writer.

The only possible benefit I can see is that the current output writer doesn't know the call depth on each Write, so it can't reconstruct the file and line number for itself. But if that's really important, it's easy to pull them from the written data.

[srinathh]

Thanks for reviewing Russ. Here is the benefit I see of enabling a more flexible logger output via an approach like OutputFn vs. using current SetOuptut

• Currently, doing any non-standard logging (either in format or in action) with the log package is a four step process using the SetOutput approach.
  1. Setup the standard logger to generate the fields required for your custom logging through SetFlags() and/or SetPrefix(). The standard flags may be insufficient.
  2. Implement a custom logger that supports the writer interface & call SetOutput() with it
  3. In the custom logger, parse each line of output from the writer to extract the required fields from the string
  4. Perform whatever is the custom logging required.
• Point 1 means the application code and the logger implementation get tightly coupled and you need to have knowledge of the input format that the custom logger implementation expects to use it. This is fragile and also depends on logging output format being guaranteed.
• Point 3 is an unnecessary extra step and possibly a performance penalty. Why should a custom logger contain code to parse a []byte and extract timestamps and file-paths and so on vs. focusing on the mechanics of the custom logging?
• The OutputFn type of approach lets logging implementations be swapped out cleanly without tying the implementation to the application code and also eliminates text parsing.

[rsc]

With the exception of file name, you can shorten to:

• SetFlags(0), SetPrefix("")
• call SetOutput(w)
• w.Write gets just the log message and can add whatever prefix or formatting it wants, without any parsing

If you want file names, then yes you have to use SetFlags(Llongfile) or Lshortfile and parse the file name. That's unfortunate, but it doesn't seem worth a whole new API. If we were starting over then yes I think we might reasonably make SetOutput take a func and have a separate SetWriter. But we're not starting over. We're working with what we have. If we created new API every time we realized we could have done something differently, we'd end up with a much larger (and arguably bloated) standard library. The bar is very high for additions.

[adg]

Deferring to Go2.

[robpike]

I think the right answer is to change Logger to be an interface, and to do that in Go 2.