JWT Authentication

[SpencerTorres]

Summary

closes #1445

This PR adds support for JWT authentication to Native/HTTP interfaces, and the database/sql connector.

To use JWT, provide a GetJWT implementation in the clickhouse.Options struct:

getJWT := func(ctx context.Context) (string, error) {
    return GetEnv("example jwt"), nil
}

opt := &clickhouse.Options{
    GetJWT: jwtFunc,
}

If this function is present in the config, the driver will ignore username/password in favor of the JWT.

Whenever a new TCP connection or HTTPS request starts, this function will be called. The JWT can be cached by the application if necessary, or generated each time.

For HTTPS connections, we can update the token immediately per request. For Native/TCP connections we have to wait for the connection to close or be forcibly closed by the server (which happens on token expiry). To force a Native/TCP connection to update, simply make a new DB instance.

If you want to use a specific JWT per query, you must be using HTTPS and not the Native/TCP interface. You can then use the following function to build a context with the given JWT:

ctx := clickhouse.Context(context.Background(), clickhouse.WithJWT("example jwt"))

Use this ctx in your query and it will use the provided JWT.

Other changes:

• I updated the HTTP request implementation to apply auth headers per-request instead of calculating them once on client initialization. This lets us update the JWT per-request, and makes the code a bit simpler. In the previous version, some functions were manually re-adding these headers which seems very backwards. The updated version cleans this up a bit.

Checklist

• Unit and integration tests covering the common scenarios were added

[SpencerTorres]

@mshustov let me know if there's any context or functionality missing for this. The native TCP test is failing due to i/o timeout, but the database/sql tests are passing for both TCP and HTTP which is odd. I need to find a better way to test this locally so I can fix it outside of GitHub Actions 👀

[jkaflik]

Let's see if we can change approach a little bit.

How is library dealing with authentication errors?

[SpencerTorres]

@jkaflik Updated to use a callback, also added context.Context override for HTTPS connections. This lets the user control the JWT via the function. They can decide whether to cache it or generate/fetch it each time. They can also swap it out between contexts to support multiple tokens over a single client instance.

Let me know what you think, if there's any concerns, or if something isn't named clearly (such as the primary GetJWT function).

Thanks!

[jkaflik]

This approach looks much better. Do you want to provide a README entry?

[SpencerTorres]

@jkaflik I can update README yes,

Re: the header changes, see the PR description for a note on this:

I updated the HTTP request implementation to apply auth headers per-request instead of calculating them once on client initialization. This lets us update the JWT per-request, and makes the code a bit simpler. In the previous version, some functions were manually re-adding these headers which seems very backwards. The updated version cleans this up a bit.

If you look at the diff you'll notice that h.headers is created when the http client is created. The only headers included on this are from clickhouse.Options, including authorization headers.

These external calls that reference h.headers were doing it backwards. They were creating a new map and then re-adding h.headers to it. What it should've been doing is simply providing per-request headers OR nil.

The best example of this is the conn_http_batch.go diff:

	headers := make(map[string]string)

	// . . .

	headers["Content-Type"] = "application/octet-stream"

	// This was removed. Why is it re-adding headers from the connection here?
	// Now the b.conn.sendStreamQuery will add the local variable "headers" instead of
	// the local variable adding the "b.conn.headers".
	for k, v := range b.conn.headers {
		headers[k] = v
	}

	// "headers" are only applied for this request.
	// The old "b.conn.headers" were only for authentication and other settings.
	// These are re-added per request internally on the HTTP conn instance.
	// It's no longer stored in "b.conn.headers"
	res, err := b.conn.sendStreamQuery(b.ctx, r, &options, headers)

And within httpConnect.createRequest we can re-add these headers:

func (h *httpConnect) createRequest(ctx context.Context, requestUrl string, reader io.Reader, options *QueryOptions, headers map[string]string) (*http.Request, error) {
	req, err := http.NewRequestWithContext(ctx, http.MethodPost, requestUrl, reader)
	if err != nil {
		return nil, err
	}

	// Apply options, such as auth / client-level HttpHeaders / etc.
	err = applyOptionsToRequest(ctx, req, h.opt)
	if err != nil {
		return nil, err
	}

	// per-request headers are added here, instead of the other way around
	for k, v := range headers {
		req.Header.Add(k, v)
	}

	// . . . 

Hopefully this makes sense, before making this change I looked closely at these files to make sure all of the headers were being appended correctly. This change is not only clearer in the code, but also required for swapping out auth/JWTs per request.

[jkaflik]

I just noticed this PR does not include happy path test coverage. Do you plan to introduce it?

[SpencerTorres]

I just noticed this PR does not include happy path test coverage. Do you plan to introduce it?

Yeah we'll expand on this PR later, just want to get the main functionality working first.

Some of the tests are mixed where they work and then intentionally break, I feel like this covers the realistic use of JWT where tokens aren't always valid, but I suppose it's fair to split these into distinct pass/fail logic