Reconciling loop breaks if the kubernetes-client perform a get.

[andreaTP]

Bug Report

Using the kubernetes-client and doing a GET directly in the reconcile loop cause this exception to be thrown.

java.lang.IllegalStateException: cannot make a new request because the previous response is still open: please call response.close()

What did you do?

Using a simple controller similar to:

@ControllerConfiguration(namespaces = Constants.WATCH_CURRENT_NAMESPACE, finalizerName = Constants.NO_FINALIZER)
public class KeycloakController implements Reconciler<Keycloak> {

    @Inject
    KubernetesClient client;

    @Override
    public UpdateControl<Keycloak> reconcile(Keycloak kc, Context context) {
          var deployment = client
                  .apps()
                  .deployments()
                  .inNamespace(kc.getMetadata().getNamespace())
                  .withName("something")
                  .get();
    }

    @Override
    public DeleteControl cleanup(Keycloak resource, Context context) {
        return Reconciler.super.cleanup(resource, context);
    }
}

What did you expect to see?

No exceptions to be thrown.

What did you see instead? Under which circumstances?

Always produces this exception:

java.lang.IllegalStateException: cannot make a new request because the previous response is still open: please call response.close()
	at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:164)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:66)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
	at okhttp3.RealCall.execute(RealCall.java:81)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl.send(OkHttpClientImpl.java:138)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.retryWithExponentialBackoff(OperationSupport.java:575)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:554)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:519)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:488)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:458)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:696)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:182)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:149)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:83)
	at org.keycloak.operator.KeycloakController.reconcile(KeycloakController.java:55)
	at org.keycloak.operator.KeycloakController.reconcile(KeycloakController.java:27)
	at org.keycloak.operator.KeycloakController_ClientProxy.reconcile(Unknown Source)
	at io.javaoperatorsdk.operator.processing.Controller$2.execute(Controller.java:95)
	at io.javaoperatorsdk.operator.processing.Controller$2.execute(Controller.java:70)
	at io.javaoperatorsdk.operator.api.monitoring.Metrics.timeControllerExecution(Metrics.java:34)
	at io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:69)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:141)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:109)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:74)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:43)
	at io.javaoperatorsdk.operator.processing.event.EventProcessor$ControllerExecution.run(EventProcessor.java:352)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

Environment

Kubernetes cluster type:

minikube

$ Mention java-operator-sdk version from pom.xml file

2.0.0-SNAPSHOT though the Quarkus extension version 3.0.0-SNAPSHOT

$ java -version

openjdk version "17.0.1" 2021-10-19
OpenJDK Runtime Environment Temurin-17.0.1+12 (build 17.0.1+12)
OpenJDK 64-Bit Server VM Temurin-17.0.1+12 (build 17.0.1+12, mixed mode, sharing)

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:35:25Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}

Possible Solution

This is possibly(likely) related to: fabric8io/kubernetes-client#3699

[csviri]

thx @andreaTP will try to take a look asap.
cc to @manusa @shawkins ; since this looks like an issue with fabric8 client

[manusa]

It's the issue reported in fabric8io/kubernetes-client#3697

Has a PR open with the fix: fabric8io/kubernetes-client#3699

[csviri]

@metacosm probably we should wait with v2 release, until this is fixed

[metacosm]

@metacosm probably we should wait with v2 release, until this is fixed

Yes. Weird that we haven't seen it in our testing… 😢

[csviri]

yep, but probably because we use everywhere informers, as you normally would.

[scrocquesel]

5.11.2 is released and fixes the issue.

[metacosm]

The project has been updated.