ManagedChannel does not return from TRANSIENT_FAILURE to CONNECTING

[programmatix]

What version of gRPC-Java are you using?

I see this issue with 1.56.0 - but 1.55.1 works as expected.
Update: on further testing I found the issue also exists in 1.55.3 and 1.58.0.

What did you expect to see?

I create a ManagedChannel, and then have some logic that's waiting for it to go to connected. (Logic below)

I'm testing this logic when the server is not running. Based on this doc I expect the channel to go from state IDLE to CONNECTING to TRANSIENT_FAILURE, and then after an exponentially increasing delay, back to CONNECTING and so on. With 1.55.1 this is exactly what happens. The notifyWhenStateChanged() callback gets called plenty of times, and my code eventually raises a TimeoutException as expected.

ConnectivityState state = managedChannel.getState(true);
notify(state, onDone, deadline);

  private void notify(ConnectivityState current, CompletableFuture<Void> onDone, Deadline deadline) {
    if (inDesiredState(current)) {
      onDone.complete(null);
    }
    else {
      this.managedChannel.notifyWhenStateChanged(current, () -> {
        ConnectivityState now = this.managedChannel.getState(true);

        if (inDesiredState(now, waitingForReady)) {
          onDone.complete(null);
        } else if (deadline.exceeded()) {
          onDone.completeExceptionally(new TimeoutException());
        } else {
          notify(now, onDone, deadline, waitingForReady);
        }
      });
    }
  }

What did you see instead?

With 1.55.3, the channel goes from IDLE to CONNECTING to TRANSIENT_FAILURE - and then stays there. It will never return to CONNECTING. So my deadline never gets checked and this wait-until-ready code hangs indefinitely.

Interestingly - if I start the server, after a short delay the channel goes to READY state. So I think internally it is still going back into CONNECTING state - but perhaps neglecting to publish this to the state notification mechanism?

Steps to reproduce the bug

Hopefully the above information and logic is sufficient to replicate.

[programmatix]

On further testing I've found that the issue also occurs in 1.55.3. So seems to have been introduced between 1.55.1 and 1.55.3? (1.55.2 doesn't appear to be available or I'd test that).

I also checked the latest, 1.58.0, which continues to have the same issue.

[sanjaypujare]

I looked at the history https://github.com/grpc/grpc-java/commits/v1.55.x and I see only one PR potentially related to channel management #10331 . @temawi could you take a look and comment on this issue?

[ejona86]

This sounds like "sticky TRANSIENT_FAILURE." See the release notes: https://github.com/grpc/grpc-java/releases/tag/v1.56.0 . pick_first was the last of the policies to adopt "sticky TF."

Basically, going back to CONNECTING causes delays for RPCs for essentially no benefit. And in the future we're going to have a backoff per-address. grpc/proposal#356

[sanjaypujare]

This sounds like "sticky TRANSIENT_FAILURE." See the release notes: https://github.com/grpc/grpc-java/releases/tag/v1.56.0

But I don't see this change making it into the 1.55.x branch. So how is @programmatix seeing this in 1.55.3 ?

[programmatix]

Thanks @ejona86 and @sanjaypujare. That 'sticky TRANSIENT_FAILURE' certainly sounds related. I'll rerun the testing to be certain we started seeing this in 1.55.3 rather than 1.56.0.

So backing up a bit - how would you achieve what we're aiming for here? We have an SDK with a public-facing "waitUntilReady()" method that waits for connections to be established. We're trying to make that work for a new GRPC-based protocol, and we need it to timeout if the connection cannot be established. There probably aren't any RPCs being sent by the user at this point, as they are checking if a connection is possible. But we could potentially send no-op ones.

I did find the idleTimeout(X) setting on the channel can be used to force the channel to go idle again after X seconds. But we'd have to set X quite low to satisfy standard waitUntilReady() timeouts (user-set but often in the 5-30 second range). Is there any other approach possible?

[programmatix]

Actually - I think we can just poll channel.getState(true) in a loop until ACTIVE or timeout. I'll try that and see how it goes.

[programmatix]

This approach seems to work fine, so I'll close this out.