Randomly unable to use Watch

[rogeralsing]

In Proto.Actor .NET we use the Kubernetes Client to Watch the Kubernetes API for active pods in a given namespace.
This works perfectly for the most clients, but in some cases, the Watch stops randomly after X time, usually days.

We have tried to bandaid this issue by recreating the client whenever there are any issues watching, but the issue is still manifesting.
https://github.com/asynkron/protoactor-dotnet/blob/dev/src/Proto.Cluster.Kubernetes/KubernetesClusterMonitor.cs#L90-L98

More to the point, the exception we get is this.
And even if we recreate a new Kubernetes Client, we still get the same issue.

Is it likely a network issue? e.g. connectivity problems?
Is it the Kubernetes API server that fails to respond?
Why is the problem persistent even if we recreate the Kubernetes Client?

I get that this might not be an issue in the Client lib itself, but that is what is throwing the exception.
Any ideas what could cause this?

exceptionMessage	The request was aborted. Unable to read data from the transport connection: Connection reset by peer. Connection reset by peer
exceptionMethod	Void ThrowRequestAborted(System.Exception)
exceptionStackTrace	at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException) at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState() at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span1 buffer, Boolean partOfSyncRead) at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken) at k8s.LineSeparatedHttpContent.CancelableStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken) at System.IO.StreamReader.ReadBufferAsync(CancellationToken cancellationToken) at System.IO.StreamReader.ReadLineAsyncInternal() at k8s.Watcher1.<>c.<CreateWatchEventEnumerator>b__21_1[TR](Task1 t) at System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke() at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at k8s.Watcher1.CreateWatchEventEnumerator(Func1 streamReaderCreator, Action1 onError, CancellationToken cancellationToken)+MoveNext() at k8s.Watcher1.CreateWatchEventEnumerator(Func1

[tg123]

Connection reset by peer means remote side sent RST
for example, might cause by server crash or lb closed the connection or any reason network related issue

you could have package captured to see why RST was sent from remote side

[brendandburns]

fwiw, you should not expect a watch to persist forever. A Watch is just a TCP stream at the end of the day, and the network is a chaotic place.

You should expect to do:

loop forever:
   List all things
   Watch for changes

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[IvanJosipovic]

Hopefully, this PR #1304 fixes this issue as well.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.