root:socket_proxy.py:88 Proxying failed to listen

[jiridanek]

https://github.com/opendatahub-io/notebooks/actions/runs/13524125351/job/37791017649#step:40:181

INFO     root:socket_proxy.py:76 Proxy listening on localhost:0
INFO     root:socket_proxy.py:85 Accepted connection from 127.0.0.1:48394
ERROR    root:socket_proxy.py:88 Proxying failed to listen
Traceback (most recent call last):
  File "/home/runner/work/notebooks/notebooks/tests/containers/socket_proxy.py", line 86, in listen_and_serve_until_canceled
    self._handle_client(client_socket)
  File "/home/runner/work/notebooks/notebooks/tests/containers/socket_proxy.py", line 113, in _handle_client
    data = remote_socket.recv(self.buffer_size)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ConnectionResetError: [Errno 104] Connection reset by peer
=============================== warnings summary ===============================
tests/containers/workbenches/workbench_image_test.py::TestWorkbenchImage::test_image_run_on_openshift[ghcr.io/opendatahub-io/notebooks/workbench-images:jupyter-trustyai-ubi9-python-3.11-main_d954ab6f6b7e476ee38dd8ed6cd22f24aa726fd7]
  /home/runner/.cache/pypoetry/virtualenvs/notebooks-OIIyMdoB-py3.12/lib/python3.12/site-packages/kubernetes/client/exceptions.py:91: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
    self.headers = http_resp.getheaders()

tests/containers/workbenches/workbench_image_test.py::TestWorkbenchImage::test_image_run_on_openshift[ghcr.io/opendatahub-io/notebooks/workbench-images:jupyter-trustyai-ubi9-python-3.11-main_d954ab6f6b7e476ee38dd8ed6cd22f24aa726fd7]
  /home/runner/.cache/pypoetry/virtualenvs/notebooks-OIIyMdoB-py3.12/lib/python3.12/site-packages/_pytest/threadexception.py:82: PytestUnhandledThreadExceptionWarning: Exception in thread Thread-1 (listen_and_serve_until_canceled)
  
  Traceback (most recent call last):
    File "/opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
      self.run()
    File "/opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/threading.py", line 1012, in run
      self._target(*self._args, **self._kwargs)
    File "/home/runner/work/notebooks/notebooks/tests/containers/socket_proxy.py", line 86, in listen_and_serve_until_canceled
      self._handle_client(client_socket)
    File "/home/runner/work/notebooks/notebooks/tests/containers/socket_proxy.py", line 113, in _handle_client
      data = remote_socket.recv(self.buffer_size)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ConnectionResetError: [Errno 104] Connection reset by peer
  
    warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

[jiridanek]

looks like there's some retry logic needed in the test

[jiridanek]

Given that the logs also say

INFO     root:kubernetes_utils.py:295 Pods matching test-ns-notebook-tests-pod/app=notebook-tests-pod are ready

could be this is actually manifestation of https://issues.redhat.com/browse/RHOAIENG-1576

[jiridanek]

On the gripping hand though, we don't have proper readiness checks in this test set up in the pod, so likely this is irrelevant for the Jira I just mentioned

[jiridanek]

This one looks also like proxy failure https://github.com/jiridanek/notebooks/actions/runs/13862724294/job/38794871867

[jiridanek]

also https://github.com/jiridanek/notebooks/actions/runs/13876401994/job/38829257849#step:40:209

[jiridanek]

apparently the proxy can also get stuck

2025-03-15 21:00:30 INFO Proxy listening on localhost:0
2025-03-15 21:00:30 INFO Accepted connection from 127.0.0.1:54048

https://github.com/jiridanek/notebooks/actions/runs/13876401994/job/38829257865#step:40:41

this was running 14+ mins

[jiridanek]

and fail again https://github.com/jiridanek/notebooks/actions/runs/13881811543/job/38841318176

[jiridanek]

when the test fails, it is unclear if it was infra issue or if the pod failed to start and listen

E urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

https://github.com/jiridanek/notebooks/actions/runs/13881811543/job/38841318193#step:40:83

[jiridanek]

once more https://github.com/opendatahub-io/notebooks/actions/runs/13907084093/job/38912570821#step:40:183