Breaks with requests 2.32.0: Not supported URL scheme http+docker

[rra]

With requests 2.32.0 (released about an hour ago as I write this), the docker library as called by tox-docker fails with the following exception:

Traceback (most recent call last):
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/requests/adapters.py", line 532, in send
    conn = self._get_connection(request, verify, proxies=proxies, cert=cert)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/requests/adapters.py", line 400, in _get_connection
    conn = self.poolmanager.connection_from_host(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/urllib3/poolmanager.py", line 304, in connection_from_host
    return self.connection_from_context(request_context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/urllib3/poolmanager.py", line 326, in connection_from_context
    raise URLSchemeUnknown(scheme)
urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme http+docker

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
                        ^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/requests/adapters.py", line 534, in send
    raise InvalidURL(e, request=request)
requests.exceptions.InvalidURL: Not supported URL scheme http+docker

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/tox/session/cmd/run/single.py", line 48, in _evaluate
    code, outcomes = run_commands(tox_env, no_test)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/tox/session/cmd/run/single.py", line 79, in run_commands
    MANAGER.tox_before_run_commands(tox_env)
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/tox/plugin/manager.py", line 88, in tox_before_run_commands
    self.manager.hook.tox_before_run_commands(tox_env=tox_env)
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/tox_docker/tox4/plugin.py", line 73, in tox_before_run_commands
    docker_build_or_pull(container_config, log)
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/tox_docker/plugin.py", line 57, in docker_build_or_pull
    docker_pull(container_config, log)
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/tox_docker/plugin.py", line 65, in docker_pull
    docker = docker_module.from_env(version="auto")
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/client.py", line 96, in from_env
    return cls(
           ^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: Not supported URL scheme http+docker

Reverting to requests 2.31.0 without any other changes fixes the problem.

[zhiyue]

same problem

In [1]: import docker
   ...: client = docker.from_env()
---------------------------------------------------------------------------
URLSchemeUnknown                          Traceback (most recent call last)
File /usr/local/lib/python3.8/dist-packages/requests/adapters.py:532, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    531 try:
--> 532     conn = self._get_connection(request, verify, proxies=proxies, cert=cert)
    533 except LocationValueError as e:

File /usr/local/lib/python3.8/dist-packages/requests/adapters.py:400, in HTTPAdapter._get_connection(self, request, verify, proxies, cert)
    398 else:
    399     # Only scheme should be lower case
--> 400     conn = self.poolmanager.connection_from_host(
    401         **host_params, pool_kwargs=pool_kwargs
    402     )
    404 return conn

File /usr/local/lib/python3.8/dist-packages/urllib3/poolmanager.py:304, in PoolManager.connection_from_host(self, host, port, scheme, pool_kwargs)
    302 request_context["host"] = host
--> 304 return self.connection_from_context(request_context)

File /usr/local/lib/python3.8/dist-packages/urllib3/poolmanager.py:326, in PoolManager.connection_from_context(self, request_context)
    325 if not pool_key_constructor:
--> 326     raise URLSchemeUnknown(scheme)
    327 pool_key = pool_key_constructor(request_context)

URLSchemeUnknown: Not supported URL scheme http+docker

During handling of the above exception, another exception occurred:

InvalidURL                                Traceback (most recent call last)
File /usr/local/lib/python3.8/dist-packages/docker/api/client.py:213, in APIClient._retrieve_server_version(self)
    212 try:
--> 213     return self.version(api_version=False)["ApiVersion"]
    214 except KeyError as ke:

File /usr/local/lib/python3.8/dist-packages/docker/api/daemon.py:181, in DaemonApiMixin.version(self, api_version)
    180 url = self._url("/version", versioned_api=api_version)
--> 181 return self._result(self._get(url), json=True)

File /usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py:44, in update_headers.<locals>.inner(self, *args, **kwargs)
     43         kwargs['headers'].update(self._general_configs['HttpHeaders'])
---> 44 return f(self, *args, **kwargs)

File /usr/local/lib/python3.8/dist-packages/docker/api/client.py:236, in APIClient._get(self, url, **kwargs)
    234 @update_headers
    235 def _get(self, url, **kwargs):
--> 236     return self.get(url, **self._set_request_timeout(kwargs))

File /usr/local/lib/python3.8/dist-packages/requests/sessions.py:602, in Session.get(self, url, **kwargs)
    601 kwargs.setdefault("allow_redirects", True)
--> 602 return self.request("GET", url, **kwargs)

File /usr/local/lib/python3.8/dist-packages/requests/sessions.py:589, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    588 send_kwargs.update(settings)
--> 589 resp = self.send(prep, **send_kwargs)
    591 return resp

File /usr/local/lib/python3.8/dist-packages/requests/sessions.py:703, in Session.send(self, request, **kwargs)
    702 # Send the request
--> 703 r = adapter.send(request, **kwargs)
    705 # Total elapsed time of the request (approximately)

File /usr/local/lib/python3.8/dist-packages/requests/adapters.py:534, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    533 except LocationValueError as e:
--> 534     raise InvalidURL(e, request=request)
    536 self.cert_verify(conn, request.url, verify, cert)

InvalidURL: Not supported URL scheme http+docker

The above exception was the direct cause of the following exception:

DockerException                           Traceback (most recent call last)
Cell In[1], line 2
      1 import docker
----> 2 client = docker.from_env()

File /usr/local/lib/python3.8/dist-packages/docker/client.py:94, in DockerClient.from_env(cls, **kwargs)
     92 version = kwargs.pop('version', None)
     93 use_ssh_client = kwargs.pop('use_ssh_client', False)
---> 94 return cls(
     95     timeout=timeout,
     96     max_pool_size=max_pool_size,
     97     version=version,
     98     use_ssh_client=use_ssh_client,
     99     **kwargs_from_env(**kwargs)
    100 )

File /usr/local/lib/python3.8/dist-packages/docker/client.py:45, in DockerClient.__init__(self, *args, **kwargs)
     44 def __init__(self, *args, **kwargs):
---> 45     self.api = APIClient(*args, **kwargs)

File /usr/local/lib/python3.8/dist-packages/docker/api/client.py:197, in APIClient.__init__(self, base_url, version, timeout, tls, user_agent, num_pools, credstore_env, use_ssh_client, max_pool_size)
    192 # version detection needs to be after unix adapter mounting
    193 if version is None or (isinstance(
    194                         version,
    195                         str
    196                         ) and version.lower() == 'auto'):
--> 197     self._version = self._retrieve_server_version()
    198 else:
    199     self._version = version

File /usr/local/lib/python3.8/dist-packages/docker/api/client.py:220, in APIClient._retrieve_server_version(self)
    215     raise DockerException(
    216         'Invalid response from docker daemon: key "ApiVersion"'
    217         ' is missing.'
    218     ) from ke
    219 except Exception as e:
--> 220     raise DockerException(
    221         f'Error while fetching server API version: {e}'
    222     ) from e

DockerException: Error while fetching server API version: Not supported URL scheme http+docker

pip list
Package                Version             
---------------------- --------------------
asttokens              2.4.1               
attrs                  19.3.0              
Automat                0.8.0               
backcall               0.2.0               
blinker                1.4                 
certifi                2019.11.28          
chardet                3.0.4               
charset-normalizer     3.3.2               
Click                  7.0                 
cloud-init             22.2                
colorama               0.4.3               
command-not-found      0.3                 
configobj              5.0.6               
constantly             15.1.0              
cryptography           2.8                 
dbus-python            1.2.16              
decorator              5.1.1               
distro                 1.4.0               
distro-info            0.23ubuntu1         
docker                 7.0.0               
entrypoints            0.3                 
executing              2.0.1               
httplib2               0.14.0              
hyperlink              19.0.0              
idna                   2.8                 
importlib-metadata     1.5.0               
incremental            16.10.1             
ipython                8.12.3              
jedi                   0.19.1              
Jinja2                 2.10.1              
jsonpatch              1.22                
jsonpointer            2.0                 
jsonschema             3.2.0               
keyring                18.0.1              
language-selector      0.1                 
launchpadlib           1.10.13             
lazr.restfulclient     0.14.2              
lazr.uri               1.0.3               
MarkupSafe             1.1.0               
matplotlib-inline      0.1.7               
more-itertools         4.2.0               
netifaces              0.10.4              
oauthlib               3.1.0               
packaging              24.0                
parso                  0.8.4               
pexpect                4.6.0               
pickleshare            0.7.5               
pip                    20.0.2              
prompt-toolkit         3.0.43              
pure-eval              0.2.2               
pyasn1                 0.4.2               
pyasn1-modules         0.2.1               
pygments               2.18.0              
PyGObject              3.36.0              
PyHamcrest             1.9.0               
PyJWT                  1.7.1               
pymacaroons            0.13.0              
PyNaCl                 1.3.0               
pyOpenSSL              19.0.0              
pyrsistent             0.15.5              
pyserial               3.4                 
python-apt             2.0.0+ubuntu0.20.4.7
python-debian          0.1.36ubuntu1       
PyYAML                 5.3.1               
requests               2.32.0              
requests-unixsocket    0.2.0               
SecretStorage          2.3.1               
service-identity       18.1.0              
setuptools             45.2.0              
simplejson             3.16.0              
six                    1.14.0              
sos                    4.3                 
ssh-import-id          5.10                
stack-data             0.6.3               
systemd-python         234                 
traitlets              5.14.3              
Twisted                18.9.0              
typing-extensions      4.11.0              
ubuntu-advantage-tools 27.9                
ufw                    0.36                
unattended-upgrades    0.1                 
urllib3                2.2.1               
wadllib                1.3.3               
wcwidth                0.2.13              
wheel                  0.34.2              
zipp                   1.0.0               
zope.interface         4.7.1 

[danb27]

same!

[felixfontein]

I'm currently debugging this a bit (ansible-collections/community.docker#860), the issue is commit psf/requests@c0813a2 in requests.

Ok, this isn't very surprising, since that commit makes send() call _get_connection() instead of get_connection(). Now the Docker SDK for Python code overwrites get_connection(), but of course doesn't magically overwrite _get_connection() as well...

See for example https://github.com/docker/docker-py/blob/main/docker/transport/unixconn.py#L66.

[felixfontein]

A simple fix is adding

    def _get_connection(self, request, *args, proxies=None, **kwargs):
        return self.get_connection(request.url, proxies)

at the bottom of https://github.com/docker/docker-py/blob/main/docker/transport/basehttpadapter.py.

Not sure how sustainable that is, but it works for now :)

[felixfontein]

I created #3257 with that fix.