pkg_resources does not find .egg-info dirs from zipimport

[thatch]

In the common case where your path entries are directories, pkg_resources will find installed metadata from either .dist-info or .egg-info dirs. When zipimport is used, it only finds .dist-info. I have a simple PR incoming, but here's info on the repro.

Start with an unpacked sdist:

[thatch@laptop ~]$ cd $(mktemp -d)
[thatch@laptop tmp.w7h4GsxPI5]$ wget https://files.pythonhosted.org/packages/bc/ef/d6c26f40a7636f43a52c9719f9d8228f08e01187081e5891702ea0754060/parameterized-0.7.4.tar.gz
[thatch@laptop tmp.w7h4GsxPI5]$ tar -xvzf parameterized-0.7.4.tar.gz 
parameterized-0.7.4/
parameterized-0.7.4/PKG-INFO
parameterized-0.7.4/parameterized.egg-info/
parameterized-0.7.4/parameterized.egg-info/PKG-INFO
parameterized-0.7.4/parameterized.egg-info/SOURCES.txt
parameterized-0.7.4/parameterized.egg-info/pbr.json
parameterized-0.7.4/parameterized.egg-info/requires.txt
parameterized-0.7.4/parameterized.egg-info/top_level.txt
parameterized-0.7.4/parameterized.egg-info/dependency_links.txt
parameterized-0.7.4/parameterized/
parameterized-0.7.4/parameterized/__init__.py
parameterized-0.7.4/parameterized/test.py
parameterized-0.7.4/parameterized/parameterized.py
parameterized-0.7.4/.hgignore
parameterized-0.7.4/setup.py
parameterized-0.7.4/.gitignore
parameterized-0.7.4/tox.ini
parameterized-0.7.4/CHANGELOG.txt
parameterized-0.7.4/setup.cfg
parameterized-0.7.4/README.rst
parameterized-0.7.4/LICENSE.txt
parameterized-0.7.4/.travis.yml
parameterized-0.7.4/misspelling-helper/
parameterized-0.7.4/misspelling-helper/nose-parametrized/
parameterized-0.7.4/misspelling-helper/nose-parametrized/MANIFEST
parameterized-0.7.4/misspelling-helper/nose-parametrized/setup.py

If you add such a dir to the path, it works as-is:

[thatch@laptop tmp.w7h4GsxPI5]$ PYTHONPATH=parameterized-0.7.4 python -c 'import parameterized, pkg_resources; print(parameterized.__file__); print(pkg_resources.get_distribution("parameterized"))'
/tmp/tmp.w7h4GsxPI5/parameterized-0.7.4/parameterized/__init__.py
parameterized 0.7.4

But if we construct a zip and put that on the path:

[thatch@laptop tmp.w7h4GsxPI5]$ (cd parameterized-0.7.4; zip -r ../parameterized.zip *)
  adding: CHANGELOG.txt (deflated 64%)
  adding: LICENSE.txt (deflated 47%)
  adding: misspelling-helper/ (stored 0%)
  adding: misspelling-helper/nose-parametrized/ (stored 0%)
  adding: misspelling-helper/nose-parametrized/setup.py (deflated 48%)
  adding: misspelling-helper/nose-parametrized/MANIFEST (stored 0%)
  adding: parameterized/ (stored 0%)
  adding: parameterized/__pycache__/ (stored 0%)
  adding: parameterized/__pycache__/parameterized.cpython-38.pyc (deflated 56%)
  adding: parameterized/__pycache__/__init__.cpython-38.pyc (deflated 31%)
  adding: parameterized/parameterized.py (deflated 70%)
  adding: parameterized/test.py (deflated 80%)
  adding: parameterized/__init__.py (deflated 38%)
  adding: parameterized.egg-info/ (stored 0%)
  adding: parameterized.egg-info/dependency_links.txt (stored 0%)
  adding: parameterized.egg-info/top_level.txt (stored 0%)
  adding: parameterized.egg-info/requires.txt (stored 0%)
  adding: parameterized.egg-info/pbr.json (stored 0%)
  adding: parameterized.egg-info/SOURCES.txt (deflated 56%)
  adding: parameterized.egg-info/PKG-INFO (deflated 79%)
  adding: PKG-INFO (deflated 79%)
  adding: README.rst (deflated 74%)
  adding: setup.cfg (deflated 7%)
  adding: setup.py (deflated 49%)
  adding: tox.ini (deflated 61%)
[thatch@laptop tmp.w7h4GsxPI5]$ PYTHONPATH=parameterized.zip python -c 'import parameterized, pkg_resources; print(parameterized.__file__); print(pkg_resources.get_distribution("parameterized"))'
/tmp/tmp.w7h4GsxPI5/parameterized.zip/parameterized/__init__.py
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 481, in get_distribution
    dist = get_provider(dist)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 357, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'parameterized' distribution was not found and is required by the application

Now a try with the upcoming PR:

[thatch@laptop tmp.w7h4GsxPI5]$ . ~/code/setuptools/.venv/bin/activate
(.venv) [thatch@laptop tmp.w7h4GsxPI5]$ PYTHONPATH=parameterized.zip python -c 'import parameterized, pkg_resources; print(parameterized.__file__); print(pkg_resources.get_distribution("parameterized"))'
/tmp/tmp.w7h4GsxPI5/parameterized.zip/parameterized/__init__.py
parameterized 0.7.4

[jaraco]

I'm a little surprised this issue is appearing now. The logic for constructing packages and resolving them as pkg_resources does has been around for over a decade. Moreover, I'm working to encourage use of importlib.metadata and importlib.resources instead.

Would it be possible instead to use importlib.metadata and importlib.resources instead? I believe they support zipfiles with egg-info as well as other importers that won't be supported by pkg_resources. They'll be more future proof.

[thatch]

The reason it's happening for me now is not a regression -- it's an optimization in my build system on a monorepo that preserves the sdist metadata, rather than build a wheel, when the build process is straightforward enough. This works fine when unpacked, but fails in something like pex that uses zipimport.

Moving away from pkg_resources is a good fix. I'm happy to send a few dozen PRs to projects I find that use it, but I would like this fixed in pkg_resources as well, because I believe 1. I'm prettysure it is a bug, and 2. it should be pretty safe by matching more dirs than before.

Over the summer I sent a few hundred PRs to fix bad MANIFEST.in entries (like gitcnd/jspline#2 ) and getting new releases cut is not very reliable for the long tail of projects that people might want to use.

[thatch]

(At least 872 of the top 5000 pypi projects by download count use pkg_resources.)