Tags missing from pypy3 on Windows 10

[brettcannon]

Comparing what pip's tags and packaging.tags under pypy3 7.0.0 on Windows 10, the following tags are only found by pip:

1. pp370-pp370-win32
2. pp3-none-any
3. py370-none-any

The first one could be considered legitimate, but the last two do not make sense.

[brettcannon]

Might need to tweak things so that in the generic case if no ABI is specified not only is none used but also an ABI for the name of the interpreter as another fallback.

[brettcannon]

Flagging as a question to ask the PyPy folks, but based on their answer either this issue should be closed or treated as a bug.

[pradyunsg]

@brettcannon Did we manage to ask PyPy folks about this at some point, and which ones does packaging.tags generate today?

[brettcannon]

@pradyunsg nope, I haven't gotten around to worrying about this issue yet.

[pradyunsg]

I'm a tweety bird: https://twitter.com/pradyunsg/status/1180465257426210818

[mattip]

Hi. PyPy dev here. Where did those tags come from, what produced them? I see tags such as cp3-none-any on PyPI, what makes pp3-none-any invalid?

[pradyunsg]

Thanks a lot for chiming in @mattip! ^>^

Where did those tags come from, what produced them?

It's produced as a part of pip's PEP 425 checks: https://www.python.org/dev/peps/pep-0425/.

You can see what pip generates as "compatible with", by running pip debug --verbose on the latest pip, on the relevant interpreter. (warning: it's a long list)

---

I think pp3 makes sense, but I'm not so sure -- my intuition on PEP 425 isn't good. :)

[mattip]

Here is the list for the upcoming pypy release which is python3.6-language compatible pypy7.2, where "language compatible" means "cpython interpreter + stdlib"

Compatible tags: 15
  pp372-pypy3_72-manylinux2010_x86_64
  pp372-pypy3_72-manylinux1_x86_64
  pp372-pypy3_72-linux_x86_64
  pp372-none-manylinux2010_x86_64
  pp372-none-manylinux1_x86_64
  pp372-none-linux_x86_64
  py3-none-manylinux2010_x86_64
  py3-none-manylinux1_x86_64
  py3-none-linux_x86_64
  pp372-none-any
  pp3-none-any
  py372-none-any
  py3-none-any
  py371-none-any
  py370-none-any

The list for cpython3.6 is much longer, it has 34 entries.

The tag pp3-none-any is for packages that have pypy-specific code (like cliquet does for cffi. The tag pp372-*does seem confusing at first glance. It is meant to reflect the PEP 425 format {python tag}-{abi tag}-{platform tag}, where the python tag is "python3 language specification (pp3)" + "pypy version 7.2 (72)". The text says " PyPy should probably use its own versions here pp18, pp19" without diving into the meaning, and we chose a format where the major-minor pypy version is always the suffix. So far we do not have concurrent python3 releases (python3.6 and 3.7 both) so it is clear which python minor version is targeted because the pypy version will always increment.

If we do release two or more concurrent minor versions, we should add the minor version to the python tag, so the "python tag" would become pp3772 for our first python 3.7 release.

[mattip]

The previous comment explains (1) and (2) from the original issue, I agree that (3) is an artifact that could be removed, py371 and py370 make no sense. For reference, cpython produces the following py python tags:

  py3-none-manylinux2010_x86_64
  py3-none-manylinux1_x86_64
  py3-none-linux_x86_64
  py36-none-any
  py3-none-any
  py35-none-any
  py34-none-any
  py33-none-any
  py32-none-any
  py31-none-any
  py30-none-any

So perhaps pypy should produce all those as well. It think this is connected to pypa/pip#3075, in particular a change that was made to get the sys.pypy_version_info as part of get_impl_version_info. While that seems to match the function name, it is used in get_supported where a non-implementation get_version (which does not exist) might have sufficed.

[pradyunsg]

Thanks for your responses here @mattip! Much appreciated! I'll defer to @brettcannon here since I have no intuition for PEP 425's operation).

---

BTW, sorry that I didn't provide more context earlier -- packaging.tags is the intended replacement for pip's pep425tags.py module (pypa/pip#6908).

---

W.R.T. how the tags differ between these two generation mechanisms, I'd suggest using this script (which uses pip's internals):

from __future__ import print_function  # Python 2.7 compatibility
from pip._internal.pep425tags import get_supported
from packaging.tags import sys_tags

pip_tags = {'{}-{}-{}'.format(*tag) for tag in get_supported()}
packaging_tags = {str(tag) for tag in sys_tags()}

li = [
    ("Only in pep425tags:", pip_tags - packaging_tags),
    ("Only in packaging.tags:", packaging_tags - pip_tags),
    ("In both groups:", packaging_tags & pip_tags),
    ("All tags:", packaging_tags | pip_tags),
]
for what, tags in li:
    print(what, "\n  ".join(sorted(tags)), sep="\n  ")
for what, tags in li:
    print(what, len(tags))

[mattip]

It seems (3) is only shows up in pip and could be considered a bug there. I would expect packaging.tags to support both cp3-none-any and pp3-none-any, which currently it does not. packaging.tags also supports extra tags (in both cpython and pypy) that pip did not: ones of the form py33-none-linux_x86_64 with the minor python version. Is that on purpose?

[brettcannon]

@mattip So I think there are two questions for me to answer here, but if I miss any let me know!

First, I consider pp3-none-any invalid because that's a tag saying "I have code specific to PyPy version 3.* of the interpreter"(e.g. PyPy 3.7); this has nothing to do with Python version support. As of now there is no cp3 on purpose because it makes no sense (at least right now) to have code tied to CPython 3.* but not any specific feature version of CPython as the compatibility guarantees for thing uinque to CPython are not the same as the language itself. Now if PyPy is different then we could consider pp3 legitimate, but it would be unique in that regard.

As for your question about whether the difference between packaging.tags and what pip provides is on purpose, the answer is emphatically yes (there were discussions on distutils-sig about this and I spent months diving into all of this). 😄 I wrote packaging.tags specifically to plug holes that pip currently has in its wheel tag support and all the changes are on purpose (especially for PyPy). The goal is to eventually rip out pip's pep425tags module that it has vendored and rely on packaging.tags instead. This issue is just an extension of that work to make sure PyPy is being supported appropriately based on what packaging.tags emits on Windows ATM.

[mattip]

Thanks for explaining, I am slowly getting up to speed. I agree (3) is not useful, and (1) would be better expressed as pp3671 : 36 from sys.version_info and 71 from sys.pypy_version_info

As for (2), I claim that there are reasons to have py3-none-any, cp3-none-any and pp3-none-any since there are implementation detail differences and performance related differences between the implementations that package authors may want to exploit. For instance, on CPython it may be better to depend on third-party c-exension packages, on PyPy it is always better to use CFFI extension modules. In the example I used above (cliquet), they would prefer to use psycopg2>2.5 on CPython and psycopg2cffi>2.7.0 on PyPy. If there was only a py3-none-any tag, how could they differentiate?