Please do not remove `setup.py install` as it is needed for distribution packagers

[mgorny]

As requested in #510 (comment) I'm opening a new issue. What I'd like for is that setup.py install --root=... (or a minimal equivalent) remains working as a 'low-level' install command for distribution packagers.

Installing packages straight from setuptools (much like distutils) has important advantages for us:

1. The --root= install mode fits just fine the packaging logic we have. We need a command that puts files in a directory we can use. We build distribution packages, and would really like to avoid building a wheel just to have to repack it into plain package.
2. We need minimal dependencies, and certainly have to avoid circular dependencies. While for tools like pip it might be acceptable to employ dirty bootstrap hacks, we really prefer not to have to rely on that to get the initial install working.
3. There are literally thousands of packages calling setup.py install one way or another. At least in Gentoo the majority of them is covered by our Python framework and could be easily swapped to use another install method but I can imagine others are not so lucky. It's not fair to ask people to change all that.
4. There are many packages that customize the install command, including some high profile ones. If the command is removed, they would all become broken, wouldn't they?

All that considered, please keep setup.py build and setup.py install commands for us. I don't care if they output huge warnings, require magical environment variables to be run but they need to stay as backend commands.

[pganssle]

I hope that I did not give you false hope that there was some possibility that we would continue maintaining the setup.py install command by suggesting that you open this issue. I am fairly confident that that will not happen for a variety of reasons. I mainly wanted to suggest that we get your requirements on the table so that we can find a better way to solve them.

Context

To give you a bit of context here, for a long time in the "distutils era", there was not really any separation between build backends and front-ends. distutils was make + compiler + package manager + test-runner + everything else. Packages started to rely on these blurry lines between front-ends and back-ends, and we ended up with a huge morass of tightly coupled interfaces that were very hard to update for the modern world. Ideally, new build tools would come around that could solve different use cases, but unfortunately so much code out there relies on setuptools / distutils (and then layer in on top of that code that assumes that pip does the installing), that any new build tools needed to be bug-for-bug compatible with the existing stuff, in a lot of ways.

PEP 517 and PEP 518 are part of a wider effort to standardize build tooling in a way that allows for new build tools to be built without requiring every build tool to actively be compatible with every other build tool. We've standardized on a way to specify build dependencies (PEP 518) and standardized the relationship between back-ends (like setuptools and flit) and front-ends (like pip and tox).

setuptools's place in the new world

setuptools has historically had features of both front-ends (e.g. installation) and back-ends (e.g. build), and since there are new de facto standards for all the front-end actions, setuptools is getting out of the business of being any sort of CLI application.

One of the biggest reasons to remove the setup.py install command is that it is broken in a large number of situations, and creates a lot of pain and confusion for end users. It creates pkg_resources-based entry points, can make your packages uninstallable, and just generally break things. It would be painful to fix and even if we did, we don't have the resources to continue maintaining it when there's a perfectly good and recommended alternative (pip). Just like when we removed setup.py upload and setup.py test, it may seem like we are removing something that works without replacing it, but this is wrong on both counts – there is a replacement and it doesn't currently work, it just may not fail terribly in some use cases.

We are aware that it will require updating a lot of existing code. We're trying our best to make it as seamless as possible, but making it seamless requires early adopters to tell us where the pain points are. Hopefully distro packagers can start trying out the alternatives and reporting where it doesn't fit with their workflows.

The supported workflow

There is an upside to this that I hope that distro packagers can see, which is that as we move to the newly standardized interface, you can write a "build and install from source" script that will work for all projects, not just ones using setuptools. setuptools has a lot of rough edges, and I would not be surprised if you start seeing a lot more projects using flit, poetry or other build tools in the coming years.

There is no requirement that you use pip for this, though. That's the beauty of the new standardized workflows. The workflow looks like this:

1. Check out source and build wheel from it using PEP 517 hooks.
2. Install the wheel using the process standardized in PEP 491.

There are tools to allow you to do both of these things, including the pep517 library and pip. There is also an effort underway to create a standardized installer for wheels to handle the second phase without pulling in pip: https://github.com/pradyunsg/installer

Since these are standards, you can use the tools provided that meet those standards, or you can build your own tools that meet the standards and satisfy your requirements – both choices are equally supported, at least in setuptools.

[pganssle]

And as for some of your original points:

2. We need minimal dependencies, and certainly have to avoid circular dependencies. While for tools like pip it might be acceptable to employ dirty bootstrap hacks, we really prefer not to have to rely on that to get the initial install working.

Unfortunately, setuptools already has circular dependencies in the same way that pip does, in that we vendor several packages that are themselves built by setuptools. Both pip and setuptools vendor them specifically because of the issue of circular dependencies. I believe you have no choice but to either allow vendoring (in which case there are no circular dependencies and in fact no dependencies) or to allow circular dependencies. I always felt that bootstrapping from a previously known-good version of setuptools was not a terribly bad way to do this, and if desirable, I think we may be able to add a support guarantee that you should be able to build setuptools from the most recent previous version of setuptools (which is to say, we won't take advantage of any new features until they've existed for at least one release).

3. There are literally thousands of packages calling setup.py install one way or another. At least in Gentoo the majority of them is covered by our Python framework and could be easily swapped to use another install method but I can imagine others are not so lucky. It's not fair to ask people to change all that.

I don't think it's a huge imposition to ask people to substitute setup.py install for what is, in almost all cases, a drop-in replacement (pip install). As is we're already giving some notice in our communications, and we're working out a strategy to make sure this is communicated with a lot of notice to allow for longer migration times.

It's better to do this than to allow users to have increasingly broken and unsupported installs.

4. There are many packages that customize the install command, including some high profile ones. If the command is removed, they would all become broken, wouldn't they?

It depends on how it's implemented, but this depends on how they've customized it. If they overrode it without calling the setuptools install command, then no it wouldn't break. Otherwise yes, it would break. There will be a reasonably long deprecation period for them to migrate, and I don't think it will be terribly painful.

[anthraxx]

the ecosystem gets literally wrecked from a distro point of view while no proper replacement exists as of today. no, pip is not a sane choice that suites proper distro needs without boilerplate and hassle.

As there isn't much point to repeat what @eli-schwartz already appropriately summarized, one can simply read here: #2080

[eli-schwartz]

It creates pkg_resources-based entry points

Hmm, this at least really doesn't seem to be a dealbreaker, because at the very least on recent versions of python setuptools should be able to use importlib.metadata instead, shouldn't it? And failing that (and more generally from back in the day) it's very trivial to get the right import path and use it directly. I was under the impression that this seemed to be a deliberate design choice of setuptools, to validate dependencies, because otherwise whyever would one not write out foo = foo.main:main as "script foo, from foo.main import main; main()".

(In keeping with the painful habit of python software writers to list a lot of dep==1.0 in their install_requires instead of doing version ranges, it seemed like this was an enforcement method. Which isn't to say that no one uses versions correctly, of course.)

Just like when we removed setup.py upload and setup.py test, it may seem like we are removing something that works without replacing it, but this is wrong on both counts – there is a replacement and it doesn't currently work, it just may not fail terribly in some use cases.

The problem here, is that "there's a replacement, it just doesn't work" means that there is not, in fact, a replacement. This is why it is so frustrating to people! Usually one implements a replacement before deprecating and removing its predecessor. Python is, instead, leaving us in a situation where there is nothing, and justifying it with the rationale "well, eventually there will be something".

And yes, I know that setup.py install is not actually removed. However, I come here by way of #2080, because I'm in a situation where setup.py install doesn't work -- people have deleted their setup.py files because setuptools' pep517 backend supports creating wheels without them, and then we cannot do anything with those wheels, because that part of the "replace setup.py install" environment doesn't exist yet...

There is also an effort underway to create a standardized installer for wheels to handle the second phase without pulling in pip: https://github.com/pradyunsg/installer

Thank you, this is very helpful information which I appreciate. I'd be more than happy to experiment with using to see if it can replace setup.py install for us, though as it seems to be merely days old, and still mostly scaffolding, I cannot make promises...

Unfortunately, setuptools already has circular dependencies in the same way that pip does, in that we vendor several packages that are themselves built by setuptools. Both pip and setuptools vendor them specifically because of the issue of circular dependencies. I believe you have no choice but to either allow vendoring (in which case there are no circular dependencies and in fact no dependencies) or to allow circular dependencies.

Note that this isn't actually a problem, since setuptools' dependencies can build with distutils alone, and this is how we bootstrapped python 3.8.0, for example. (I also know that previous major releases of python were bootstrapped on Arch Linux by tediously editing a build container to sideload setuptools. But this is a problem for reproducible builds and given we can now cleanly bootstrap I see no good reason to go back.)

It's fairly simple to do, at least for the very small handful of packages which have tried to support this (and which don't need entrypoints or other things distutils is simply incapable of at all):

# we are a dependency of setuptools, so using setuptools creates a circular
# dependency when building a python stack from source. We therefore allow
# falling back to distutils.
try:
    from setuptools import setup
except:
    from distutils.core import setup

[pganssle]

@anthraxx No one says you have to use pip if you don't find it acceptable.

I'll note that Fedora uses pip in their build workflow.

As I mentioned above, installation workflow is moving to be standards-based, and setup.py install does not meet those standards, nor do we have the resources to maintain it to do so.

I think you're probably being a bit over-dramatic here. No one is going to yank setup.py install out from under you. I suspect we won't actively remove it for at least a year, and the same lack of resources that are preventing us from maintaining it is also preventing us from actively removing it.

Still, as I've mentioned, it's been unmaintained for a few years now. You should really look to migrate now. You don't have to go to pip, it's just that pip is pretty much guaranteed to work because everyone uses it. The difficulties of packaging pip itself aside (though my understanding is that you have to do that anyway, since you would want to make pip available as a system package, no?), you can do pip install --no-deps some_tarball -t <location> and it will be more or less guaranteed to work no matter what some_tarball is.

If that's not acceptable, it should be easy enough to write a wheel installer (you can collaborate on the one that's already in progress and then you probably don't even have to maintain it, going forward).

Again I'd like to emphasize two things that seem to be getting lost:

1. We will not deliberately break this (more than it is already broken) suddenly and without warning.
2. The goal of PEP 517, 518 and a lot of the other standardization work will take a lot of the uncertainty out of your job, and it creates clear responsibilities so there's no need to play the "blame game" trying to decide who handles one thing or another.

[pganssle]

Hmm, this at least really doesn't seem to be a dealbreaker, because at the very least on recent versions of python setuptools should be able to use importlib.metadata instead, shouldn't it? And failing that (and more generally from back in the day) it's very trivial to get the right import path and use it directly. I was under the impression that this seemed to be a deliberate design choice of setuptools, to validate dependencies, because otherwise whyever would one not write out foo = foo.main:main as "script foo, from foo.main import main; main()".

It is semi-deliberate in that it pre-dates importlib.{resources,metadata} and it is one of the motivating factors behind moving to there. We have not changed it because direct invocations of setup.py install are unsupported. It's not impossible to change it, I suppose, but I tend towards inaction here because if we change something it may break existing code that people would then have to upgrade, only to have the thing they upgraded to deprecated or removed.

The problem here, is that "there's a replacement, it just doesn't work" means that there is not, in fact, a replacement. This is why it is so frustrating to people! Usually one implements a replacement before deprecating and removing its predecessor. Python is, instead, leaving us in a situation where there is nothing, and justifying it with the rationale "well, eventually there will be something".

I think you're missing my point here, I was not saying that the replacement doesn't work, I was saying that setup.py install doesn't work. It is a never-ending source of bugs that are solved by pointing people at pip install. It doesn't work with the latest standards, and it uses a bunch of unsupported and broken code paths.

I understand that pip is less convenient for you in some ways (is it just the circular dependency when devendoring?), but it's the replacement that actually works.

The PyPA also has spent ages and ages defining clear standards so that if you have unusual needs not met by the tools we've been able to prepare, you can go your own way without needing to veer into "unsupported" territory.

And yes, I know that setup.py install is not actually removed. However, I come here by way of #2080, because I'm in a situation where setup.py install doesn't work -- people have deleted their setup.py files because setuptools' pep517 backend supports creating wheels without them, and then we cannot do anything with those wheels, because that part of the "replace setup.py install" environment doesn't exist yet...

Yes, this is one reason why setup.py install doesn't work. You will see more and more packages that rely on PEP 517 and PEP 518, including projects that do not use setuptools. They are all valid Python packages.

Note that this isn't actually a problem, since setuptools' dependencies can build with distutils alone, and this is how we bootstrapped python 3.8.0, for example. (I also know that previous major releases of python were bootstrapped on Arch Linux by tediously editing a build container to sideload setuptools. But this is a problem for reproducible builds and given we can now cleanly bootstrap I see no good reason to go back.)

I hate to be the bearer of even more bad news, but distutils is super deprecated, unfortunately. We will remove it from CPython as soon as we have the opportunity and resources to migrate it into setuptools. It is not a long-term solution to your problem.

Though there's also the possibility that one or more of them will migrate to using flit or another build that requires a PEP 517 backend. I believe you will need to solve the bootstrap problem eventually. The easiest solution, to me, would be to always build setuptools and its dependencies using the previous version of setuptools. Presumably this or something like it is what is done for bootstrapped compilers like gcc. That said, it's up to you how you want to solve this problem, and deferring it until it becomes an immediate threat is a reasonable choice, just wanted to give you something to think about.

[eli-schwartz]

The easiest solution, to me, would be to always build setuptools and its dependencies using the previous version of setuptools. Presumably this or something like it is what is done for bootstrapped compilers like gcc. That said, it's up to you how you want to solve this problem, and deferring it until it becomes an immediate threat is a reasonable choice, just wanted to give you something to think about.

We've discussed internally, a few times, the thought of creating a bootstrap repository for toolchains. Nothing is final.

Simply depending on the previous version of setuptools doesn't work as that logic breaks down when moving from python 3.8 to python 3.9...

One priority for bootstrapping would be not minimizing the number of packages it takes to bootstrap. If:

• setuptools (6 packages)
• toml (after setuptools is built) plus pep517.build
• pytoml (after setuptools is built) plus flit_core
• 'installer', however that turns out

is a viable way forward, we'd have a couple paths to bootstrapping the ecosystem with only a handful of packages.
(Graduating a battle-tested toml library to the stdlib would be super useful, truth be told. It's become central to packaging, everything depends on one, and in fact, some things only depend on one, so it would be interesting to see what could simply build itself without any external dependencies.)

Bootstrapping pip requires 24 packages, and that rises to 40 packages if you count sphinx, which we need to build documentation in manpage format, and that's leaving aside the issue of pip having generally undesirable behavior for packaging. We currently document various mechanisms for installing python software, with pip caveats here: https://wiki.archlinux.org/index.php/Python_package_guidelines#pip

(but using pip for this is a hassle, so no one does it, so no one focuses on documenting the pitfalls, so it might be out of date.)

Since I'm currently unsure how bootstrapping in a possible post-distutils world might look, I think we'll have to see how the 'installer' package or similar tools play out before settling on one way to do it.

And in the meantime, we use setuptools, which, for our purposes at least, works essentially flawlessly.

[mgorny]

And yes, I know that setup.py install is not actually removed. However, I come here by way of #2080, because I'm in a situation where setup.py install doesn't work -- people have deleted their setup.py files because setuptools' pep517 backend supports creating wheels without them, and then we cannot do anything with those wheels, because that part of the "replace setup.py install" environment doesn't exist yet...

This is one thing I can help you with, at least for now. I've written https://github.com/mgorny/pyproject2setuppy to help with packaging this stuff for Gentoo. Its ~200 lines of code, has only toml as a dep and installs everything via setuptools without requiring you to install two new huge package managers you won't ever use or dephell which deserves the name. Of course, it will stop working when setup.py install is killed.

Besides, these PEPs pretty much prove where Python ecosystem is heading. We have new fancy way of installing stuff which uses integrated package managers in place of build systems, we have new fancy markup language that is required to deal with this… and at the same time, Python team refuses to include TOML in Python 3.9 because… the spec isn't finished yet.

So it's not yet ready to be included in Python so that it would reduce bootstrap trouble at least by a tiny bit. At the same time, it was mature enough to reinvent the whole build mechanics and announce killing the old one.

[eli-schwartz]

pypa/pyproject-hooks#81 is a hard blocker for using pep517... apparently I was wrong to be optimistic, not even that works.

[pganssle]

pypa/pep517#81 is a hard blocker for using pep517... apparently I was wrong to be optimistic, not even that works.

That's disappointing — I thought that it was possible to supply your own install hooks to the pep517 library. I've definitely had conversations with other Linux packagers (possibly someone from debian) about the possibility of a PEP 517 backend implementation that would use a chroot of some sort to satisfy the dependencies.

Presumably for packagers who are planning to satisfy the dependency from system packages, it would be split into two stages: stage 1 being an automated/semi-automated process to translate the PEP 518 metadata into system package dependencies (e.g. to populate the makedepends of your PKGBUILD), and the second phase being a PEP 517 build tool that satisfies all the other requirements of PEP 517 and assumes that build isolation is satisfied by the system.

This may have also come up in a discussion with @encukou about the possibility of a tox plugin that would do something similar.

[eli-schwartz]

Well, it looks like we could, we would just need to create a second python package just to reimplement main() around Pep517HookCaller without calling pip first.

This is quite awkward, not least because it's an explosion of tools and Arch Linux dislikes an explosion of custom distro tools (we devoutly avoid build recipe macros, templates, and inferred hooks with the rationale that they're unreadable and turn the act of packaging into some secret invite-only club of people who actually understand the endless layers). It would be best if pep517 could just... not call pip when it knows all the Distributions are there. The point of this reinvention of the python packaging landscape was to make things less coupled together, right? So it seems silly to couple pep517 to pip for any reason.

[pganssle]

Well, it looks like we could, we would just need to create a second python package just to reimplement main() around Pep517HookCaller without calling pip first.

Ah, so this is a simpler issue. I think that pep517 is intended to be an implementation detail for PEP 517 front-ends. It added a minimal PEP 517 build tool some time after its creation because no such thing existed at the time, but that was supposed to be a stop-gap measure. It's now a few years later and no one has bothered to build such a minimal builder tool.

This leaves us in something of a limbo where pep517.build is the best build tool out there, but it's not perfect. See this SO question and answer.

For a long time, I've been advocating for minimal, composable Unix-philosophy tools here, but there is a strong desire from many corners to have "all-in-one" tools like cargo and npm — see for example this thread.

Using pep517' as a CLI tool is also not great because it doesn't have pip's graceful fallback mechanism (which will populate the build-system table for you with reasonable defaults if you don't specify it), so you'd need some sort of thin wrapper script anyway.

So, in short, pep517 is a good way to take care of all the "heavy lifting" of the PEP 517 implementation, but you need to write a thin wrapper tool around it if you want to make good use of it. It shouldn't be that hard to do this.

That said, if you need something now, before anyone gets around to creating a simple builder, it may be worth making it slightly more general than it needs to be and proposing that it be an official PyPA recommendation to replace setup.py sdist bdist_wheel, living under the PyPA namespace. I'd be willing to advocate for that for sure.

[encukou]

Presumably for packagers who are planning to satisfy the dependency from system packages, it would be split into two stages: stage 1 being an automated/semi-automated process to translate the PEP 518 metadata into system package dependencies (e.g. to populate the makedepends of your PKGBUILD), and the second phase being a PEP 517 build tool that satisfies all the other requirements of PEP 517 and assumes that build isolation is satisfied by the system.

This may have also come up in a discussion with @encukou about the possibility of a tox plugin that would do something similar.

We use the tox plugin for running tests of projects that use tox for testing and recording test dependencies.
I wouldn't want to use tox for installation – for one thing, that would mean tox (and its dependencies like virtualenv, six) would be needed for any install.