Add support for ext_modules in static setup.cfg

[embray]

In the process of porting an old project from the now defunct d2to1 I noticed a feature of d2to1 that is still missing from setuptools is support for configuring extension modules in setup.cfg.

At least, that's what my reading of the source suggests, especially ConfigOptionsHandler

In d2to1 this looks something like:

[extension=<module_name>]
sources = <file1> <file2> ...
language = c
<keyword> = <value>

where each key/value pair correspond to setuptools.Extension arguments.

For setuptools it might look something more like:

[options.ext_modules.<module_name>]
<key>=<value>

Though I'm open to other suggestions for the format, and would be happy to provide a PR once we agree on and nail down a format.

[jaraco]

Thanks for the report. Your instinct seems right here.

I worry a little bit about the arbitrarily-named sections. Is there precedent for that? A quick look through the code, I suspect the options.ext_modules.<module_name> would be handled by the ConfigOptionsHandler.

Yes, I think it could work.

I'm a little concerned about the . being used to designate the setuptools configuration namespace, but also the user's own namespace (module name), and wonder if the separator should be different to indicate a different naming domain, maybe something like [options.ext_modules:<module_name>] (=, ;, - also seem like suitable separators; I'll refrain from suggesting the separator-that-shall-not-be-named). It's probably not super-important what the separator is unless it's possible that the module_name could contain the separator.

Quickly scanning the docs, I see that pkg.foo is a possible module name, so I'd avoid using . to separate options.ext_modules from the module name. What do you think about :?

Feel free to get started on a PR; I'm hopeful there are few decisions to be made before an implementation can be accepted.

[embray]

I agree with your point about using a different delimiter to separate the config namespace from the module name. I'm fine with :, I think it's a good choice.

One problem with all this I realized, is that while it would work fine for simple cases, I noticed that it's still fairly common in several of my packages to involve a little bit of code in configuring extension modules. Most commonly was using numpy.get_include() to get the path to the Numpy headers. Another is using cythonize to build Cython modules. In d2to1 I believe I included special cases for both of these, but hard-coding special cases doesn't scale well.

Of course, this could get arbitrarily complex to the point where not every case can be supported in a static config. I think there could be a way for code to provide extensions to the syntax while still allowing most of the configuration to be static. But maybe as a first pass it would be best to just implement the "obvious" static configuration and then figure out more complex cases later.

[samvv]

Does there happen to be a roadmap for this? I'd be very interested to see this happen.

[jaraco]

Another is using cythonize to build Cython modules.

I believe Setuptools has built-in support for building Cython modules without anything extra. Just needs (a) cython installed, which is possible through build-requires and even legacy setup_requries, and (b) the source is a .pyx file. I'd hope that would cover a lot of cases.

Most commonly was using numpy.get_include() to get the path to the Numpy headers.

Numpy is important enough that it might deserve its own support... although I would be interested in exploring a generalizable technique for soliciting includes, independent of the configuration mechanism. For example, if Setuptools could solicit an entry point for setuptools.build_dirs or similar, then numpy (or perhaps a separate numpy-build) could implement setuptools.build_dirs as numpy.get_include(), thereby enabling the behavior simply by including the library in the build (no two-step process). In any case, let's explore the zero-config ideas before committing to a syntax in the config.

Does there happen to be a roadmap for this? I'd be very interested to see this happen.

No roadmap; just what you see above.

[bostonrwalker]

@embray I like this feature idea a lot. Did you start any work on it?

If possible, you should also be able to compile and include C libraries through the "libraries" argument.

[eggplants]

+1

It will help to greatly reduce the code from setup.py, especially for projects that provide C bindings.

[tvladyslav]

Would like to have this feature for C bindings.

[bostonrwalker]

I created a PR to add a similar feature to Python Poetry here. The PR is ready but the maintainers have said they don't have enough time to review it. If you guys want to have that functionality in Poetry, please go over there and help bug the maintainers for me.

[eli-schwartz]

It's not clear to me why setuptools users care what some other tool does? How does this help achieve a world where setuptools can encode ext_modules statically?

[bostonrwalker]

It's not clear to me why setuptools users care what some other tool does? How does this help achieve a world where setuptools can encode ext_modules statically?

I made this out of frustration with the lack of this feature originally in setuptools. You can feel free to take note and maybe create something similar in setuptools. Or not.