[BUG] Valid syntax for define-macros in pyproject.toml is invalid for ccompiler

[echoix]

setuptools version

75.8.0 (main branch at fb7f3d3)

Python version

3.12.7 and 3.13

OS

Ubuntu 22.04 and Ubuntu 24.04, in containers

Additional environment information

No response

Description

I'm porting a project (wxPython) from using a very custom build system+setup.py, to use a pyproject.toml and be buildable by standard build frontends.
I am struggling to find the correct syntax to use for defining a Cython extension module statically in pyproject.toml (experimental, introduced in 74.1.0).
There are no examples available anywhere on the web nor in tests of other fields than "name" and "sources". According to the jsonschema (https://github.com/pypa/setuptools/blob/main/setuptools/config/setuptools.schema.json), that isn't updated on schemastore by the way so my IDE is having a hard time to help me, the define-macros must be an array, of arrays, where there are one or two elements. By trial and error, I got to a point where my pyproject.toml was not failing the validation, and could continue.

I got this (shortened):

[project]
name = "wxPython"
requires-python = ">=3.9"
dynamic = ["readme", "version"]
dependencies = [
    "numpy ; python_version >= '3.0'",
    "typing-extensions; python_version < '3.11'",
]

[build-system]
requires = [
    "setuptools>=74.1.0",
    "cython == 3.0.11",
    "sip == 6.9.1",
]
build-backend = "setuptools.build_meta"

[tool.setuptools]
license-files = ["LICENSE.txt"]

[[tool.setuptools.ext-modules]]
name = "svg._nanosvg"
sources = ["wx/svg/_nanosvg.pyx"]
include-dirs = ["ext/nanosvg/src"]
define-macros = [
    [
        "NANOSVG_IMPLEMENTATION",
        "1",
    ],
    [
        "NANOSVGRAST_IMPLEMENTATION",
        "1",
    ],
    [
        "NANOSVG_ALL_COLOR_KEYWORDS",
        "1",
    ],
]

But when it comes to actually build the Cython extension, I get that my macros defined must be a one or two element tuple. The stack trace is below, but mentions TypeError: bad macro definition '['NANOSVG_IMPLEMENTATION', '1']': each element of 'macros' list must be a 1- or 2-tuple. This is from gen_preprocess_options, in

setuptools/setuptools/_distutils/ccompiler.py

Lines 1189 to 1230 in fb7f3d3

	def gen_preprocess_options(macros, include_dirs):
	"""Generate C pre-processor options (-D, -U, -I) as used by at least
	two types of compilers: the typical Unix compiler and Visual C++.
	'macros' is the usual thing, a list of 1- or 2-tuples, where (name,)
	means undefine (-U) macro 'name', and (name,value) means define (-D)
	macro 'name' to 'value'. 'include_dirs' is just a list of directory
	names to be added to the header file search path (-I). Returns a list
	of command-line options suitable for either Unix compilers or Visual
	C++.
	"""
	# XXX it would be nice (mainly aesthetic, and so we don't generate
	# stupid-looking command lines) to go over 'macros' and eliminate
	# redundant definitions/undefinitions (ie. ensure that only the
	# latest mention of a particular macro winds up on the command
	# line). I don't think it's essential, though, since most (all?)
	# Unix C compilers only pay attention to the latest -D or -U
	# mention of a macro on their command line. Similar situation for
	# 'include_dirs'. I'm punting on both for now. Anyways, weeding out
	# redundancies like this should probably be the province of
	# CCompiler, since the data structures used are inherited from it
	# and therefore common to all CCompiler classes.
	pp_opts = []
	for macro in macros:
	if not (isinstance(macro, tuple) and 1 <= len(macro) <= 2):
	raise TypeError(
	f"bad macro definition '{macro}': "
	"each element of 'macros' list must be a 1- or 2-tuple"
	)
	if len(macro) == 1: # undefine this macro
	pp_opts.append(f"-U{macro[0]}")
	elif len(macro) == 2:
	if macro[1] is None: # define with no explicit value
	pp_opts.append(f"-D{macro[0]}")
	else:
	# XXX *don't* need to be clever about quoting the
	# macro value here, because we're going to avoid the
	# shell at all costs when we spawn the command!
	pp_opts.append("-D{}={}".format(*macro))
	pp_opts.extend(f"-I{dir}" for dir in include_dirs)
	return pp_opts

Details

This is from a run using `uv build`, but I tried and got the same with pip and also the `build` package (it's just a bit harder to iterate on, as the project I'm working on currently needs the root directory in path, and has the current build system script named build.py, so running "python -m build" doesn't work from there, I need to be in another directory).

Compiling wx/svg/_nanosvg.pyx because it changed.
[1/1] Cythonizing wx/svg/_nanosvg.pyx
building 'svg._nanosvg' extension
Traceback (most recent call last):
  File "<string>", line 11, in <module>
    wheel_filename = backend.build_wheel("/workspaces/Phoenix/dist", {}, None)
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/build_meta.py", line 435, in build_wheel
    return _build(['bdist_wheel'])
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/build_meta.py", line 426, in _build
    return self._build_with_temp_dir(
           ~~~~~~~~~~~~~~~~~~~~~~~~~^
        cmd,
        ^^^^
    ...<3 lines>...
        self._arbitrary_args(config_settings),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/build_meta.py", line 407, in _build_with_temp_dir
    self.run_setup()
    ~~~~~~~~~~~~~~^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/build_meta.py", line 320, in run_setup
    exec(code, locals())
    ~~~~^^^^^^^^^^^^^^^^
  File "<string>", line 448, in <module>
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/__init__.py", line 117, in setup
    return distutils.core.setup(**attrs)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/core.py", line 186, in setup
    return run_commands(dist)
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
    dist.run_commands()
    ~~~~~~~~~~~~~~~~~^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 983, in run_commands
    self.run_command(cmd)
    ~~~~~~~~~~~~~~~~^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/dist.py", line 999, in run_command
    super().run_command(command)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 1002, in run_command
    cmd_obj.run()
    ~~~~~~~~~~~^^
  File "<string>", line 232, in run
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/cmd.py", line 339, in run_command
    self.distribution.run_command(command)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/dist.py", line 999, in run_command
    super().run_command(command)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 1002, in run_command
    cmd_obj.run()
    ~~~~~~~~~~~^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/command/build.py", line 136, in run
    self.run_command(cmd_name)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/cmd.py", line 339, in run_command
    self.distribution.run_command(command)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/dist.py", line 999, in run_command
    super().run_command(command)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 1002, in run_command
    cmd_obj.run()
    ~~~~~~~~~~~^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/command/build_ext.py", line 99, in run
    _build_ext.run(self)
    ~~~~~~~~~~~~~~^^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/command/build_ext.py", line 365, in run
    self.build_extensions()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/command/build_ext.py", line 481, in build_extensions
    self._build_extensions_serial()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/command/build_ext.py", line 507, in _build_extensions_serial
    self.build_extension(ext)
    ~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/command/build_ext.py", line 264, in build_extension
    _build_ext.build_extension(self, ext)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/Cython/Distutils/build_ext.py", line 135, in build_extension
    super(build_ext, self).build_extension(ext)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/command/build_ext.py", line 562, in build_extension
    objects = self.compiler.compile(
        sources,
    ...<5 lines>...
        depends=ext.depends,
    )
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/ccompiler.py", line 597, in compile
    macros, objects, extra_postargs, pp_opts, build = self._setup_compile(
                                                      ~~~~~~~~~~~~~~~~~~~^
        output_dir, macros, include_dirs, sources, depends, extra_postargs
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/ccompiler.py", line 355, in _setup_compile
    pp_opts = gen_preprocess_options(macros, incdirs)
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/ccompiler.py", line 1213, in gen_preprocess_options
    raise TypeError(
    ...<2 lines>...
    )
TypeError: bad macro definition '['NANOSVG_IMPLEMENTATION', '1']': each element of 'macros' list must be a 1- or 2-tuple
  × Failed to build `/workspaces/Phoenix`
  ├─▶ The build backend returned an error
  ╰─▶ Call to `setuptools.build_meta.build_wheel` failed (exit status: 1)
      hint: This usually indicates a problem with the package or the build environment.

So we see that it absolutely requires tuples, but how is really possible to express tuples in toml, and even so, how to get it to pass the toml schema validation?

Expected behavior

Be able to enter all other parameters of ext-modules in pyproject.toml, as mentionned in https://setuptools.pypa.io/en/latest/userguide/ext_modules.html

Optionally any other parameter of setuptools.Extension can be defined in the configuration file (but in the case of pyproject.toml they must be written using kebab-case convention).

I think that it is on the distutils side that they are too strict.

How to Reproduce

To help out, I made a test case of what I think gen_preprocess_options expects as input, just below test_pyproject_sets_attribute in setuptools/tests/config/test_apply_pyprojecttoml.py like when the ext-modules functionality was added in #4568.

I made it run in CI in: echoix#1

    def test_pyproject_sets_define_macros(self, tmp_path, monkeypatch):
        monkeypatch.chdir(tmp_path)
        pyproject = Path("pyproject.toml")
        toml_config = """
        [project]
        name = "test"
        version = "42.0"
        [tool.setuptools]
        ext-modules = [
          {name = "my.ext", sources = ["hello.c", "world.c"], define-macros = [ ["FIRST_SINGLE"], ["SECOND_TWO", "1"]]}
        ]
        """
        pyproject.write_text(cleandoc(toml_config), encoding="utf-8")
        with pytest.warns(pyprojecttoml._ExperimentalConfiguration):
            dist = pyprojecttoml.apply_configuration(Distribution({}), pyproject)
        assert len(dist.ext_modules) == 1
        assert dist.ext_modules[0].name == "my.ext"
        assert set(dist.ext_modules[0].sources) == {"hello.c", "world.c"}
        assert dist.ext_modules[0].define_macros[0] == ("FIRST_SINGLE",)
        assert dist.ext_modules[0].define_macros[1] == ("SECOND_TWO", "1")

I didn't understand enough the existing tests in setuptools/_distutils/tests/test_build_ext.py to be able to do something there, as it is where I think it is more there that the problem is.

In my project, I was running uv build --verbose --verbose, but that work isn't completed yet and I don't expect a successful build yet.

Output

(same as above)

Compiling wx/svg/_nanosvg.pyx because it changed.
[1/1] Cythonizing wx/svg/_nanosvg.pyx
building 'svg._nanosvg' extension
Traceback (most recent call last):
  File "<string>", line 11, in <module>
    wheel_filename = backend.build_wheel("/workspaces/Phoenix/dist", {}, None)
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/build_meta.py", line 435, in build_wheel
    return _build(['bdist_wheel'])
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/build_meta.py", line 426, in _build
    return self._build_with_temp_dir(
           ~~~~~~~~~~~~~~~~~~~~~~~~~^
        cmd,
        ^^^^
    ...<3 lines>...
        self._arbitrary_args(config_settings),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/build_meta.py", line 407, in _build_with_temp_dir
    self.run_setup()
    ~~~~~~~~~~~~~~^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/build_meta.py", line 320, in run_setup
    exec(code, locals())
    ~~~~^^^^^^^^^^^^^^^^
  File "<string>", line 448, in <module>
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/__init__.py", line 117, in setup
    return distutils.core.setup(**attrs)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/core.py", line 186, in setup
    return run_commands(dist)
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
    dist.run_commands()
    ~~~~~~~~~~~~~~~~~^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 983, in run_commands
    self.run_command(cmd)
    ~~~~~~~~~~~~~~~~^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/dist.py", line 999, in run_command
    super().run_command(command)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 1002, in run_command
    cmd_obj.run()
    ~~~~~~~~~~~^^
  File "<string>", line 232, in run
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/cmd.py", line 339, in run_command
    self.distribution.run_command(command)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/dist.py", line 999, in run_command
    super().run_command(command)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 1002, in run_command
    cmd_obj.run()
    ~~~~~~~~~~~^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/command/build.py", line 136, in run
    self.run_command(cmd_name)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/cmd.py", line 339, in run_command
    self.distribution.run_command(command)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/dist.py", line 999, in run_command
    super().run_command(command)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 1002, in run_command
    cmd_obj.run()
    ~~~~~~~~~~~^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/command/build_ext.py", line 99, in run
    _build_ext.run(self)
    ~~~~~~~~~~~~~~^^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/command/build_ext.py", line 365, in run
    self.build_extensions()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/command/build_ext.py", line 481, in build_extensions
    self._build_extensions_serial()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/command/build_ext.py", line 507, in _build_extensions_serial
    self.build_extension(ext)
    ~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/command/build_ext.py", line 264, in build_extension
    _build_ext.build_extension(self, ext)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/Cython/Distutils/build_ext.py", line 135, in build_extension
    super(build_ext, self).build_extension(ext)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/command/build_ext.py", line 562, in build_extension
    objects = self.compiler.compile(
        sources,
    ...<5 lines>...
        depends=ext.depends,
    )
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/ccompiler.py", line 597, in compile
    macros, objects, extra_postargs, pp_opts, build = self._setup_compile(
                                                      ~~~~~~~~~~~~~~~~~~~^
        output_dir, macros, include_dirs, sources, depends, extra_postargs
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/ccompiler.py", line 355, in _setup_compile
    pp_opts = gen_preprocess_options(macros, incdirs)
  File "/home/vscode/.cache/uv/builds-v0/.tmpA9wUTw/lib/python3.13/site-packages/setuptools/_distutils/ccompiler.py", line 1213, in gen_preprocess_options
    raise TypeError(
    ...<2 lines>...
    )
TypeError: bad macro definition '['NANOSVG_IMPLEMENTATION', '1']': each element of 'macros' list must be a 1- or 2-tuple
  × Failed to build `/workspaces/Phoenix`
  ├─▶ The build backend returned an error
  ╰─▶ Call to `setuptools.build_meta.build_wheel` failed (exit status: 1)
      hint: This usually indicates a problem with the package or the build environment.

[abravalheri]

Thanks @echoix, because this error is raising in distutils I think it is probably easier if we just post-process the dict coming from the TOML into a tuple before sending it down the stack.

I.e. changes needed around

setuptools/setuptools/config/_apply_pyprojecttoml.py

Lines 238 to 239 in 5c9d980

	args = ({k.replace("-", "_"): v for k, v in x.items()} for x in val)
	new = [Extension(**kw) for kw in args]

.