Add SWIG and cmake to manylinux docker images

[HDembinski]

It would be great if the standard manylinux docker images could be replaced with some that have recent cmake and swig preinstalled. These tools are fairly common dependencies.

[Czaki]

You should open issue in this project https://github.com/pypa/manylinux

cibuildwheel depends on it.

[HDembinski]

Ok, see pypa/manylinux#659

Nevertheless, you could also make updated docker images and use those, it does not necessarily have to be fixed upstream...

[Czaki]

You also can use own docker images CIBW_MANYLINUX_X86_64_IMAGE.
Last cmake can be obtained from pypi https://pypi.org/project/cmake/

Swig is more problematic...

[YannickJadoul]

As @Czaki says, we provide an option to use your own docker image. In principle, extending an existing image should be reasonably simple?

But I don't feel anything for extending the manylinux images in cibuildwheel. manylinux is a subtle enough project, trying to ensure compatibility between different Linux systems, and I'm rather happy we can just use these images out of the box and most of the time don't have to worry about this. Besides, it's a good idea to keep the image as small as possible and not extend others' build time by adding a bunch of extra dependencies, I would think. Frankly speaking, I also don't wish to extend and take responsibility to maintain another part of the build toolchain, within this project.

For your use case, if you don't wish to create your own custom docker image: as @Czaki mentions, adding CIBW_BEFORE_BUILD="pip install cmake" works perfectly (in my project), and some combination of curl, tar, and make could also be added, no? I do something very similar in my project, and using the new CIBW_BEFORE_ALL option, this should be even simpler:

https://github.com/YannickJadoul/Parselmouth/blob/f899207fb60dbcfea6acd3d87d62d3be8ad62c1c/.travis.yml#L141-L145

EDIT: Feel free to try changing my mind, though :-)

[HDembinski]

Yes, I know I can do it differently, and I do.

I install swig and cmake as needed, using the appropriate tools for the platform, and that is easy on MacOS and Windows, but not on manylinux, because the yum sources are ancient, and the server connection fails regularly. The network issues cause random failures in wheel builds.

I thought the purpose of this project is to make building wheels for Python modules that need compilation as easy as possible. Currently, that promise only holds for projects that use pybind11 or Cython, but not those that use swig.

I am not a particular fan of swig, pybind11 is far surperior, but it is hard to kill/replace, because it can generate bindings for multiple languages. So we have to deal with it.

[YannickJadoul]

because the yum sources are ancient, and the server connection fails regularly

Yes, I know it's annoying, but this old CentOS is what's specified as manylinux image (intentionally, so it's compatible with lots of systems), so there's not too much we can do about it, I think. I also don't use yum for ccache, because it's tool old, but downloading/building works easily enough there.

About SWIG: there's two steps, no, conversion and compilation? Is there no way you can do the conversion before calling cibuildwheel? Only the compilation needs to happen inside the old docker image.

[HDembinski]

I don't have control over the software that I am building wheels for, the one that uses swig. But this is a good idea, perhaps I can pre-generate the swig code outside docker. On the other hand, I still need a recent cmake.

Whatever solution I chose, it adds additional complexity.

I am a beginner with docker, but it seems that generating and publishing an image based on manylinux is easy, so this seems like the most promising option.

[Czaki]

I am a beginner with docker, but it seems that generating and publishing an image based on manylinux is easy, so this seems like the most promising option.

It is not a problem.
You can look to this repository https://github.com/czaki/imagecodecs_build/blob/master/build_utils/Dockerfile and https://github.com/Czaki/imagecodecs_build/blob/master/build_utils/docker_64.sh
or even You can use this docker images

https://hub.docker.com/repository/docker/bokota/imagecodecs_64
https://hub.docker.com/repository/docker/bokota/imagecodecs_2014_64
https://hub.docker.com/repository/docker/bokota/imagecodecs_i686

But I update it only when there is release of imagecodecs which need new library or new version of library.
And this images has much more libraries compiled so images are bigger.

[HDembinski]

@Czaki Thank you for the helpful links, I will have a look.