Add AVIF to wheels using only aomenc and dav1d AVIF codecs for reduced size

[fdintino]

Fixes #8856

This removes all libavif codecs except for AOM for encoding and dav1d for decoding. It also builds libavif as a shared library on linux and macOS, which modestly decreases the file size because it allows the linker to omit unused objects from the codec libraries. Building with -Os and -flto reduces the size by another further 0.7-1.5MB.

Below is a comparison of the average CPython wheel sizes by platform for the wheels (MiB) prior to the merge of the avif pull request, the current 11.2.0 wheels, and the wheels from this pull request. An additional few-hundred K could be saved by using aom's decoder instead of dav1d, but I think the benefits of dav1d outweigh those cost savings.

Since Pillow does not yet support multi-channel high bit depth images, I've compiled libaom without high-bit-depth support. This further reduces wheel sizes by ~200KiB.

In absolute terms, the uncompressed libavif shared library file size is somewhere between 2.8MiB (macOS ARM) and 5.0MiB (musllinux x86_64), with the exception of manylinux2014 x86_64 which, because of a gcc bug, is missing some optimizations and so is 7.4MiB.

platform	pre-AVIF	11.2.0	aomenc + dav1d	-flto -Os	further optimizations
macosx_86_64	3.0	10.9	6.9	5.8	5.1
macosx_arm64	2.9	8.5	5.7	4.9	4.5
win_amd64	2.5	13.1	7.4	6.9	6.7
manylinux_2_17_aarch64	4.2	16.3	7.7	6.3	5.6
manylinux_2_17_x86_64	4.3	18.0	8.2	8.2	7.3
manylinux_2_28_aarch64	4.3	16.8	8.0	6.5	5.7
manylinux_2_28_x86_64	4.4	18.7	8.6	7.2	6.3
musllinux_1_2_aarch64	4.3	17.2	8.3	6.6	5.8
musllinux_1_2_x86_64	4.4	18.8	8.7	7.3	6.4

[fdintino]

I'm going to see what effect @wantehchang's suggestion has on the wheel sizes.

[wantehchang]

Did you mean to change BUILD_SHARED_LIBS to ON? It is still OFF in winbuild/build_prepare.py.

[fdintino]

I did. Using the linker to generate the shared library file allows it to remove unused objects. There is a mechanism for including shared library dependencies of a python extension inside a wheel file on macOS and linux (delocate and auditwheel, respectively) but there is no such standard mechanism for doing the same on windows. So the windows library has to be built statically.

[fdintino]

The savings are modest, about 0.5 MB, but there isn't any cost to switching to a shared library.

[radarhere]

When you say 0.5mb, I presume that's per wheel. Which wheel(s) were you using to check that?

[wantehchang]

Frankie: Here are some other ideas to further reduce the binary size.

1. On Linux, use the compiler flags -ffunction-sections -fdata-sections and the linker flag -Wl,--gc-sections. See, for example, https://www.sandordargo.com/blog/2023/07/19/binary-sizes-and-compiler-flags and https://www.sandordargo.com/blog/2023/07/19/binary-sizes-and-compiler-flags.
2. Look at the binary sizes of libsharpyuv and libyuv.
   • If libsharpyuv is big, consider changing -DAVIF_LIBSHARPYUV=LOCAL to -DAVIF_LIBSHARPYUV=OFF.
   • libyuv is critical to performance. If libyuv is big, I will look into how to make it smaller.
3. In libaom, it may be possible to exclude the fallback SIMD optimizations and only include the "highest" SIMD optimizations. For example, if a function has an AVX2 optimization, then exclude its SSE4 and SSE2 optimizations. I will look into this.

[hugovk]

The alternative #8869 removed installation of things like rav1e and svt-av1, are they needed here?

[fdintino]

They have been removed here as well. The only difference between this PR and that one is that this one uses the dav1d decoder instead of AOM's. And it leaves open the possibility of users compiling for more codecs

[wantehchang]

@hugovk Another difference between this PR and the alternative #8869 is that this PR has additional changes to the build system to reduce the binary size, specifically:

• Change -DBUILD_SHARED_LIBS=OFF to -DBUILD_SHARED_LIBS=ON (except on Windows)
• Change -DCMAKE_BUILD_TYPE=Release to -DCMAKE_BUILD_TYPE=MinSizeRel under some condition (I haven't figured out what the condition is)
• Add -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=ON

For a fair comparison, these changes should be applied to the alternative #8869.

[radarhere]

If I compare the wheel sizes from https://github.com/python-pillow/Pillow/actions/runs/14773502497?pr=8858,
with a commit where I remove this line, I find that https://github.com/radarhere/Pillow/actions/runs/14773542213 is actually 8kb smaller.

If this line is purely about reducing file size, then I don't think it's fulfilling that role.

[fdintino]

Ah, it's because there is a typo. I'll fix it.

[fdintino]

It actually had to do with a cmake policy that was disabling LTO on linux. See the build here: https://github.com/fdintino/Pillow/actions/runs/14777836226

[fdintino]

In the PR description, I've updated the paragraph that begins with "In absolute terms,..." and the last column of the table with the latest wheel and shared library file sizes. You can look at the github diff under the list of edits to see the change.

[wantehchang]

We should not disable support for encoding high bit depth images. Most AVIF HDR images are 10 bits. Can we avoid this?

[fdintino]

Pillow does not yet support high bit depth images (see #1888). It is currently a work-in-progress, but at the moment there can only ever be 8-bit images passed to the C API.

[wantehchang]

Thanks for the reply.

Nit: Change "another flag" to "a flag" because we are removing the previous paragraph (lines 131-132).

We can look into disabling support for decoding high bit depth images in the dav1d library. It has this build option:

option('bitdepths',
    type: 'array',
    choices: ['8', '16'],
    description: 'Enable only specified bitdepths')

But it seems hard to pass this build option to dav1d through libavif's cmake command line.

[fdintino]

I think we might still want to allow decoding of 10-bit images, even if that necessarily means downsampling to 8 bit to load them into Pillow.

[hugovk]

This PR was about x1.8 in April (#8856 (comment)), and we're getting close to the next release, how is it looking now?

[doublex]

Yes, please.
That would greatly simplify deployment.

[radarhere]

This PR was about x1.8 in April (#8856 (comment)), and we're getting close to the next release, how is it looking now?

The size of Pillow 11.2.1 artefacts (wheels and sdist zips) was 321.03mb

The size from main is currently 374.13mb - it's expected that would go up though, as we now include Python 3.14 beta wheels.

The size from this PR is currently 591.47mb - so that's 1.58 times main.

[radarhere]

I expect libsharpyuv doesn't have a license file here because it comes from libwebp.