Reproducing the precompiled executable

[bjorn3]

It seems you used a nightly version of rustc (rustc version 1.73.0-nightly (f3623871c 2023-08-06)), but which linker version and which musl version? And what Cargo.lock was used? How can I reproduce a bit for bit identical executable?

https://reproducible-builds.org/

[dvdsk]

I think we can take this a bit further. How about building the executable on Github CI? Then the method of creating the executable is documented in the repo. It would be trivial to replicate (fork the repo and run CI). The executable itself might even be added/replaced on each release by CI. That might ease maintenance.

Can anyone with experience and or real knowledge with supply chain security pitch in if this could work?

(at the very least this approach seems to assume GitHub is trusted).

[messense]

How about building the executable on Github CI?

+1 for this. It'd be nice to also publish sigstore signatures using sigstore/gh-action-sigstore-python.

[frank10gm]

It would be preferable to avoid using precompiled binaries altogether.

[dvdsk]

It would be preferable to avoid using precompiled binaries altogether.

Lets keep on subject before this issue derails and any useful comments get drowned out. If you have unrelated problems or ideas please add them to an existing issue that specifically addresses those if it exists or open a new issue.

Though it might make Microsoft very happy lets not turn GitHub into a new social media platform :)

[dtolnay]

From examining the diffoscope comparison in #2538 (comment) between my build and one inside rust:alpine (https://pkgbuild.com/~kpcyrd/2023-08-18-serde-derive-forensics/diffoscope-serde_derive-1.0.177.html), I think the differences are:

1. Initially, a different choice of nightly rustc, which means the standard library contains different code and the compiler contains different optimizations. I think build.sh should be updated to run cargo +nightly-YYYY-MM-DD build with a particular date, rather than latest +nightly.

2. Different linker — this would account for the presence or absence of a GNU build-id note in the program header, leading to various 1-word offsets.

3. Different version of the quote and syn dependencies. I checked the changes here since July 27 and I don't immediately see any that would make a difference other than in debuginfo (which is not present), so I don't know if these are a factor, but it's something that needs to be addressed anyway with a lockfile going forward.

For number 2, probably docker is the easiest way to reconcile this? If so, I would love to accept a PR that updates build.sh to run docker (/ podman?) with some usefully pinned base image and copies the result out.

[aDotInTheVoid]

Would you mind sharing what linux distro and linker version you used to build the existing precompilled binaries?

[dtolnay]

$ cat /etc/issue
Ubuntu 22.04.3 LTS \n \l

$ ld --version
GNU ld (GNU Binutils for Ubuntu) 2.38
Copyright (C) 2022 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

[aDotInTheVoid]

I've got marginally closer to reproducing, using the same rustc version and linker. It now has the same elf header. Relocations still seem different.

Code here. Sorry it's a bit of a mess. My hypothesis is different dependency versions may change relocations, even if not code. I'll investigate more tomorrow (unless someone else solves it)

[V0ldek]

Since the idea of relying on GitHub CI was thrown around, let me chime in that if the binary is indeed the direction going forward, it should be the only acceptable choice.

If serde wants to do this, do it right: maintain and verify the binary's provenance.

Implementing SLSA for Rust builds is very easy in GitHub CI (see SLSA GitHub Generator - Provenance for Rust. It would give all users an attestation that the binary included with their serde was verifiably built by this repo's CI, from available source, without tampering.

There's quite a few high-profile projects that implement SLSA (e.g. Python's flask). It would be a good thing for the Rust community to have a high-profile project that also treats supply-chain vulnerabilities with a high level of importance.

[LikeLakers2]

I support V0ldek's suggestion here. Normally I would simply leave a react, but I feel it's important to explain why I support it.

I'm a user of serde, and frankly I'm not a big fan of the precompiled executable stuff as it currently is. The main problem (but definitely not the only problem) I have with it is that it forces me to trust in a manner that makes me (and, as I can tell, many others) uncomfortable: We have to trust that dtolnay (or even a third-party) hasn't tampered with the executable that he's packaged into serde_derive. I would much rather just build serde_derive from scratch every build than need to trust in that manner.

If the precompiled executable was opt-in, that would be the end of that. However, assuming an opt-in function wasn't on the table (for whatever reason), the suggestion proposed in V0ldek's comment still feels far more comfortable to me. It allows dtolnay to precompile serde_derive without me needing to worry about if the binary was edited before distribution.

Hence, I support V0ldek's suggestion.