Provide a way for build scripts to invoke rustc the way cargo would

[RalfJung]

Problem

It is a common pattern in build scripts to run a rustc 'build probe': compile some code with rustc, see if that works, and then enable some cfg flags depending on the result. Crates use this to automatically detect if some API they optionally use is available on the current compiler. Examples of this include anhow, thiserror, eyre, and the autocfg crate which abstracts that pattern into a library.

However, invoking rustc in the right way is non-trivial. build scripts have to remember to check RUSTC and RUSTC_WRAPPER, to pass on the RUSTFLAGS (using CARGO_ENCODED_RUSTFLAGS where possible so spaces are preserved properly), and to set --target based on the TARGET env var. And even then the final rustc invocation is not always the same as what cargo itself uses, causing issues like the one described here: it is impossible to tell, from the information a build script has, whether cargo will pass --target to rustc or not, and there are some situations (rustc bootstrap and Miri, for instance) where this really matters. As a consequence, the aforementioned crates sometimes break in rustc bootstrap and Miri when they are used as dependency of a proc macro.

Proposed Solution

Cargo should just tell the build script how it will invoke rustc. I'm imagining an env var like CARGO_ENCODED_RUSTC (name to be bikeshed), which build scripts could use like this:

use std::path::Path;
use std::env;
use std::fs;
use std::process::Command;

fn probe(probe: &str) -> bool {
    let out_dir = env::var_os("OUT_DIR").unwrap();
    let probefile = Path::new(&out_dir).join("probe.rs");
    fs::write(&probefile, probe).expect("failed to write in OUT_DIR");

    let rustc = env::var("CARGO_ENCODED_RUSTC").unwrap();
    let mut rustc = rustc.split('\x1f');
    let mut cmd = Command::new(rustc.next().unwrap());
    cmd
        .args(rustc)
        .arg("--crate-name=build_probe")
        .arg("--crate-type=lib")
        .arg("--emit=metadata")
        .arg("--out-dir")
        .arg(out_dir)
        .arg(probefile);
    cmd.status().expect("failed to invoke rustc").success()
}

Notes

Does this need an RFC?

[RalfJung]

With issues like rust-lang/rust#114839, I am not even sure any more if merely providing all the right arguments to rustc is sufficient. If the build script invokes rustc, that also means cargo is unable to track the dependencies of that build in the usual way -- so even if some dependency used during the probe build changes, cargo won't know to re-run the build script.

Maybe there is a way this can be resolved with more flags though, i.e. if the build script instructed rustc to put build-dep information into a particular location and then cargo would pick it up there -- then the CARGO_ENCODED_RUSTC interface could still work.

[weihanglo]

A solution might be #12432, if there is an API can give you the ability of "build probe". And people might be happy and avoid re-implement argparse for rustc.

[epage]

Sounds like options include

• Cargo providing some solution
• The build script API capturing the generic "call rustc" pattern
• The build script API more encapsulating the build probe pattern

I'd like to throw out another one: we focus on #[cfg(accessible)]. That seems like the best long-term strategy, for what it can encompass. For things not covered by it, I suspect we should explore what those should be.

[RalfJung]

cfg(accessible) does not help when a nightly API changes its type. Then the item might be "accessible" but the crate will still fail to build. Build probes test more than just whether the item exists.

• The build script API capturing the generic "call rustc" pattern
• The build script API more encapsulating the build probe pattern

These will require cooperation with cargo, since currently build scripts just do not have the information required to generate the right rustc invocation.

[epage]

How big of a use case is supporting multiple flavors of a nightly feature? To me, that sounds like a relatively small use case. Could you expand your uses for doing that?

[RalfJung]

How big of a use case is supporting multiple flavors of a nightly feature? To me, that sounds like a relatively small use case. Could you expand your uses for doing that?

I did not talk about supporting multiple flavors of a nightly feature. I talked about what happens when a nightly feature changes the type of an item. Code written cfg(accessible) will now fail to build, because the item still exists but has the wrong type. The entire point of build probes is to avoid that situation, where when the nightly feature changes in a way that client code needs to be adjusted, it just gets disabled.

[epage]

For myself, I've grown increasingly frustrated with build probes being used in this way (implicit opt-in to nightly features) to the point that I feel its an anti-pattern. Also been discussing that in dtolnay/anyhow#323.