Skip to content

rust_extern_with_linkage___dso_handle+0x0): undefined reference to `__dso_handle' #140955

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tkennedy1-godaddy opened this issue May 12, 2025 · 23 comments
Labels
A-cross Area: Cross compilation A-linkage Area: linking into static, shared libraries and binaries C-bug Category: This is a bug. E-needs-bisection Call for participation: This issue needs bisection: https://github.com/rust-lang/cargo-bisect-rustc E-needs-mcve Call for participation: This issue has a repro, but needs a Minimal Complete and Verifiable Example needs-triage This issue may need triage. Remove it if it has been sufficiently triaged.

Comments

@tkennedy1-godaddy
Copy link

I tried this code:

Trying to cross compile (from aarch64 to x86_86 on linux)

I expected to see this happen: explanation

The software compiled correctly.

Instead, this happened: explanation

The linker failed on trying to link Rust's libstd.

  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: /usr/x86_64-linux-gnu/bin/ld: warning: cannot find entry symbol _start; defaulting to 0000000000005030
          /usr/x86_64-linux-gnu/bin/ld: /home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-f6265b21db1f990f.rlib(std-f6265b21db1f990f.std.d9e466a2d75004a2-cgu.0.rcgu.o):(.data._
rust_extern_with_linkage___dso_handle+0x0): undefined reference to `__dso_handle'
          /usr/x86_64-linux-gnu/bin/ld: /home/debian/envoy-cli/target/x86_64-unknown-linux-gnu/debug/deps/oktaplz-a2cfe24976663532: hidden symbol `__dso_handle' isn't defined
          /usr/x86_64-linux-gnu/bin/ld: final link failed: bad value

Meta

rustc --version --verbose:

1.81
Backtrace

<backtrace>

This does work correctly on rust 1.74.0 on the same system. I have been unable to test other versions yet, nor create a self-contained example.

The build command for this is:

RUSTFLAGS="-Clinker=/usr/x86_64-linux-gnu/bin/ld -L/lib/gcc-cross/x86_64-linux-gnu/12 -L/lib/gcc-cross/x86_64-linux-gnu/11" PKG_CONFIG_SYSROOT_DIR=/usr/x86_64-linux-gnu CC=clang cargo build --target x86_64-unknown-linux-gnu
@tkennedy1-godaddy tkennedy1-godaddy added the C-bug Category: This is a bug. label May 12, 2025
@rustbot rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label May 12, 2025
@tkennedy1-godaddy
Copy link
Author

The list of dependencies for the project are:

[dependencies]
base64 = "0.21"
chrono = "0.4"
color-eyre = "0.6"
ctrlc = "3.1"
dirs = "3.0"
futures = "0.3"
futures-channel = "0.3"
keyring = "0.10"
open = "1.4"
otpcli-tkennedy1 = "1.0"
promptly = "0.3"
rpassword = "5.0"
rusoto_core = "0.47"
rusoto_sts = "0.47"
rusoto_iam = "0.47"
rusoto_ssm = "0.47"
rusoto_secretsmanager = "0.47"
select = "0.6"
sentry = "0.23"
sentry-slog = "0.23"
serde-xml-rs = "0"
serde_json = "1.0"
skim = "0.10"
slog = "2.7"
slog-stdlog = "4.0"
slog-term = "2.7"
toml = "0.5"
url = "2"
warp = "0.3"
indicatif = "0.16"
rust-ini = "0.21.0"

@mati865
Copy link
Contributor

mati865 commented May 13, 2025

You didn't provide crt objects to the linker. When calling linker directly you have to take care of it.

@tkennedy1-godaddy
Copy link
Author

I didn’t on the version that compiled. Same exact build command, same dependencies installed. Builds on 1.74, fails on 1.81.

What would be the proper library to include on the linker for version 1.81?

@moxian
Copy link
Contributor

moxian commented May 13, 2025

@tkennedy1-godaddy would you like to run cargo bisect-rustc to narrow down the rustc commit range that intoduced this change in behavior? (I don't know much about CRT myself, but bisected PR description is often insightful in these cases, in my experience)

@rustbot label: +A-linkage +A-cross

@rustbot rustbot added A-cross Area: Cross compilation A-linkage Area: linking into static, shared libraries and binaries labels May 13, 2025
@jieyouxu jieyouxu added E-needs-bisection Call for participation: This issue needs bisection: https://github.com/rust-lang/cargo-bisect-rustc E-needs-mcve Call for participation: This issue has a repro, but needs a Minimal Complete and Verifiable Example labels May 13, 2025
@tkennedy1-godaddy
Copy link
Author

@moxian yes i can get to that this afternoon pdt

@tkennedy1-godaddy
Copy link
Author

@moxian

searched toolchains nightly-2023-10-01 through nightly-2025-05-13


********************************************************************************
Regression in nightly-2024-03-18
********************************************************************************

fetching https://static.rust-lang.org/dist/2024-03-17/channel-rust-nightly-git-commit-hash.txt
nightly manifest 2024-03-17: 40 B / 40 B [===============================================================================================================================================================================] 100.00 % 17.86 KB/s
converted 2024-03-17 to 766bdce744d531267d53ba2a3f9ffcda69fb9b17
fetching https://static.rust-lang.org/dist/2024-03-18/channel-rust-nightly-git-commit-hash.txt
nightly manifest 2024-03-18: 40 B / 40 B [==============================================================================================================================================================================] 100.00 % 250.13 KB/s
converted 2024-03-18 to eb45c844407968ea54df0d9870ebce9e3235b706
looking for regression commit between 2024-03-17 and 2024-03-18
fetching (via remote github) commits from max(766bdce744d531267d53ba2a3f9ffcda69fb9b17, 2024-03-15) to eb45c844407968ea54df0d9870ebce9e3235b706
ending github query because we found starting sha: 766bdce744d531267d53ba2a3f9ffcda69fb9b17
get_commits_between returning commits, len: 10
  commit[0] 2024-03-16: Auto merge of #122602 - ChrisDenton:rollup-f0vumtg, r=ChrisDenton
  commit[1] 2024-03-16: Auto merge of #122594 - Mark-Simulacrum:bump-version, r=Mark-Simulacrum
  commit[2] 2024-03-17: Auto merge of #122607 - fmease:rollup-ozl1eeq, r=fmease
  commit[3] 2024-03-17: Auto merge of #121885 - reitermarkus:generic-nonzero-inner, r=oli-obk,wesleywiser
  commit[4] 2024-03-17: Auto merge of #122321 - majaha:mingw_ci_new, r=Mark-Simulacrum
  commit[5] 2024-03-17: Auto merge of #122611 - rust-lang:cargo_update, r=clubby789
  commit[6] 2024-03-17: Auto merge of #122625 - matthiaskrgr:rollup-ue4dmnx, r=matthiaskrgr
  commit[7] 2024-03-17: Auto merge of #122628 - lnicola:sync-from-ra, r=lnicola
  commit[8] 2024-03-17: Auto merge of #122637 - matthiaskrgr:rollup-bczj5bp, r=matthiaskrgr
  commit[9] 2024-03-17: Auto merge of #122653 - matthiaskrgr:rollup-28h37ym, r=matthiaskrgr
ERROR: no CI builds available between 766bdce744d531267d53ba2a3f9ffcda69fb9b17 and eb45c844407968ea54df0d9870ebce9e3235b706 within last 167 days

@tkennedy1-godaddy
Copy link
Author

Amazingly this build system was finalized in Feb 2024, so it looks like I really dodged a bullet there :)

@mati865
Copy link
Contributor

mati865 commented May 13, 2025

I didn’t on the version that compiled. Same exact build command, same dependencies installed. Builds on 1.74, fails on 1.81.

I think something else must have pulled the necessary objects.
The fact it worked doesn't imply it was supported.

What would be the proper library to include on the linker for version 1.81?

Just as with C.
Try to link C executable but pass -### to GCC or Clang, it'll print all the libraries and objects that are required. Rust should pull required libraries via https://github.com/rust-lang/libc/blob/a1e464928440edefb96f1d3131210a98e0e9eb54/src/unix/mod.rs#L465-L470, but you'll have to supply the objects (crt*.o) to the linker.

@tkennedy1-godaddy
Copy link
Author

tkennedy1-godaddy commented May 13, 2025

The crt*.0 libraries are in the path being provided to the linker though.

debian@debian:~/envoy-cli$ ls /lib/gcc-cross/x86_64-linux-gnu/12/*crt*
/lib/gcc-cross/x86_64-linux-gnu/12/crtbegin.o   /lib/gcc-cross/x86_64-linux-gnu/12/crtend.o       /lib/gcc-cross/x86_64-linux-gnu/12/crtoffloadbegin.o  /lib/gcc-cross/x86_64-linux-gnu/12/crtprec32.o
/lib/gcc-cross/x86_64-linux-gnu/12/crtbeginS.o  /lib/gcc-cross/x86_64-linux-gnu/12/crtendS.o      /lib/gcc-cross/x86_64-linux-gnu/12/crtoffloadend.o    /lib/gcc-cross/x86_64-linux-gnu/12/crtprec64.o
/lib/gcc-cross/x86_64-linux-gnu/12/crtbeginT.o  /lib/gcc-cross/x86_64-linux-gnu/12/crtfastmath.o  /lib/gcc-cross/x86_64-linux-gnu/12/crtoffloadtable.o  /lib/gcc-cross/x86_64-linux-gnu/12/crtprec80.o
debian@debian:~/envoy-cli$ ls /lib/gcc-cross/x86_64-linux-gnu/11/*crt*
/lib/gcc-cross/x86_64-linux-gnu/11/crtbegin.o   /lib/gcc-cross/x86_64-linux-gnu/11/crtend.o       /lib/gcc-cross/x86_64-linux-gnu/11/crtoffloadbegin.o  /lib/gcc-cross/x86_64-linux-gnu/11/crtprec32.o
/lib/gcc-cross/x86_64-linux-gnu/11/crtbeginS.o  /lib/gcc-cross/x86_64-linux-gnu/11/crtendS.o      /lib/gcc-cross/x86_64-linux-gnu/11/crtoffloadend.o    /lib/gcc-cross/x86_64-linux-gnu/11/crtprec64.o
/lib/gcc-cross/x86_64-linux-gnu/11/crtbeginT.o  /lib/gcc-cross/x86_64-linux-gnu/11/crtfastmath.o  /lib/gcc-cross/x86_64-linux-gnu/11/crtoffloadtable.o  /lib/gcc-cross/x86_64-linux-gnu/11/crtprec80.o

These paths are passed in via the -L/lib/gcc-cross/x86_64-linux-gnu/12 -L/lib/gcc-cross/x86_64-linux-gnu/11 parts of the RUSTFLAGS environment variable.

And more importantly, if I leave them out, the linker is unable to link to gcc_s, which is is one of the libraries located in those paths. Here's the error if I omit those -L flags:

  = note: /usr/x86_64-linux-gnu/bin/ld: cannot find -lgcc_s: No such file or directory

And here's the library it's looking for.

debian@debian:~/envoy-cli$ ls /lib/gcc-cross/x86_64-linux-gnu/*/libgcc_s.so
/lib/gcc-cross/x86_64-linux-gnu/11/libgcc_s.so  /lib/gcc-cross/x86_64-linux-gnu/12/libgcc_s.so

The fact it worked doesn't imply it was supported.

The fact that we're referring to __dso_handle, which is both present in the all versions of the stdlib:

debian@debian:~/.rustup/toolchains$ for LIB in $(find -type f -name *libstd-*rlib); do echo $LIB; readelf -s $LIB | grep __dso_handle; done
./1.74-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-84a09a48c3c6b54d.rlib
  7808: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle
./1.74-aarch64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-b149a04e58514815.rlib
  5020: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle
./1.69-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-946b15357ac77df4.rlib
  7520: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle
./1.61-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-7d5a620ab77655ab.rlib
  7663: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle
./stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-d99d33bfbb8d1b5c.rlib
  6953: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle
./stable-aarch64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-f6265b21db1f990f.rlib
  4486: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle
./1.81-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-2bf0b2a5e0a60917.rlib
  6475: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle
./1.81-aarch64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-1c4b19562077c20d.rlib
  4189: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle
./1.67-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-349359eac2fa563a.rlib
  7659: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle
./bisector-nightly-2025-04-07-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-998e3cd732077b76.rlib
  6919: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle
./bisector-nightly-2025-04-07-aarch64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-77b6a10324684199.rlib
  4462: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle

And has been an issue in the past, specifically in Rust makes me think something else might be going on here other than "it wasn't supported and now it's not supported at all".

It might be a "it wasn't supposed to be supported and so it was an accident that it was supported" type thing, but given that even providing the correct libraries still causes this to break and the information in that gist, I'm not thoroughly convinced.

@mati865
Copy link
Contributor

mati865 commented May 13, 2025

The crt*.0 libraries are in the path being provided to the linker though.

These paths are passed in via the -L/lib/gcc-cross/x86_64-linux-gnu/12 -L/lib/gcc-cross/x86_64-linux-gnu/11 parts of the RUSTFLAGS environment variable.

The linker doesn't link all everything it can find inside directories provided with -L, you have to explicitly tell it what to link.

And more importantly, if I leave them out, the linker is unable to link to gcc_s, which is is one of the libraries located in those paths. Here's the error if I omit those -L flags:

  = note: /usr/x86_64-linux-gnu/bin/ld: cannot find -lgcc_s: No such file or directory

And here's the library it's looking for.

debian@debian:~/envoy-cli$ ls /lib/gcc-cross/x86_64-linux-gnu/*/libgcc_s.so
/lib/gcc-cross/x86_64-linux-gnu/11/libgcc_s.so  /lib/gcc-cross/x86_64-linux-gnu/12/libgcc_s.so

Yep, that's a library linked by libc crate I mentioned before. Rustc tells the linker to link it (and some others), but it doesn't tell the linker to link runtime objects (crt*.o).

And has been an issue in the past, specifically in Rust makes me think something else might be going on here other than "it wasn't supported and now it's not supported at all".

It might be a "it wasn't supposed to be supported and so it was an accident that it was supported" type thing, but given that even providing the correct libraries still causes this to break and the information in that gist, I'm not thoroughly convinced.

I cannot give meaningful insight on that part, but maybe some Rust's ELF expert can help.

@tkennedy1-godaddy
Copy link
Author

@mati865 no, the -L (upper case command) includes all libraries in the same directory. (This is the same in both GNU ld and LLVM's lld)

debian@debian:~/envoy-cli$ ld --help | grep -C 5 -- -L
  -I PROGRAM, --dynamic-linker PROGRAM
                              Set PROGRAM as the dynamic linker to use
  --no-dynamic-linker         Produce an executable with no program interpreter header
  -l LIBNAME, --library LIBNAME
                              Search for library LIBNAME
  -L DIRECTORY, --library-path DIRECTORY
                              Add DIRECTORY to library search path
  --sysroot=<DIRECTORY>       Override the default sysroot location
  -m EMULATION                Set emulation
  -M, --print-map             Print map file on standard output
  -n, --nmagic                Do not page align data

I did, however, try that, both with a wildcard to let shell-globbing fill it in, and once with all the crt libraries provided, to the same exact result as above -- the __dso_handle is still an undefined reference.

@tkennedy1-godaddy
Copy link
Author

(and yes, I also tried prefixing them with -l to different results, but still failures)

@mati865
Copy link
Contributor

mati865 commented May 16, 2025

@tkennedy1-godaddy no, -L does not include any libraries with any reasonable linker (ld.bfd, ld.gold, LLD, Mold, Wild, ...).

The help that you quoted confirms that: Add DIRECTORY to library search path.
When you use -lXYZ (lowercase -l) the linker will search for libXYZ or XYZ in library search paths that were provided to it via -L/some/path (uppercase -L`).

@tkennedy1-godaddy
Copy link
Author

The problem is that crt isn't a "library" per se. The crt stuff is part the standard C runtime (hence crt). This library is included by default in linking/compiling (you have to specifically disable it or compile your package statically).

I am filing this bug because the rust compiler is supposed to handle this for you specifically as well. Given that it worked in the past, and no longer works without a specific reference in the changesets makes me strongly feel this is a regression and not something that is "working as intended."

Passing in -lcrt does nothing because its not a library. The rust compiler is supposed to automatically provide this (and linking this application statically, while possible, is a bit of a different hill to climb). Linking to -lgcc does not provide the symbol, and there is no libstd (there is a libstdc++, but that does not solve this problem when specifically linked to either).

@bjorn3
Copy link
Member

bjorn3 commented May 18, 2025

When compiling for any Unix target, you should avoid directly invoking the linker. Rather you should use gcc or clang as linker driver. They will handle passing the linker search path, any extra libraries and object files that need to be linked, set the dynamic linker, any exploit mitigations and whatever other flags are necessary to link. Also even gcc and clang don't directly invoke the linker on many systems, rather they invoke collect2 which is another wrapper around the linker that is on some systems necessary to get global constructors working. For example compiling on my debian system gcc will run the following:

/usr/lib/gcc/x86_64-linux-gnu/12/collect2 -plugin /usr/lib/gcc/x86_64-linux-gnu/12/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/12/lto-wrapper -plugin-opt=-fresolution=/tmp/ccPbxMPB.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu --as-needed -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie /usr/lib/gcc/x86_64-linux-gnu/12/../../../x86_64-linux-gnu/Scrt1.o /usr/lib/gcc/x86_64-linux-gnu/12/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/12/crtbeginS.o -L/usr/lib/gcc/x86_64-linux-gnu/12 -L/usr/lib/gcc/x86_64-linux-gnu/12/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/12/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/12/../../.. a.o -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/x86_64-linux-gnu/12/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/12/../../../x86_64-linux-gnu/crtn.o

If you are cross-compiling you should use the gcc or clang that is part of the cross-compilation toolchain as -Clinker argument.

@tkennedy1-godaddy
Copy link
Author

I'm not invoking the linker directly, rather telling (via the -Clinker and -Clink-arg directives) the compiler which linker I need it to invoke. It's still passing in a laundry list of files (the output from the linker step is... very large, in addition to all the object files created, it's passing in

"-Bdynamic" "-lssl" "-lcrypto" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "--eh-frame-hdr" "-z" "noexecstack" "-L" "/home/debian/.rustup/toolchains/1.81-aarch64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/home/debian/envoy-cli/target/x86_64-unknown-linux-gnu/debug/deps/envoy_cli-3e878a4ec98cd548" "--gc-sections" "-pie" "-z" "relro" "-z" "now"

(It also doesn't matter if I use clang or x86_64-linux-gnu-gcc as the compiler, both result in the same linker error)

@mati865
Copy link
Contributor

mati865 commented May 19, 2025

This is basically invoking the static linker directly from rustc, rather than going through the supported way of letting GCC+collect2 or Clang invoke the linker.

@tkennedy1-godaddy
Copy link
Author

How would you recommend I explain to the compiler that it needs to invoke a specific linker for this platform?

There are several ways to tell cargo what to do, but even if I do this:

RUSTFLAGS="-L/usr/lib/x86_64-linux-gnu -L/lib/gcc-cross/x86_64-linux-gnu/12" PKG_CONFIG_SYSROOT_DIR=/ CC_x86_64_unknown_linux_gnu=x86_64-linux-gnu-gcc CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-linux-gnu-ld cargo build --target x86_64-unknown-linux-gnu

I still get the same error with the __dso_handle.

If I remove all the overrides for the compiler/linker I'm left with the obnoxious linker issue of

cc: error: unrecognized command-line option ‘-m64’

Even if I use the not-documented-as-far-as-I-can-tell CROSS_COMPILE=x86_64-linux-gnu env variable, I still get that -m64 error. That error only disappears when I tell it to use a specific linker. (Even using the awkward CC_x86_64_unknown_gnu, AR_x86_64_unknown_gnu variables doesn't fix the problem.)

HOWEVER.

Through tons of "reading posts" I happened across someone using rust-lld as their linker. Given lld is supposed to be like clang and cross platform, I figured I'd give this a shot (since using lld directly causes issues with missing libraries) and low and behold:

RUSTFLAGS="-Clinker=rust-lld -L/usr/lib/x86_64-linux-gnu -L/lib/gcc-cross/x86_64-linux-gnu/12" PKG_CONFIG_SYSROOT_DIR=/ CC=clang  cargo build --target x86_64-unknown-linux-gnu

[lots of logs omitted}

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 31.91s
debian@debian:~/envoy-cli$ file target/x86_64-unknown-linux-gnu/debug/envoy-cli
target/x86_64-unknown-linux-gnu/debug/envoy-cli: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, with debug_info, not stripped

So @mati865 I do need to provide the linker via the -Clinker flag, I just need to get the right linker in there... (FWIW I can use both clang and x86_64-unknown-gnu-gcc as the compiler and rust-lld as the linker)

@bjorn3
Copy link
Member

bjorn3 commented May 20, 2025

The right linker to pass as argument to -Clinker is either clang or x86_64-linux-gnu-gcc (or however the cross-compilation version of gcc is called) at your option.

@mati865
Copy link
Contributor

mati865 commented May 20, 2025

How would you recommend I explain to the compiler that it needs to invoke a specific linker for this platform?

Like I and bjorn3 already said, the right way to link is by calling gcc or clang.

There are several ways to tell cargo what to do, but even if I do this:

RUSTFLAGS="-L/usr/lib/x86_64-linux-gnu -L/lib/gcc-cross/x86_64-linux-gnu/12" PKG_CONFIG_SYSROOT_DIR=/ CC_x86_64_unknown_linux_gnu=x86_64-linux-gnu-gcc CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-linux-gnu-ld cargo build --target x86_64-unknown-linux-gnu

I still get the same error with the __dso_handle.

This is expected because you still call ld directly.

If I remove all the overrides for the compiler/linker I'm left with the obnoxious linker issue of

cc: error: unrecognized command-line option ‘-m64’

That means your cc is either 32-bit or non x86 compiler. In which case, you need to use x86_64-linux-gnu-gcc or clang with the right target argument (dunno if rustc automatically passes correct one) as bjorn3 said.

That error only disappears when I tell it to use a specific linker. (Even using the awkward CC_x86_64_unknown_gnu, AR_x86_64_unknown_gnu variables doesn't fix the problem.)

rustc doesn't pass -m64 when not linking via gcc/clang:

base.add_pre_link_args(LinkerFlavor::Gnu(Cc::Yes, Lld::No), &["-m64"]);

because static linkers don't support this argument. But to link via static linker you have to manually pass correct runtime objects in correct order, which you didn't do. Hence, the __dso_handle error.

Given lld is supposed to be like clang and cross platform, I figured I'd give this a shot (since using lld directly causes issues with missing libraries)

No, lld (or rust-lld) behaves like ld, it has nothing to do with clang. That's why using LLD directly gave you the same errors as with LD.

low and behold:

RUSTFLAGS="-Clinker=rust-lld -L/usr/lib/x86_64-linux-gnu -L/lib/gcc-cross/x86_64-linux-gnu/12" PKG_CONFIG_SYSROOT_DIR=/ CC=clang  cargo build --target x86_64-unknown-linux-gnu

[lots of logs omitted}

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 31.91s
debian@debian:~/envoy-cli$ file target/x86_64-unknown-linux-gnu/debug/envoy-cli
target/x86_64-unknown-linux-gnu/debug/envoy-cli: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, with debug_info, not stripped

So @mati865 I do need to provide the linker via the -Clinker flag, I just need to get the right linker in there... (FWIW I can use both clang and x86_64-unknown-gnu-gcc as the compiler and rust-lld as the linker)

This is definitely not a supported way to use rust-lld for *-linux-gnu targets. It might break tomorrow, or remain working for a few years.

You can verify used linker with readelf -p .comment target/x86_64-unknown-linux-gnu/debug/envoy-cli.

@tkennedy1-godaddy
Copy link
Author

@mati865 at no time in this thread did you suggest that I should do that, except possibly in #140955 (comment).

@bjorn3 I appreciate the straight forward advice.

This feels like all of this information should be properly documented somewhere. I'm happy to do that -- is there a recommendation for where this should all go? Most of this stuff was cobbled together from blog posts or threads on the rust discussions where most of these solutions are proffered as the correct way to go about this.

@mati865
Copy link
Contributor

mati865 commented May 20, 2025

@tkennedy1-godaddy I assumed it was a deliberate decision to avoid calling GCC or Clang, so I focused on the last thing you were missing and also missed the fact you were not using x86_64 host (which would be more obvious from rustc --version --verbose).

This feels like all of this information should be properly documented somewhere. I'm happy to do that -- is there a recommendation for where this should all go? Most of this stuff was cobbled together from blog posts or threads on the rust discussions where most of these solutions are proffered as the correct way to go about this.

I just looked it up, and indeed, the official documentation is basically absent (or not found by Google and Brave Search). I think there used to be something, but maybe the memory fails me. Definitely something worth acting on.

Also, it could be helpful if more targets included linker in their spec, like windows-gnu/windows-gnullvm do. Then users don't have to bother with setting the correct linker. Just do cargo b --target <target> and in case of simple crates that's all.

FWIW the best way to configure linker when cross-compiling is via CARGO_TARGET_<triple>_LINKER, using RUSTFLAGS=-Clinker=XYZ would break proc-macros if they are present.
So it should work if you do: CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-linux-gnu-gcc PKG_CONFIG_SYSROOT_DIR=/ cargo build --target x86_64-unknown-linux-gnu should do the trick.

@Noratrieb
Copy link
Member

I improved the docs for -Clinker in #141554

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-cross Area: Cross compilation A-linkage Area: linking into static, shared libraries and binaries C-bug Category: This is a bug. E-needs-bisection Call for participation: This issue needs bisection: https://github.com/rust-lang/cargo-bisect-rustc E-needs-mcve Call for participation: This issue has a repro, but needs a Minimal Complete and Verifiable Example needs-triage This issue may need triage. Remove it if it has been sufficiently triaged.
Projects
None yet
Development

No branches or pull requests

7 participants