[build] Add a new `--cross-compile-build-swift-tools` flag to disable cross-compiling the compiler #38441

finagolfin · 2021-07-16T18:51:13Z

This is useful when building cross-compilation toolchains where you want the stdlib and corelibs cross-compiled but don't want the Swift compiler cross-compiled too with --cross-compile-hosts. This pull demonstrates using the new flag for the Android build presets.

I'm submitting this after discussing cross-compiling the corelibs on the Android CI with @drodriguez, as the easiest way to do so after #33724 is by using --cross-compile-hosts but we don't want to cross-compile the Swift compiler too.

To build this, I first had to unpack three Termux packages for Android, libandroid-spawn, libcurl, and libxml2 (I also unpacked libicu but the Android CI does that differently) as detailed here, because those libraries are dependencies of swift-corelibs-foundation.

I then passed that swift-android-aarch64-24-sdk path into this build-script invocation with the July 9 trunk source snapshot to build an Android cross-compilation toolchain that includes the corelibs:

./swift/utils/build-script -RA --android --android-ndk /home/butta/src/android-ndk-r21e/
--android-arch aarch64 --android-api-level 24
--android-icu-uc /home/butta/swift-android-aarch64-24-sdk/usr/lib/libicuuc.so
--android-icu-uc-include /home/butta/swift-android-aarch64-24-sdk/usr/include/
--android-icu-i18n /home/butta/swift-android-aarch64-24-sdk/usr/lib/libicui18n.so
--android-icu-i18n-include /home/butta/swift-android-aarch64-24-sdk/usr/include/
--android-icu-data /home/butta/swift-android-aarch64-24-sdk/usr/lib/libicudata.so
--xctest --cross-compile-hosts=android-aarch64 --install-swift --install-libdispatch
--install-foundation --install-xctest
--cross-compile-deps-path=/home/butta/swift-android-aarch64-24-sdk
--cross-compile-build-swift-tools=0 --llvm-ninja-targets-for-cross-compile-hosts=help

swift-corelibs-foundation recently added a dependency on some posix_spawn functions on Android, which as discussed on the forum isn't officially supported before Android API 28, so I linked against Termux's backported library with these additional patches:

diff --git a/swift/stdlib/public/Platform/CMakeLists.txt b/swift/stdlib/public/Platform/CMakeLists.txt
index a7a60a063d9..8894800509a 100644
--- a/swift/stdlib/public/Platform/CMakeLists.txt
+++ b/swift/stdlib/public/Platform/CMakeLists.txt
@@ -99,6 +99,7 @@ foreach(sdk ${SWIFT_SDKS})
         OUTPUT "${glibc_modulemap_out}"
         FLAGS
             "-DCMAKE_SDK=${sdk}"
+            "-DSDK_INCLUDE_PATH=${SWIFT_${sdk}_${arch}_ICU_UC_INCLUDE}"
             "-DGLIBC_INCLUDE_PATH=${SWIFT_SDK_${sdk}_ARCH_${arch}_LIBC_INCLUDE_DIRECTORY}"
             "-DGLIBC_ARCH_INCLUDE_PATH=${SWIFT_SDK_${sdk}_ARCH_${arch}_LIBC_ARCHITECTURE_INCLUDE_DIRECTORY}")
 
@@ -142,6 +143,7 @@ foreach(sdk ${SWIFT_SDKS})
         SOURCE "${glibc_modulemap_source}"
         OUTPUT "${glibc_sysroot_relative_modulemap_out}"
         FLAGS "-DCMAKE_SDK=${sdk}"
+              "-DSDK_INCLUDE_PATH=${SWIFT_${sdk}_${arch}_ICU_UC_INCLUDE}"
               "-DGLIBC_INCLUDE_PATH=${absolute_libc_include_path}"
               "-DGLIBC_ARCH_INCLUDE_PATH=${absolute_libc_arch_include_path}")
 
diff --git a/swift/stdlib/public/Platform/bionic.modulemap.gyb b/swift/stdlib/public/Platform/bionic.modulemap.gyb
index e44f9082653..09a92df50bb 100644
--- a/swift/stdlib/public/Platform/bionic.modulemap.gyb
+++ b/swift/stdlib/public/Platform/bionic.modulemap.gyb
@@ -185,7 +185,7 @@ module SwiftGlibc [system] {
       export *
     }
     module spawn {
-      header "${GLIBC_INCLUDE_PATH}/spawn.h"
+      header "${SDK_INCLUDE_PATH}/spawn.h"
       export *
     }
     module syslog {
diff --git a/swift-corelibs-foundation/Sources/Foundation/CMakeLists.txt b/swift-corelibs-foundation/Sources/Foundation/CMakeLists.txt
index 016bf294..1473f4dc 100644
--- a/swift-corelibs-foundation/Sources/Foundation/CMakeLists.txt
+++ b/swift-corelibs-foundation/Sources/Foundation/CMakeLists.txt
@@ -167,6 +167,10 @@ if(CMAKE_SYSTEM_NAME STREQUAL Windows)
     $<TARGET_OBJECTS:CoreFoundationResources>)
 elseif(NOT CMAKE_SYSTEM_NAME STREQUAL Darwin)
   target_link_options(Foundation PRIVATE "SHELL:-no-toolchain-stdlib-rpath")
+  if(CMAKE_SYSTEM_NAME STREQUAL Android)
+    target_link_libraries(Foundation PRIVATE android-spawn)
+    target_link_directories(Foundation PUBLIC ${CMAKE_FIND_ROOT_PATH}/usr/lib)
+  endif()
 endif()

Alternatively, since the build only fails when linking plutil against libFoundation.a and we can't run the tests for the cross-compiled corelibs, we could hack out building a static plutil or revert the small pull adding that dependency, as done in vgorloff/swift-everywhere-toolchain#116.

@drodriguez, let me know what you think.

utils/build-presets.ini

finagolfin · 2021-07-16T19:09:54Z

utils/build-script-impl

@@ -1909,6 +1910,12 @@ for host in "${ALL_HOSTS[@]}"; do
                    cmake_options=(
                        "${cmake_options[@]}"
                        -DLLVM_TABLEGEN=$(build_directory "${LOCAL_HOST}" llvm)/bin/llvm-tblgen
+                        -DSWIFT_INCLUDE_TOOLS:BOOL=$(true_false "${CROSS_COMPILE_BUILD_SWIFT_TOOLS}")


This changes the way --build-swift-tools works, where before it would control building both native and cross-compiled host tools, but now it only affects the native ones. Since very few people were likely using this flag, I think it's better to split the functionality up like this for cross-compiling the host tools, rather than keeping --build-swift-tools controlling both even after --cross-compile-build-swift-tools is added.

drodriguez

I will have to re-read and play around with the posix_spawn problems to understand them. In general, I would say that things should work with vanilla Android, without special libraries or headers.

For the time being, let's see if this works in macOS and Linux and we are not going to break another platform by mistake.

utils/build-presets.ini

drodriguez · 2021-07-21T01:13:41Z

@swift-ci please test

swift-ci · 2021-07-21T02:36:53Z

Build failed
Swift Test Linux Platform
Git Sha - 90f4c15bf87f039691be43ee86057637245847b2

swift-ci · 2021-07-21T05:14:44Z

Build failed
Swift Test OS X Platform
Git Sha - 90f4c15bf87f039691be43ee86057637245847b2

finagolfin · 2021-07-21T07:27:06Z

I will have to re-read and play around with the posix_spawn problems to understand them. In general, I would say that things should work with vanilla Android, without special libraries or headers.

The problem is that swiftlang/swift-corelibs-foundation#2928 added a dependency on some posix_spawnattr_* functions that were not added to Android till API 28, whereas this preset builds against API 21. There are various ways we could remedy that, such as asking CoreFoundation if it found those APIs and only using them if available?

let's see if this works in macOS and Linux and we are not going to break another platform by mistake

A single failing test because I added a new path variable to the Android preset, didn't know I had to do something else for that, will fix it.

finagolfin · 2021-07-25T15:30:28Z

I added a line to fix the single failing test on the CI and fixed the preset arch issue we discussed above.

@drodriguez, I think this is ready for you to try on your local version of the Android CI. I think the only remaining issue will be adding library dependencies that the corelibs depend on to the Android CI, which will require the Termux packages I mentioned above for the Android corelibs and perhaps the native Ubuntu packages to build the native linux corelibs, if those Ubuntu packages like libcurl aren't already installed on the Android CI.

utils/build-presets.ini

drodriguez · 2021-07-28T21:55:50Z

The problem is that apple/swift-corelibs-foundation#2928 added a dependency on some posix_spawnattr_* functions that were not added to Android till API 28, whereas this preset builds against API 21. There are various ways we could remedy that, such as asking CoreFoundation if it found those APIs and only using them if available?

I remember that code 😀. I don't think anyone will object if we switch (again) all the direct usages of posix_spawnattr_* into some _CFPosixSpawn* equivalents. The treatment of the attributes was not done, but as long as it is limited to just POSIX_SPAWN_SETPGROUP, maybe it is doable with setpgid(0, /* the pgroup value */) in _CFPosixSpawnImplPre28 after the comment // This is the child (probably before the actions are processed).

drodriguez · 2021-07-28T21:56:00Z

@swift-ci please test

finagolfin · 2021-07-28T22:49:54Z

Alright, I will look into implementing that CF polyfill on top of that posix_spawn wrapper you added a couple years ago and submit a pull for that.

swift-ci · 2021-07-28T23:29:49Z

Build failed
Swift Test Linux Platform
Git Sha - 26f0200b8d9f5723533cb763ec2392c39e3de6f9

swift-ci · 2021-07-28T23:49:19Z

Build failed
Swift Test OS X Platform
Git Sha - 26f0200b8d9f5723533cb763ec2392c39e3de6f9

finagolfin · 2021-07-29T06:56:25Z

Gah, playing whack-a-mole with the non-Android CI: I had missed that preset variables need that s at the end, actually ran the single failing test this time to make sure it works now.

drodriguez · 2021-07-29T15:49:51Z

@swift-ci please test

… cross-compiling the compiler This is useful when building cross-compilation toolchains where you want the stdlib and corelibs cross-compiled but don't want the Swift compiler cross-compiled too with '--cross-compile-hosts'.

finagolfin · 2025-05-28T20:25:55Z

@swift-ci smoke test

finagolfin · 2025-05-28T20:28:18Z

I've stripped this down to just adding the single build flag, with no changes to any of the Android presets, so the Foundation considerations no longer apply.

@drodriguez, if you have any feedback on this final version, let me know.

finagolfin · 2025-05-29T13:46:20Z

@swift-ci smoke test macos

finagolfin · 2025-06-03T09:26:25Z

Merging since this won't break anything, just wondered if Daniel had any input, still listening if he comes up with anything later.

…hen cross-compiling Foundation macros (#82163) Follow-on to #38441, missed this there.

finagolfin commented Jul 16, 2021

View reviewed changes

utils/build-presets.ini Outdated Show resolved Hide resolved

finagolfin commented Jul 16, 2021

View reviewed changes

utils/build-presets.ini Outdated Show resolved Hide resolved

finagolfin commented Jul 16, 2021

View reviewed changes

utils/build-presets.ini Outdated Show resolved Hide resolved

finagolfin commented Jul 16, 2021

View reviewed changes

drodriguez reviewed Jul 21, 2021

View reviewed changes

utils/build-presets.ini Outdated Show resolved Hide resolved

utils/build-presets.ini Outdated Show resolved Hide resolved

finagolfin force-pushed the cross-compiler branch from 90f4c15 to 40ef38d Compare July 25, 2021 15:05

finagolfin force-pushed the cross-compiler branch from 40ef38d to 26f0200 Compare July 25, 2021 21:12

finagolfin commented Jul 25, 2021

View reviewed changes

utils/build-presets.ini Outdated Show resolved Hide resolved

finagolfin force-pushed the cross-compiler branch from 26f0200 to fd79ea8 Compare July 29, 2021 06:52

finagolfin mentioned this pull request Aug 27, 2021

[android] Update headers that were split in NDK 23 swiftlang/swift-corelibs-libdispatch#568

Merged

finagolfin mentioned this pull request Oct 2, 2021

[android] Update to NDK 23 #39045

Merged

finagolfin mentioned this pull request Dec 19, 2021

[android] Remove ICU build flags since that requirement was dropped in #40340 #40625

Merged

marcprux mentioned this pull request May 26, 2025

Run the compiler validation suite for Android swift-android-sdk/swift-docker#8

Open

marcprux added a commit to swift-android-sdk/swift-android-sdk that referenced this pull request May 27, 2025

Add build-script patch from swiftlang/swift#38441

08df408

marcprux added a commit to swift-android-sdk/swift-docker that referenced this pull request May 27, 2025

Add --cross-compile-build-swift-tools=0 from swiftlang/swift#38441

fcde0f1

finagolfin force-pushed the cross-compiler branch from fd79ea8 to 5b1bff8 Compare May 28, 2025 20:24

finagolfin changed the title ~~[build] Add a new flag to disable cross-compiling the compiler~~ [build] Add a new --cross-compile-build-swift-tools flag to disable cross-compiling the compiler May 29, 2025

finagolfin merged commit d8243f8 into swiftlang:main Jun 3, 2025
4 of 5 checks passed

finagolfin deleted the cross-compiler branch June 3, 2025 09:29

finagolfin mentioned this pull request Jun 10, 2025

[build] Switch over to new --cross-compile-build-swift-tools flag when cross-compiling Foundation macros #82163

Merged

finagolfin added a commit that referenced this pull request Jun 11, 2025

[build] Switch over to new --cross-compile-build-swift-tools flag w…

2d735e8

…hen cross-compiling Foundation macros (#82163) Follow-on to #38441, missed this there.

finagolfin mentioned this pull request Jun 11, 2025

[6.2][android] Fix the remaining tests and add a build flag that enables standalone cross-compilation SDKs #82186

Open

[build] Add a new --cross-compile-build-swift-tools flag to disable cross-compiling the compiler #38441

[build] Add a new --cross-compile-build-swift-tools flag to disable cross-compiling the compiler #38441

Uh oh!

Conversation

finagolfin commented Jul 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

finagolfin Jul 16, 2021

Choose a reason for hiding this comment

Uh oh!

drodriguez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

drodriguez commented Jul 21, 2021

Uh oh!

swift-ci commented Jul 21, 2021

Uh oh!

swift-ci commented Jul 21, 2021

Uh oh!

finagolfin commented Jul 21, 2021

Uh oh!

finagolfin commented Jul 25, 2021

Uh oh!

Uh oh!

drodriguez commented Jul 28, 2021

Uh oh!

drodriguez commented Jul 28, 2021

Uh oh!

finagolfin commented Jul 28, 2021

Uh oh!

swift-ci commented Jul 28, 2021

Uh oh!

swift-ci commented Jul 28, 2021

Uh oh!

finagolfin commented Jul 29, 2021

Uh oh!

drodriguez commented Jul 29, 2021

Uh oh!

finagolfin commented May 28, 2025

Uh oh!

finagolfin commented May 28, 2025

Uh oh!

finagolfin commented May 29, 2025

Uh oh!

finagolfin commented Jun 3, 2025

Uh oh!

Uh oh!

Uh oh!

[build] Add a new `--cross-compile-build-swift-tools` flag to disable cross-compiling the compiler #38441

[build] Add a new `--cross-compile-build-swift-tools` flag to disable cross-compiling the compiler #38441

finagolfin commented Jul 16, 2021 •

edited

Loading