flang-rt creates a second llvm_gtest library that can cause problems in parallel builds

[DavidSpickett]

I have been chasing seemingly random link and build failures we have had on Linaro's 2 stage AArch64 Linux bots in the last couple of weeks.

The error always happens while checking stage 2 of the build, it never happens when checking stage 1 on any of our bots. It's always something to do with llvm_gtest, or a unittest binary that uses that library. All the failing bots have flang and flang-rt enabled. This error is not consistent, it happens maybe once every 2 days.

ld usually complains about a corrupted file:

[955/1388] Linking CXX executable unittests/ExecutionEngine/Orc/OrcJITTests
FAILED: unittests/ExecutionEngine/Orc/OrcJITTests 
: && /home/tcwg-buildbot/worker/clang-aarch64-full-2stage/stage1.install/bin/clang++ -mcpu=neoverse-n1 -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG -Wl,--export-dynamic  -Wl,--gc-sections unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/CoreAPIsTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/ExecutorAddressTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/ExecutionSessionWrapperFunctionCallsTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/EPCGenericJITLinkMemoryManagerTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/EPCGenericMemoryAccessTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/IndirectionUtilsTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/JITTargetMachineBuilderTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/LazyCallThroughAndReexportsTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/LookupAndRecordAddrsTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/MachOPlatformTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/MapperJITLinkMemoryManagerTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/MemoryFlagsTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/MemoryMapperTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/ObjectFormatsTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/ObjectLinkingLayerTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/OrcCAPITest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/OrcTestCommon.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/ResourceTrackerTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/RTDyldObjectLinkingLayerTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/SharedMemoryMapperTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/SimpleExecutorMemoryManagerTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/SimplePackedSerializationTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/SymbolStringPoolTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/TaskDispatchTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/ThreadSafeModuleTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/WrapperFunctionUtilsTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/JITLinkRedirectionManagerTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/ReOptimizeLayerTest.cpp.o -o unittests/ExecutionEngine/Orc/OrcJITTests  lib/libLLVMCore.a  lib/libLLVMExecutionEngine.a  lib/libLLVMIRReader.a  lib/libLLVMJITLink.a  lib/libLLVMObject.a  lib/libLLVMOrcDebugging.a  lib/libLLVMOrcJIT.a  lib/libLLVMOrcShared.a  lib/libLLVMOrcTargetProcess.a  lib/libLLVMPasses.a  lib/libLLVMRuntimeDyld.a  lib/libLLVMSupport.a  lib/libLLVMTargetParser.a  lib/libLLVMSupport.a  lib/libLLVMAArch64CodeGen.a  lib/libLLVMAArch64AsmParser.a  lib/libLLVMAArch64Desc.a  lib/libLLVMAArch64Disassembler.a  lib/libLLVMAArch64Info.a  lib/libLLVMAArch64Utils.a  lib/libllvm_gtest_main.a  lib/libllvm_gtest.a  lib/libLLVMTestingSupport.a  lib/libLLVMExecutionEngine.a  lib/libLLVMJITLink.a  lib/libLLVMOrcTargetProcess.a  lib/libLLVMCoroutines.a  lib/libLLVMHipStdPar.a  lib/libLLVMipo.a  lib/libLLVMFrontendOpenMP.a  lib/libLLVMFrontendOffloading.a  lib/libLLVMObjectYAML.a  lib/libLLVMFrontendAtomic.a  lib/libLLVMLinker.a  lib/libLLVMIRPrinter.a  lib/libLLVMInstrumentation.a  lib/libLLVMRuntimeDyld.a  lib/libLLVMWindowsDriver.a  lib/libLLVMOption.a  lib/libLLVMOrcShared.a  lib/libLLVMCFGuard.a  lib/libLLVMGlobalISel.a  lib/libLLVMVectorize.a  lib/libLLVMSandboxIR.a  lib/libLLVMAsmPrinter.a  lib/libLLVMSelectionDAG.a  lib/libLLVMCodeGen.a  lib/libLLVMTarget.a  lib/libLLVMObjCARCOpts.a  lib/libLLVMScalarOpts.a  lib/libLLVMAggressiveInstCombine.a  lib/libLLVMInstCombine.a  lib/libLLVMTransformUtils.a  lib/libLLVMCGData.a  lib/libLLVMBitWriter.a  lib/libLLVMAnalysis.a  lib/libLLVMProfileData.a  lib/libLLVMSymbolize.a  lib/libLLVMDebugInfoGSYM.a  lib/libLLVMDebugInfoDWARF.a  lib/libLLVMDebugInfoPDB.a  lib/libLLVMObject.a  lib/libLLVMIRReader.a  lib/libLLVMAsmParser.a  lib/libLLVMBitReader.a  lib/libLLVMTextAPI.a  lib/libLLVMDebugInfoCodeView.a  lib/libLLVMDebugInfoMSF.a  lib/libLLVMDebugInfoBTF.a  lib/libLLVMMCParser.a  lib/libLLVMAArch64Desc.a  lib/libLLVMCodeGenTypes.a  lib/libLLVMAArch64Info.a  lib/libLLVMAArch64Utils.a  lib/libLLVMCore.a  lib/libLLVMRemarks.a  lib/libLLVMBitstreamReader.a  lib/libLLVMMCDisassembler.a  lib/libLLVMMC.a  lib/libLLVMBinaryFormat.a  lib/libLLVMTargetParser.a  lib/libllvm_gtest.a  lib/libLLVMSupport.a  lib/libLLVMDemangle.a  -lrt  -ldl  -lm  /usr/lib/aarch64-linux-gnu/libz.so  -lpthread && :
/usr/bin/ld: error in lib/libllvm_gtest_main.a(TestMain.cpp.o)(.eh_frame); no .eh_frame_hdr table will be created
/usr/bin/ld: lib/libllvm_gtest.a(gtest-all.cc.o): bad reloc symbol index (0xa814a714 >= 0xabc) for offset 0xa414a314a202ee64 in section `.eh_frame'
/usr/bin/ld: lib/libllvm_gtest.a(gmock-all.cc.o): bad reloc symbol index (0x65722e00 >= 0x264) for offset 0x456574697277356d in section `.eh_frame'
/usr/bin/ld: corrupt input: lib/libllvm_gtest.a(gtest-all.cc.o)
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

And bots using lld have some variant of no such file:

FAILED: unittests/Passes/Plugins/PluginsTests 
: && /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage1.install/bin/clang++ -mcpu=neoverse-512tvb -mllvm -scalable-vectorization=preferred -mllvm -treat-scalable-fixed-error-as-warning=false -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG -fuse-ld=lld -Wl,--color-diagnostics   -Wl,--export-dynamic  -Wl,--gc-sections unittests/Passes/Plugins/CMakeFiles/PluginsTests.dir/PluginsTest.cpp.o -o unittests/Passes/Plugins/PluginsTests  lib/libLLVMSupport.a  lib/libLLVMPasses.a  lib/libLLVMCore.a  lib/libLLVMAsmParser.a  lib/libLLVMSupport.a  lib/libllvm_gtest_main.a  lib/libllvm_gtest.a  lib/libLLVMTestingSupport.a  lib/libLLVMCFGuard.a  lib/libLLVMGlobalISel.a  lib/libLLVMSelectionDAG.a  lib/libLLVMCodeGen.a  lib/libLLVMCGData.a  lib/libLLVMCodeGenTypes.a  lib/libLLVMCoroutines.a  lib/libLLVMHipStdPar.a  lib/libLLVMipo.a  lib/libLLVMBitWriter.a  lib/libLLVMFrontendOpenMP.a  lib/libLLVMFrontendOffloading.a  lib/libLLVMObjectYAML.a  lib/libLLVMFrontendAtomic.a  lib/libLLVMLinker.a  lib/libLLVMIRPrinter.a  lib/libLLVMObjCARCOpts.a  lib/libLLVMScalarOpts.a  lib/libLLVMAggressiveInstCombine.a  lib/libLLVMInstCombine.a  lib/libLLVMTarget.a  lib/libLLVMVectorize.a  lib/libLLVMSandboxIR.a  lib/libLLVMInstrumentation.a  lib/libLLVMTransformUtils.a  lib/libLLVMAnalysis.a  lib/libLLVMProfileData.a  lib/libLLVMSymbolize.a  lib/libLLVMDebugInfoGSYM.a  lib/libLLVMDebugInfoPDB.a  lib/libLLVMDebugInfoCodeView.a  lib/libLLVMDebugInfoMSF.a  lib/libLLVMDebugInfoBTF.a  lib/libLLVMDebugInfoDWARF.a  lib/libLLVMObject.a  lib/libLLVMIRReader.a  lib/libLLVMAsmParser.a  lib/libLLVMBitReader.a  lib/libLLVMCore.a  lib/libLLVMRemarks.a  lib/libLLVMBitstreamReader.a  lib/libLLVMMCParser.a  lib/libLLVMTextAPI.a  lib/libLLVMMC.a  lib/libLLVMBinaryFormat.a  lib/libLLVMTargetParser.a  lib/libllvm_gtest.a  lib/libLLVMSupport.a  -lrt  -ldl  -lm  /usr/lib/aarch64-linux-gnu/libz.so  lib/libLLVMDemangle.a  -lpthread && :
clang++: warning: argument unused during compilation: '-mllvm -scalable-vectorization=preferred' [-Wunused-command-line-argument]
clang++: warning: argument unused during compilation: '-mllvm -treat-scalable-fixed-error-as-warning=false' [-Wunused-command-line-argument]
ld.lld: error: cannot open lib/libllvm_gtest.a: No such file or directory
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

[1223/1388] Linking CXX executable unittests/IR/IRTests
FAILED: unittests/IR/IRTests 
: && /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage1.install/bin/clang++ -mcpu=neoverse-512tvb -mllvm -scalable-vectorization=preferred -mllvm -treat-scalable-fixed-error-as-warning=false -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG -fuse-ld=lld -Wl,--color-diagnostics    -Wl,--gc-sections unittests/IR/CMakeFiles/IRTests.dir/AbstractCallSiteTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/AsmWriterTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/AttributesTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/BasicBlockTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/BasicBlockDbgInfoTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/CFGBuilder.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ConstantFPRangeTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ConstantRangeTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ConstantRangeListTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ConstantsTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/DataLayoutTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/DebugInfoTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/DebugTypeODRUniquingTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/DemandedBitsTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/DominatorTreeTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/DominatorTreeBatchUpdatesTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/DroppedVariableStatsIRTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/FunctionTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/PassBuilderCallbacksTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/IRBuilderTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/InstructionsTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/IntrinsicsTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/LegacyPassManagerTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/MDBuilderTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/MemoryModelRelaxationAnnotationsTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ManglerTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/MetadataTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ModuleTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ModuleSummaryIndexTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/PassManagerTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/PatternMatch.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ShuffleVectorInstTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/StructuralHashTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/TimePassesTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/TypesTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/UseTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/UserTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ValueHandleTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ValueMapTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ValueTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/VectorBuilderTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/VectorTypeUtilsTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/VectorTypesTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/VerifierTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/VFABIDemanglerTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/VPIntrinsicTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/CoreBindings.cpp.o -o unittests/IR/IRTests  lib/libLLVMAnalysis.a  lib/libLLVMAsmParser.a  lib/libLLVMCore.a  lib/libLLVMSupport.a  lib/libLLVMPasses.a  lib/libLLVMTargetParser.a  lib/libLLVMTransformUtils.a  lib/libLLVMScalarOpts.a  lib/libLLVMSupport.a  lib/libllvm_gtest_main.a  lib/libllvm_gtest.a  lib/libLLVMTestingSupport.a  lib/libLLVMCFGuard.a  lib/libLLVMGlobalISel.a  lib/libLLVMSelectionDAG.a  lib/libLLVMCodeGen.a  lib/libLLVMCGData.a  lib/libLLVMCodeGenTypes.a  lib/libLLVMCoroutines.a  lib/libLLVMHipStdPar.a  lib/libLLVMipo.a  lib/libLLVMBitWriter.a  lib/libLLVMFrontendOpenMP.a  lib/libLLVMScalarOpts.a  lib/libLLVMAggressiveInstCombine.a  lib/libLLVMFrontendOffloading.a  lib/libLLVMObjectYAML.a  lib/libLLVMFrontendAtomic.a  lib/libLLVMLinker.a  lib/libLLVMInstCombine.a  lib/libLLVMIRPrinter.a  lib/libLLVMObjCARCOpts.a  lib/libLLVMTarget.a  lib/libLLVMVectorize.a  lib/libLLVMSandboxIR.a  lib/libLLVMInstrumentation.a  lib/libLLVMTransformUtils.a  lib/libLLVMAnalysis.a  lib/libLLVMProfileData.a  lib/libLLVMSymbolize.a  lib/libLLVMDebugInfoGSYM.a  lib/libLLVMDebugInfoPDB.a  lib/libLLVMDebugInfoCodeView.a  lib/libLLVMDebugInfoMSF.a  lib/libLLVMDebugInfoBTF.a  lib/libLLVMDebugInfoDWARF.a  lib/libLLVMObject.a  lib/libLLVMIRReader.a  lib/libLLVMAsmParser.a  lib/libLLVMBitReader.a  lib/libLLVMCore.a  lib/libLLVMRemarks.a  lib/libLLVMBitstreamReader.a  lib/libLLVMMCParser.a  lib/libLLVMTextAPI.a  lib/libLLVMMC.a  lib/libLLVMBinaryFormat.a  lib/libLLVMTargetParser.a  lib/libllvm_gtest.a  lib/libLLVMSupport.a  lib/libLLVMDemangle.a  -lrt  -ldl  -lm  /usr/lib/aarch64-linux-gnu/libz.so  -lpthread && :
clang++: error: no such file or directory: 'lib/libllvm_gtest.a'
clang++: error: no such file or directory: 'lib/libllvm_gtest.a'
[1227/1388] Linking CXX executable unittests/Target/PowerPC/PowerPCTests

I've tried a few things to figure this out but for some reason got the idea to check flang-rt, and it contains this cmake code:

# Add GTest if not already present.
# Using a function so LLVM_SUBPROJECT_TITLE does not propagate.
function (build_gtest)
  set(LLVM_SUBPROJECT_TITLE "Third-Party/Google Test")
  add_subdirectory("${LLVM_THIRD_PARTY_DIR}/unittest" "${CMAKE_CURRENT_BINARY_DIR}/third-party/unittest")
endfunction ()
if (NOT TARGET llvm_gtest)
  build_gtest()
endif ()

If I configure a build with:

$ cmake ../llvm-project/llvm -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="llvm;clang;flang;mlir" -DLLVM_ENABLE_RUNTIMES=flang-rt -G Ninja

I can see that it adds an llvm_gtest target (debug print is added by me ofc):

-- Targeting X86
-- Targeting XCore
-- <<<<<<<<<<<<<<<<<<<<<<<< adding llvm_gtest library >>>>>>>>>>>>>>>>>
-- Performing Test C_SUPPORTS_WERROR_IMPLICIT_FUNCTION_DECLARATION
-- Performing Test C_SUPPORTS_WERROR_IMPLICIT_FUNCTION_DECLARATION - Success
-- Performing Test C_SUPPORTS_WUNDEF

And later when it gets to the runtimes it does not see the target and adds another one:

-- Looking for sinl in m - found
-- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> no gtest target <<<<<<<<<<<<<<<<<<<<<<<<<<
-- <<<<<<<<<<<<<<<<<<<<<<<< adding llvm_gtest library >>>>>>>>>>>>>>>>>
-- Performing Test LLVM_LINKER_SUPPORTS_B_SYMBOLIC_FUNCTIONS

And this actually happens for every ninja check-flang-rt:

$ ninja check-flang-rt
[1/2] cd /home/david.spickett/flang-build/runtimes/...imes-bins/ --target check-flang-rt --config Release
[0/1] Re-running CMake...
-- Performing bootstrapping runtimes build.
-- Building with -fPIC
-- LLVM host triple: aarch64-unknown-linux-gnu
-- LLVM default target triple: aarch64-unknown-linux-gnu
-- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> no gtest target <<<<<<<<<<<<<<<<<<<<<<<<<<
-- <<<<<<<<<<<<<<<<<<<<<<<< adding llvm_gtest library >>>>>>>>>>>>>>>>>
-- Configuring done (0.5s)
-- Generating done (0.0s)
-- Build files have been written to: /home/david.spickett/flang-build/runtimes/runtimes-bins
[1/2] Running the Flang-RT regression tests

I know llvm builds the library as lib/libllvm_gtest.a:

$ ninja check-llvm-unit -d explain
<...>
ninja explain: lib/libllvm_gtest_main.a is dirty
ninja explain: lib/libllvm_gtest.a is dirty
ninja explain: lib/libllvm_gtest.a is dirty

If I remove the files and run ninja check-flang-rt:

$ ninja check-flang-rt -d explain
<...>
ninja explain: output lib/libllvm_gtest.a doesn't exist
ninja explain: output lib/libllvm_gtest_main.a doesn't exist
ninja explain: lib/libllvm_gtest_main.a is dirty
ninja explain: lib/libllvm_gtest.a is dirty

Now that might be lib/ in the runtimes bins folder, so let's check that:

$ find -name "*gtest*" | grep "\.a$"
./lib/libllvm_gtest.a
./lib/libllvm_gtest_main.a

Only one location for these libraries.

So what I think is happening is that there is one llvm_gtest from the llvm build and one from flang-rt. check-llvm-unittest and check-flang-rt don't have dependencies on each other, so they could start building at around the same time, and clash trying to write, overwrite, or modify llvm_gtest.a.

In a single stage build, we ccache everything which I expect reduces the chances of this problem happening. Stage 2 is uncached because the compiler is different every time.

I looked at other runtimes and I see that libc does use llvm_gtest but it does not add a new target for it. flang-rt and offload both check for an existing target.

I've never seen this happen with offload - but we rarely build it and if we do, it's never tested. It's just so flang can run some openmp testing.

So I am wondering if this was ever correct. Can the runtimes build see targets from the llvm build? Is there a way to pass them through to the runtimes build like we do some of the configuration options?

[llvmbot]

@llvm/issue-subscribers-flang-runtime

Author: David Spickett (DavidSpickett)

I have been chasing seemingly random link and build failures we have had on Linaro's 2 stage AArch64 Linux bots in the last couple of weeks.

The error always happens while checking stage 2 of the build, it never happens when checking stage 1 on any of our bots. It's always something to do with llvm_gtest, or a unittest binary that uses that library. All the failing bots have flang and flang-rt enabled. This error is not consistent, it happens maybe once every 2 days.

ld usually complains about a corrupted file:

[955/1388] Linking CXX executable unittests/ExecutionEngine/Orc/OrcJITTests
FAILED: unittests/ExecutionEngine/Orc/OrcJITTests 
: && /home/tcwg-buildbot/worker/clang-aarch64-full-2stage/stage1.install/bin/clang++ -mcpu=neoverse-n1 -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG -Wl,--export-dynamic  -Wl,--gc-sections unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/CoreAPIsTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/ExecutorAddressTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/ExecutionSessionWrapperFunctionCallsTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/EPCGenericJITLinkMemoryManagerTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/EPCGenericMemoryAccessTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/IndirectionUtilsTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/JITTargetMachineBuilderTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/LazyCallThroughAndReexportsTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/LookupAndRecordAddrsTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/MachOPlatformTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/MapperJITLinkMemoryManagerTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/MemoryFlagsTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/MemoryMapperTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/ObjectFormatsTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/ObjectLinkingLayerTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/OrcCAPITest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/OrcTestCommon.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/ResourceTrackerTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/RTDyldObjectLinkingLayerTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/SharedMemoryMapperTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/SimpleExecutorMemoryManagerTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/SimplePackedSerializationTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/SymbolStringPoolTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/TaskDispatchTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/ThreadSafeModuleTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/WrapperFunctionUtilsTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/JITLinkRedirectionManagerTest.cpp.o unittests/ExecutionEngine/Orc/CMakeFiles/OrcJITTests.dir/ReOptimizeLayerTest.cpp.o -o unittests/ExecutionEngine/Orc/OrcJITTests  lib/libLLVMCore.a  lib/libLLVMExecutionEngine.a  lib/libLLVMIRReader.a  lib/libLLVMJITLink.a  lib/libLLVMObject.a  lib/libLLVMOrcDebugging.a  lib/libLLVMOrcJIT.a  lib/libLLVMOrcShared.a  lib/libLLVMOrcTargetProcess.a  lib/libLLVMPasses.a  lib/libLLVMRuntimeDyld.a  lib/libLLVMSupport.a  lib/libLLVMTargetParser.a  lib/libLLVMSupport.a  lib/libLLVMAArch64CodeGen.a  lib/libLLVMAArch64AsmParser.a  lib/libLLVMAArch64Desc.a  lib/libLLVMAArch64Disassembler.a  lib/libLLVMAArch64Info.a  lib/libLLVMAArch64Utils.a  lib/libllvm_gtest_main.a  lib/libllvm_gtest.a  lib/libLLVMTestingSupport.a  lib/libLLVMExecutionEngine.a  lib/libLLVMJITLink.a  lib/libLLVMOrcTargetProcess.a  lib/libLLVMCoroutines.a  lib/libLLVMHipStdPar.a  lib/libLLVMipo.a  lib/libLLVMFrontendOpenMP.a  lib/libLLVMFrontendOffloading.a  lib/libLLVMObjectYAML.a  lib/libLLVMFrontendAtomic.a  lib/libLLVMLinker.a  lib/libLLVMIRPrinter.a  lib/libLLVMInstrumentation.a  lib/libLLVMRuntimeDyld.a  lib/libLLVMWindowsDriver.a  lib/libLLVMOption.a  lib/libLLVMOrcShared.a  lib/libLLVMCFGuard.a  lib/libLLVMGlobalISel.a  lib/libLLVMVectorize.a  lib/libLLVMSandboxIR.a  lib/libLLVMAsmPrinter.a  lib/libLLVMSelectionDAG.a  lib/libLLVMCodeGen.a  lib/libLLVMTarget.a  lib/libLLVMObjCARCOpts.a  lib/libLLVMScalarOpts.a  lib/libLLVMAggressiveInstCombine.a  lib/libLLVMInstCombine.a  lib/libLLVMTransformUtils.a  lib/libLLVMCGData.a  lib/libLLVMBitWriter.a  lib/libLLVMAnalysis.a  lib/libLLVMProfileData.a  lib/libLLVMSymbolize.a  lib/libLLVMDebugInfoGSYM.a  lib/libLLVMDebugInfoDWARF.a  lib/libLLVMDebugInfoPDB.a  lib/libLLVMObject.a  lib/libLLVMIRReader.a  lib/libLLVMAsmParser.a  lib/libLLVMBitReader.a  lib/libLLVMTextAPI.a  lib/libLLVMDebugInfoCodeView.a  lib/libLLVMDebugInfoMSF.a  lib/libLLVMDebugInfoBTF.a  lib/libLLVMMCParser.a  lib/libLLVMAArch64Desc.a  lib/libLLVMCodeGenTypes.a  lib/libLLVMAArch64Info.a  lib/libLLVMAArch64Utils.a  lib/libLLVMCore.a  lib/libLLVMRemarks.a  lib/libLLVMBitstreamReader.a  lib/libLLVMMCDisassembler.a  lib/libLLVMMC.a  lib/libLLVMBinaryFormat.a  lib/libLLVMTargetParser.a  lib/libllvm_gtest.a  lib/libLLVMSupport.a  lib/libLLVMDemangle.a  -lrt  -ldl  -lm  /usr/lib/aarch64-linux-gnu/libz.so  -lpthread && :
/usr/bin/ld: error in lib/libllvm_gtest_main.a(TestMain.cpp.o)(.eh_frame); no .eh_frame_hdr table will be created
/usr/bin/ld: lib/libllvm_gtest.a(gtest-all.cc.o): bad reloc symbol index (0xa814a714 >= 0xabc) for offset 0xa414a314a202ee64 in section `.eh_frame'
/usr/bin/ld: lib/libllvm_gtest.a(gmock-all.cc.o): bad reloc symbol index (0x65722e00 >= 0x264) for offset 0x456574697277356d in section `.eh_frame'
/usr/bin/ld: corrupt input: lib/libllvm_gtest.a(gtest-all.cc.o)
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

And bots using lld have some variant of no such file:

FAILED: unittests/Passes/Plugins/PluginsTests 
: && /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage1.install/bin/clang++ -mcpu=neoverse-512tvb -mllvm -scalable-vectorization=preferred -mllvm -treat-scalable-fixed-error-as-warning=false -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG -fuse-ld=lld -Wl,--color-diagnostics   -Wl,--export-dynamic  -Wl,--gc-sections unittests/Passes/Plugins/CMakeFiles/PluginsTests.dir/PluginsTest.cpp.o -o unittests/Passes/Plugins/PluginsTests  lib/libLLVMSupport.a  lib/libLLVMPasses.a  lib/libLLVMCore.a  lib/libLLVMAsmParser.a  lib/libLLVMSupport.a  lib/libllvm_gtest_main.a  lib/libllvm_gtest.a  lib/libLLVMTestingSupport.a  lib/libLLVMCFGuard.a  lib/libLLVMGlobalISel.a  lib/libLLVMSelectionDAG.a  lib/libLLVMCodeGen.a  lib/libLLVMCGData.a  lib/libLLVMCodeGenTypes.a  lib/libLLVMCoroutines.a  lib/libLLVMHipStdPar.a  lib/libLLVMipo.a  lib/libLLVMBitWriter.a  lib/libLLVMFrontendOpenMP.a  lib/libLLVMFrontendOffloading.a  lib/libLLVMObjectYAML.a  lib/libLLVMFrontendAtomic.a  lib/libLLVMLinker.a  lib/libLLVMIRPrinter.a  lib/libLLVMObjCARCOpts.a  lib/libLLVMScalarOpts.a  lib/libLLVMAggressiveInstCombine.a  lib/libLLVMInstCombine.a  lib/libLLVMTarget.a  lib/libLLVMVectorize.a  lib/libLLVMSandboxIR.a  lib/libLLVMInstrumentation.a  lib/libLLVMTransformUtils.a  lib/libLLVMAnalysis.a  lib/libLLVMProfileData.a  lib/libLLVMSymbolize.a  lib/libLLVMDebugInfoGSYM.a  lib/libLLVMDebugInfoPDB.a  lib/libLLVMDebugInfoCodeView.a  lib/libLLVMDebugInfoMSF.a  lib/libLLVMDebugInfoBTF.a  lib/libLLVMDebugInfoDWARF.a  lib/libLLVMObject.a  lib/libLLVMIRReader.a  lib/libLLVMAsmParser.a  lib/libLLVMBitReader.a  lib/libLLVMCore.a  lib/libLLVMRemarks.a  lib/libLLVMBitstreamReader.a  lib/libLLVMMCParser.a  lib/libLLVMTextAPI.a  lib/libLLVMMC.a  lib/libLLVMBinaryFormat.a  lib/libLLVMTargetParser.a  lib/libllvm_gtest.a  lib/libLLVMSupport.a  -lrt  -ldl  -lm  /usr/lib/aarch64-linux-gnu/libz.so  lib/libLLVMDemangle.a  -lpthread && :
clang++: warning: argument unused during compilation: '-mllvm -scalable-vectorization=preferred' [-Wunused-command-line-argument]
clang++: warning: argument unused during compilation: '-mllvm -treat-scalable-fixed-error-as-warning=false' [-Wunused-command-line-argument]
ld.lld: error: cannot open lib/libllvm_gtest.a: No such file or directory
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

[1223/1388] Linking CXX executable unittests/IR/IRTests
FAILED: unittests/IR/IRTests 
: && /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage1.install/bin/clang++ -mcpu=neoverse-512tvb -mllvm -scalable-vectorization=preferred -mllvm -treat-scalable-fixed-error-as-warning=false -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG -fuse-ld=lld -Wl,--color-diagnostics    -Wl,--gc-sections unittests/IR/CMakeFiles/IRTests.dir/AbstractCallSiteTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/AsmWriterTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/AttributesTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/BasicBlockTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/BasicBlockDbgInfoTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/CFGBuilder.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ConstantFPRangeTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ConstantRangeTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ConstantRangeListTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ConstantsTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/DataLayoutTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/DebugInfoTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/DebugTypeODRUniquingTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/DemandedBitsTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/DominatorTreeTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/DominatorTreeBatchUpdatesTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/DroppedVariableStatsIRTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/FunctionTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/PassBuilderCallbacksTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/IRBuilderTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/InstructionsTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/IntrinsicsTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/LegacyPassManagerTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/MDBuilderTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/MemoryModelRelaxationAnnotationsTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ManglerTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/MetadataTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ModuleTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ModuleSummaryIndexTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/PassManagerTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/PatternMatch.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ShuffleVectorInstTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/StructuralHashTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/TimePassesTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/TypesTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/UseTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/UserTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ValueHandleTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ValueMapTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/ValueTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/VectorBuilderTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/VectorTypeUtilsTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/VectorTypesTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/VerifierTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/VFABIDemanglerTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/VPIntrinsicTest.cpp.o unittests/IR/CMakeFiles/IRTests.dir/CoreBindings.cpp.o -o unittests/IR/IRTests  lib/libLLVMAnalysis.a  lib/libLLVMAsmParser.a  lib/libLLVMCore.a  lib/libLLVMSupport.a  lib/libLLVMPasses.a  lib/libLLVMTargetParser.a  lib/libLLVMTransformUtils.a  lib/libLLVMScalarOpts.a  lib/libLLVMSupport.a  lib/libllvm_gtest_main.a  lib/libllvm_gtest.a  lib/libLLVMTestingSupport.a  lib/libLLVMCFGuard.a  lib/libLLVMGlobalISel.a  lib/libLLVMSelectionDAG.a  lib/libLLVMCodeGen.a  lib/libLLVMCGData.a  lib/libLLVMCodeGenTypes.a  lib/libLLVMCoroutines.a  lib/libLLVMHipStdPar.a  lib/libLLVMipo.a  lib/libLLVMBitWriter.a  lib/libLLVMFrontendOpenMP.a  lib/libLLVMScalarOpts.a  lib/libLLVMAggressiveInstCombine.a  lib/libLLVMFrontendOffloading.a  lib/libLLVMObjectYAML.a  lib/libLLVMFrontendAtomic.a  lib/libLLVMLinker.a  lib/libLLVMInstCombine.a  lib/libLLVMIRPrinter.a  lib/libLLVMObjCARCOpts.a  lib/libLLVMTarget.a  lib/libLLVMVectorize.a  lib/libLLVMSandboxIR.a  lib/libLLVMInstrumentation.a  lib/libLLVMTransformUtils.a  lib/libLLVMAnalysis.a  lib/libLLVMProfileData.a  lib/libLLVMSymbolize.a  lib/libLLVMDebugInfoGSYM.a  lib/libLLVMDebugInfoPDB.a  lib/libLLVMDebugInfoCodeView.a  lib/libLLVMDebugInfoMSF.a  lib/libLLVMDebugInfoBTF.a  lib/libLLVMDebugInfoDWARF.a  lib/libLLVMObject.a  lib/libLLVMIRReader.a  lib/libLLVMAsmParser.a  lib/libLLVMBitReader.a  lib/libLLVMCore.a  lib/libLLVMRemarks.a  lib/libLLVMBitstreamReader.a  lib/libLLVMMCParser.a  lib/libLLVMTextAPI.a  lib/libLLVMMC.a  lib/libLLVMBinaryFormat.a  lib/libLLVMTargetParser.a  lib/libllvm_gtest.a  lib/libLLVMSupport.a  lib/libLLVMDemangle.a  -lrt  -ldl  -lm  /usr/lib/aarch64-linux-gnu/libz.so  -lpthread && :
clang++: error: no such file or directory: 'lib/libllvm_gtest.a'
clang++: error: no such file or directory: 'lib/libllvm_gtest.a'
[1227/1388] Linking CXX executable unittests/Target/PowerPC/PowerPCTests

I've tried a few things to figure this out but for some reason got the idea to check flang-rt, and it contains this cmake code:

# Add GTest if not already present.
# Using a function so LLVM_SUBPROJECT_TITLE does not propagate.
function (build_gtest)
  set(LLVM_SUBPROJECT_TITLE "Third-Party/Google Test")
  add_subdirectory("${LLVM_THIRD_PARTY_DIR}/unittest" "${CMAKE_CURRENT_BINARY_DIR}/third-party/unittest")
endfunction ()
if (NOT TARGET llvm_gtest)
  build_gtest()
endif ()

If I configure a build with:

$ cmake ../llvm-project/llvm -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="llvm;clang;flang;mlir" -DLLVM_ENABLE_RUNTIMES=flang-rt -G Ninja

I can see that it adds an llvm_gtest target (debug print is added by me ofc):

-- Targeting X86
-- Targeting XCore
-- <<<<<<<<<<<<<<<<<<<<<<<< adding llvm_gtest library >>>>>>>>>>>>>>>>>
-- Performing Test C_SUPPORTS_WERROR_IMPLICIT_FUNCTION_DECLARATION
-- Performing Test C_SUPPORTS_WERROR_IMPLICIT_FUNCTION_DECLARATION - Success
-- Performing Test C_SUPPORTS_WUNDEF

And later when it gets to the runtimes it does not see the target and adds another one:

-- Looking for sinl in m - found
-- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> no gtest target <<<<<<<<<<<<<<<<<<<<<<<<<<
-- <<<<<<<<<<<<<<<<<<<<<<<< adding llvm_gtest library >>>>>>>>>>>>>>>>>
-- Performing Test LLVM_LINKER_SUPPORTS_B_SYMBOLIC_FUNCTIONS

And this actually happens for every ninja check-flang-rt:

$ ninja check-flang-rt
[1/2] cd /home/david.spickett/flang-build/runtimes/...imes-bins/ --target check-flang-rt --config Release
[0/1] Re-running CMake...
-- Performing bootstrapping runtimes build.
-- Building with -fPIC
-- LLVM host triple: aarch64-unknown-linux-gnu
-- LLVM default target triple: aarch64-unknown-linux-gnu
-- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> no gtest target <<<<<<<<<<<<<<<<<<<<<<<<<<
-- <<<<<<<<<<<<<<<<<<<<<<<< adding llvm_gtest library >>>>>>>>>>>>>>>>>
-- Configuring done (0.5s)
-- Generating done (0.0s)
-- Build files have been written to: /home/david.spickett/flang-build/runtimes/runtimes-bins
[1/2] Running the Flang-RT regression tests

I know llvm builds the library as lib/libllvm_gtest.a:

$ ninja check-llvm-unit -d explain
<...>
ninja explain: lib/libllvm_gtest_main.a is dirty
ninja explain: lib/libllvm_gtest.a is dirty
ninja explain: lib/libllvm_gtest.a is dirty

If I remove the files and run ninja check-flang-rt:

$ ninja check-flang-rt -d explain
<...>
ninja explain: output lib/libllvm_gtest.a doesn't exist
ninja explain: output lib/libllvm_gtest_main.a doesn't exist
ninja explain: lib/libllvm_gtest_main.a is dirty
ninja explain: lib/libllvm_gtest.a is dirty

Now that might be lib/ in the runtimes bins folder, so let's check that:

$ find -name "*gtest*" | grep "\.a$"
./lib/libllvm_gtest.a
./lib/libllvm_gtest_main.a

Only one location for these libraries.

So what I think is happening is that there is one llvm_gtest from the llvm build and one from flang-rt. check-llvm-unittest and check-flang-rt don't have dependencies on each other, so they could start building at around the same time, and clash trying to write, overwrite, or modify llvm_gtest.a.

In a single stage build, we ccache everything which I expect reduces the chances of this problem happening. Stage 2 is uncached because the compiler is different every time.

I looked at other runtimes and I see that libc does use llvm_gtest but it does not add a new target for it. flang-rt and offload both check for an existing target.

I've never seen this happen with offload - but we rarely build it and if we do, it's never tested. It's just so flang can run some openmp testing.

So I am wondering if this was ever correct. Can the runtimes build see targets from the llvm build? Is there a way to pass them through to the runtimes build like we do some of the configuration options?

[DavidSpickett]

@petrhosek You don't need to read my wall of text above, but as an expert on the runtimes build, is it expected that a runtime can check whether the "main" build configured from llvm/ has defined some target? Or is this A: bad practice and/or B: impossible?

@Meinersbur The code I reference seems to have been present from the start. Do you recall including it for a specific reason, perhaps to fix issues in your own builds? Also, have you seen problems like this downstream?

[DavidSpickett]

Caveat here that I might be barking up the wrong tree entirely, but this behaviour is certainly strange to me.

[Meinersbur]

I have a strong suspicion what is going on.

Flang-rt adds GTest by add_subdirectory(third-party/unittest/CMakeLists.txt) (if not yet done so by offload). That one uses llvm_add_library to build libgtest_llvm.a and sets the output directory to LLVM_LIBRARY_OUTPUT_INTDIR. In a bootstrapping-build, it is set to the same directory as the stage-1 LLVM build. If that build builds GTest as well, they will overwrite each other.

I originally didn't think using GTest in runtimes-build was as problematic, as it is used by compiler-rt and offload as well. But compiler-rt's uses of GTest works very differently. And if crosscompiling the dependency of GTest to LLVMSupport.so/.a for the non-default (i.h. host triple) target cannot work since LLVMSupport.so/.a will only be available for the host triple. This wasn't a problem so far since flang doesn't support cross-compiling yet anyway.

I think the output directory for libgtest_llvm.a must be modified, or ensure that it is already built by stage-1 LLVM, i.e. not use add_subdirectory(third-party/unittest/CMakeLists.txt) but add a tool dependency before building the runtimes.

[DavidSpickett]

(if not yet done so by offload)

So flang-rt isn't expecting the llvm build to have added a target for it, it's offload that might have done that. Which makes much more sense than what I read it as.

I think the output directory for libgtest_llvm.a must be modified, or ensure that it is already built by stage-1 LLVM, i.e. not use add_subdirectory(third-party/unittest/CMakeLists.txt) but add a tool dependency before building the runtimes.

What do you mean by "a tool dependency"?

For changing the directory, I thought I might see that flang-rt's gtest was in runtimes-bins/lib. Maybe we can use that.

I have been looking at it trying to figure out a way to maybe not add the gtest target and assume it will be there from something else, but I think I'll break standalone / runtimes only configurations by doing that.

I noticed that libc uses gtest but doesn't have cmake code like flang-rt, but I haven't had time to look at how it works yet. They could be assuming they are always part of a bootstrapping build that includes clang.

[Meinersbur]

(if not yet done so by offload)

So flang-rt isn't expecting the llvm build to have added a target for it, it's offload that might have done that. Which makes much more sense than what I read it as.

There is a third way target llvm_gtest is made available to the runtimes: LLVM_INSTALL_GTEST=ON. I created #143682 to remove these confusions.

What do you mean by "a tool dependency"?

This: https://github.com/llvm/llvm-project/blob/main/llvm/runtimes/CMakeLists.txt#L493-L494

When writing this, I didn't know it already exists.

For changing the directory, I thought I might see that flang-rt's gtest was in runtimes-bins/lib. Maybe we can use that.

Shouldn't be there; The (bootstrapping) runtimes build changes everything to be emitted into a single lib/ directory.

I have been looking at it trying to figure out a way to maybe not add the gtest target and assume it will be there from something else, but I think I'll break standalone / runtimes only configurations by doing that.

I noticed that libc uses gtest but doesn't have cmake code like flang-rt, but I haven't had time to look at how it works yet. They could be assuming they are always part of a bootstrapping build that includes clang.

It assumes LLVM_INSTALL_GTEST=ON. Since it is only enabled via LIBC_INCLUDE_BENCHMARKS=ON, it usually gets unnoticed.