Skip to content

post_process_shard_results needs to be more robust #84488

Open
@rorth

Description

@rorth

When running a LLVM 18.1.0 build on FreeBSD 14.0/amd64 with test-release.sh, llvm-lit didn't properly summarize the test results. The build log shows

[...]
PASS: llvm-libc++-shared.cfg.in :: std/utilities/variant/variant.visit/visit_return_type.pass.cpp (97112 of 97112)
Traceback (most recent call last):
  File "/var/llvm/reltest/llvm-18.1.0/final/llvm-project/llvm/utils/lit/lit/formats/googletest.py", line 304, in post_process_shard_results
    testsuites = json.load(f)["testsuites"]
  File "/usr/local/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/llvm/reltest/llvm-18.1.0/final/Phase3/Release/llvmCore-18.1.0-final.obj/./bin/llvm-lit", line 65, in <module>
    main(builtin_parameters)
  File "/var/llvm/reltest/llvm-18.1.0/final/llvm-project/llvm/utils/lit/lit/main.py", line 129, in main
    selected_tests, discovered_tests = GoogleTest.post_process_shard_results(
  File "/var/llvm/reltest/llvm-18.1.0/final/llvm-project/llvm/utils/lit/lit/formats/googletest.py", line 306, in post_process_shard_results
    raise RuntimeError(
RuntimeError: Failed to parse json file: /var/llvm/reltest/llvm-18.1.0/final/Phase3/Release/llvmCore-18.1.0-final.obj/runtimes/runtimes-bins/compiler-rt/lib/msan/tests/./Msan-x86_64-Test-MemorySanitizer-Unit-37844-0-9.json

FAILED: CMakeFiles/check-all /var/llvm/reltest/llvm-18.1.0/final/Phase3/Release/llvmCore-18.1.0-final.obj/CMakeFiles/check-all

I could trace this to two msan tests producing empty *.json files (Issue #84482), which googletest.py chokes on. This must not happen: llvm-lit needs to be robust enough to cope with such input, invalid or no.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions