[L0 v2] Flaky graph tests failing on Intel Arc A-Series Graphics

[frasercrmck]

Describe the bug

At least three graph tests are failing flakily on multiple jobs.

Three fails discovered on PR #18666, CI job

 Failed Tests (3):
  SYCL :: Graph/RecordReplay/basic_buffer.cpp
  SYCL :: Graph/RecordReplay/buffer_copy.cpp
  SYCL :: Graph/RecordReplay/queue_constructor_buffer.cpp

One on CI job for #18661:

Failed Tests (1):
  SYCL :: Graph/Explicit/basic_buffer.cpp

One on CI job for #18626:

 Failed Tests (1):
  SYCL :: Graph/Explicit/buffer_copy.cpp

Full log for context:

FAIL: SYCL :: Graph/RecordReplay/basic_buffer.cpp (1082 of 2380)
******************** TEST 'SYCL :: Graph/RecordReplay/basic_buffer.cpp' FAILED ********************
Exit Code: -6

Command Output (stdout):
--
# RUN: at line 2
env ONEAPI_DEVICE_SELECTOR=level_zero:gpu  /__w/llvm/llvm/build-e2e/Graph/RecordReplay/Output/basic_buffer.cpp.tmp.out
# executed command: env ONEAPI_DEVICE_SELECTOR=level_zero:gpu /__w/llvm/llvm/build-e2e/Graph/RecordReplay/Output/basic_buffer.cpp.tmp.out
# note: command had no output on stdout or stderr
# RUN: at line 2
env env UR_LOADER_USE_LEVEL_ZERO_V2=1 ONEAPI_DEVICE_SELECTOR=level_zero:gpu  /__w/llvm/llvm/build-e2e/Graph/RecordReplay/Output/basic_buffer.cpp.tmp.out
# executed command: env env UR_LOADER_USE_LEVEL_ZERO_V2=1 ONEAPI_DEVICE_SELECTOR=level_zero:gpu /__w/llvm/llvm/build-e2e/Graph/RecordReplay/Output/basic_buffer.cpp.tmp.out
# .---command stdout------------
# | Unexpected value at index 0 for HostAccA: 3 (got) vs 6 (expected)
# `-----------------------------
# .---command stderr------------
# | basic_buffer.cpp.tmp.out: /__w/llvm/llvm/llvm/sycl/test-e2e/Graph/RecordReplay/../Inputs/basic_buffer.cpp:49: int main(): Assertion `check_value(i, ReferenceA[i], HostAccA[i], "HostAccA")' failed.
# `-----------------------------
# error: command failed with exit status: -6

--

********************
FAIL: SYCL :: Graph/RecordReplay/queue_constructor_buffer.cpp (1124 of 2380)
******************** TEST 'SYCL :: Graph/RecordReplay/queue_constructor_buffer.cpp' FAILED ********************
Exit Code: -6

Command Output (stdout):
--
# RUN: at line 2
env ONEAPI_DEVICE_SELECTOR=level_zero:gpu  /__w/llvm/llvm/build-e2e/Graph/RecordReplay/Output/queue_constructor_buffer.cpp.tmp.out
# executed command: env ONEAPI_DEVICE_SELECTOR=level_zero:gpu /__w/llvm/llvm/build-e2e/Graph/RecordReplay/Output/queue_constructor_buffer.cpp.tmp.out
# note: command had no output on stdout or stderr
# RUN: at line 2
env env UR_LOADER_USE_LEVEL_ZERO_V2=1 ONEAPI_DEVICE_SELECTOR=level_zero:gpu  /__w/llvm/llvm/build-e2e/Graph/RecordReplay/Output/queue_constructor_buffer.cpp.tmp.out
# executed command: env env UR_LOADER_USE_LEVEL_ZERO_V2=1 ONEAPI_DEVICE_SELECTOR=level_zero:gpu /__w/llvm/llvm/build-e2e/Graph/RecordReplay/Output/queue_constructor_buffer.cpp.tmp.out
# note: command had no output on stdout or stderr
# RUN: at line 4
env SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=0 env UR_L0_LEAKS_DEBUG=1 env ONEAPI_DEVICE_SELECTOR=level_zero:gpu  /__w/llvm/llvm/build-e2e/Graph/RecordReplay/Output/queue_constructor_buffer.cpp.tmp.out 2>&1 | /__w/llvm/llvm/toolchain/bin/FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/Graph/RecordReplay/queue_constructor_buffer.cpp --implicit-check-not=LEAK
# executed command: env SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=0 env UR_L0_LEAKS_DEBUG=1 env ONEAPI_DEVICE_SELECTOR=level_zero:gpu /__w/llvm/llvm/build-e2e/Graph/RecordReplay/Output/queue_constructor_buffer.cpp.tmp.out
# note: command had no output on stdout or stderr
# executed command: /__w/llvm/llvm/toolchain/bin/FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/Graph/RecordReplay/queue_constructor_buffer.cpp --implicit-check-not=LEAK
# note: command had no output on stdout or stderr
# RUN: at line 4
env SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=0 env UR_L0_LEAKS_DEBUG=1 env env UR_LOADER_USE_LEVEL_ZERO_V2=1 ONEAPI_DEVICE_SELECTOR=level_zero:gpu  /__w/llvm/llvm/build-e2e/Graph/RecordReplay/Output/queue_constructor_buffer.cpp.tmp.out 2>&1 | /__w/llvm/llvm/toolchain/bin/FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/Graph/RecordReplay/queue_constructor_buffer.cpp --implicit-check-not=LEAK
# executed command: env SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=0 env UR_L0_LEAKS_DEBUG=1 env env UR_LOADER_USE_LEVEL_ZERO_V2=1 ONEAPI_DEVICE_SELECTOR=level_zero:gpu /__w/llvm/llvm/build-e2e/Graph/RecordReplay/Output/queue_constructor_buffer.cpp.tmp.out
# note: command had no output on stdout or stderr
# error: command failed with exit status: -6
# executed command: /__w/llvm/llvm/toolchain/bin/FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/Graph/RecordReplay/queue_constructor_buffer.cpp --implicit-check-not=LEAK
# note: command had no output on stdout or stderr

--

********************
FAIL: SYCL :: Graph/RecordReplay/buffer_copy.cpp (1143 of 2380)
******************** TEST 'SYCL :: Graph/RecordReplay/buffer_copy.cpp' FAILED ********************
Exit Code: -6

Command Output (stdout):
--
# RUN: at line 2
env ONEAPI_DEVICE_SELECTOR=level_zero:gpu  /__w/llvm/llvm/build-e2e/Graph/RecordReplay/Output/buffer_copy.cpp.tmp.out
# executed command: env ONEAPI_DEVICE_SELECTOR=level_zero:gpu /__w/llvm/llvm/build-e2e/Graph/RecordReplay/Output/buffer_copy.cpp.tmp.out
# note: command had no output on stdout or stderr
# RUN: at line 2
env env UR_LOADER_USE_LEVEL_ZERO_V2=1 ONEAPI_DEVICE_SELECTOR=level_zero:gpu  /__w/llvm/llvm/build-e2e/Graph/RecordReplay/Output/buffer_copy.cpp.tmp.out
# executed command: env env UR_LOADER_USE_LEVEL_ZERO_V2=1 ONEAPI_DEVICE_SELECTOR=level_zero:gpu /__w/llvm/llvm/build-e2e/Graph/RecordReplay/Output/buffer_copy.cpp.tmp.out
# .---command stdout------------
# | Unexpected value at index 0 for HostAccA: 31 (got) vs 73 (expected)
# `-----------------------------
# .---command stderr------------
# | buffer_copy.cpp.tmp.out: /__w/llvm/llvm/llvm/sycl/test-e2e/Graph/RecordReplay/../Inputs/buffer_copy.cpp:118: int main(): Assertion `check_value(i, ReferenceA[i], HostAccA[i], "HostAccA")' failed.
# `-----------------------------
# error: command failed with exit status: -6

--