[SYCL][E2E] Fix XFAIL statements that use negated device features

[ayylol]

Because of device specific features we have to perform our own checks to see whether or not a test should XFAIL. However, a similar check is also done in internal lit code, but due to the fact that this bit of code only knows about the non device specific features there can be situations where it incorrectly reports a test as expected to fail (Like negating a device specific feature, like XFAIL: !cpu).

To avoid this we set the XFAIL conditions to an empty list before returning our result, this way expected_to_fail is always false in the code below, and our result is not altered.

llvm/llvm/utils/lit/lit/Test.py

Lines 275 to 292 in d60cd27

	def setResult(self, result):
	assert self.result is None, "result already set"
	assert isinstance(result, Result), "unexpected result type"
	try:
	expected_to_fail = self.isExpectedToFail()
	except ValueError as err:
	# Syntax error in an XFAIL line.
	result.code = UNRESOLVED
	result.output = str(err)
	else:
	if expected_to_fail:
	# pass -> unexpected pass
	if result.code is PASS:
	result.code = XPASS
	# fail -> expected fail
	elif result.code is FAIL:
	result.code = XFAIL
	self.result = result

[sarnex]

sorry so which features are known by the general lit infra and which are known only by our custom code?

[ayylol]

The features in test.config.available_features are known by the general lit infra, the device specific features which are in test.config.sycl_dev_features are not.

[sarnex]

could we add the dev features to available_features before we build/run for each target and remove it after?

[ayylol]

I think the issue would be in situations with multiple devices where a single device might be marked as XFAIL, while the rest pass. If we add all the device features to available_features and let the lit infra do their own XFAIL logic, then this test would be expected to XFAIL, while in reality we expect it to pass, while the XFAIL devices are skipped.

The reason I did it this way was mostly to avoid doing this redundant second XFAIL check.

Also in the case that we change the logic for XFAILs in our code to properly account for running on multiple devices, it is likely that we would want this internal XFAIL check turned off in favor of our own logic.

[sarnex]

does the xfail check only happen once in the lit infra with multiple devices?

[ayylol]

Yes the internal lit infra XFAIL check only happens once after we return the result of our test.

[sarnex]

and the result contains all device runs? sorry just making sure we can't use the lit infra

[ayylol]

Yeah its a single result for the amalgamation of all the device runs.

i.e.,
PASS -> if all device pass (Some but not all devices may be xfail/unsupported, those are skipped)
FAIL -> if a single non xfail/unsupported device fails
XFAIL/XPASS -> can't happen afaik when running a test on multiple devices
UNSUPPORTED -> if all devices are unsupported, or xfailed

[sarnex]

got it, thanks

[ayylol]

Actually fails on BMG, but this statement was an example of one that was being incorrectly processed and marked as XFAIL on BMG.

https://github.com/intel/llvm/actions/runs/15194270458/job/42735337999

[sarnex]

does our outer lit code handle XFAILs for both device specific features and normal features?

[ayylol]

Yep the part just above this

llvm/sycl/test-e2e/format.py

Lines 391 to 395 in e5d5382

	if len(triples) == 1 and test.config.test_mode == "build-only":
	result.code = map_result(test.config.available_features, result.code)
	if len(devices_for_test) == 1:
	device = devices_for_test[0]
	result.code = map_result(test.config.sycl_dev_features[device], result.code)

For build only if the test is only ran for a single triple we only use the normal features, and on full/run-only if the test is only ran for one device we use test.config.sycl_dev_features[device] which contains both the normal features as well as the device features.