[CI] MixedClusterDownsampleRestIT class failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #23528 / 9.0.2_bwc-snapshots
• elasticsearch-pull-request #73515 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #73492 / 8.19.0_bwc-snapshots
• elasticsearch-intake #23402 / 9.0.2_bwc-snapshots
• elasticsearch-intake #23371 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #72816 / 8.19.0_bwc-snapshots

Reproduction Line:

undefined

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

undefined

Issue Reasons:

• [main] 6 failures in class org.elasticsearch.xpack.downsample.MixedClusterDownsampleRestIT (0.6% fail rate in 960 executions)
• [main] 2 failures in step 9.0.2_bwc-snapshots (0.5% fail rate in 412 executions)
• [main] 4 failures in step 8.19.0_bwc-snapshots (1.0% fail rate in 413 executions)
• [main] 3 failures in pipeline elasticsearch-intake (3.4% fail rate in 89 executions)
• [main] 3 failures in pipeline elasticsearch-pull-request (0.9% fail rate in 337 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[gmarouli]

It looks like an infrastructure failure to me. The cluster was not formed because the unicast_hosts.txt was missing while the cluster is configured to use it as the elasticsearch.yml indicates with this setting: discovery.seed_providers: file.

The error we see in the logs:

[2025-05-30T04:18:28,576][WARN ][o.e.d.FileBasedSeedHostsProvider] [test-cluster-0] expected, but did not find, a dynamic hosts list at [/dev/shm/bk/bk-agent-prod-gcp-1748614123473367580/elastic/elasticsearch-intake/x-pack/plugin/downsample/qa/mixed-cluster/build/testrun/v9.0.2_yamlRestTest/temp/test-cluster4758936068662428096/test-cluster-0/config/unicast_hosts.txt]