pytest based synthesis tests

[marco66colombo]

Description

This PR introduces synthesis tests into the pytest framework for hls4ml.

The goal of this change is to automate the validation of HLS synthesis reports against predefined baselines to ensure stability and correctness over time, using pytest.

Main Changes

• New submodule in tests/pytests called baselines, which is a Git submodule pointing to an external repository containing synthesis baselines.

  • The structure of that repository follows the format:
    backend_tool/version/test_artifacts.json
  • Note: Currently, this is a temporary personal repo and must be replaced by a proper one under the organization before merging.

• Baseline management:

  • Baselines are organized by backend (e.g., Vivado, Vitis), tool version (e.g., 2020.1, 2023.1), and test case.
  • A manual and recurrent update policy for these baselines should be defined and enforced.

• New conftest.py file to manage shared pytest fixtures:

  • Introduces the synthesis_config fixture, which provides:
    • Whether to run synthesis tests (RUN_SYNTHESIS environment variable).
    • Tool versions (via env vars, e.g., VIVADO_VERSION, defaulting to 2023.1).
    • Build arguments (backend-dependent) for model.build(...).

• Synthesis logic centralized in synthesis_helpers.py:

  • Main method: test_synthesis.
  • If RUN_SYNTHESIS=true, runs synthesis and compares the report to baseline.
  • Skips test if:
    • Tool is not available,
    • Baseline is missing
    • Currently, in the above cases, the whole test is skipped, so it might has to change in future. It is also true that this method is called after any other assert in the test case, so it would still fail in case of a wrong assert in the code before this method is called.

• Tolerances for report values can be specified using get_tolerance.

  • Currently, the tolerance values are placeholders and need to be properly defined.

• CI integration:

  • Synthesis reports are saved as JSON and collected as artifacts via GitLab CI (ci-template.yml updated accordingly).

• Current test coverage:

  • Only test_keras_api.py includes synthesis tests at the moment.
  • Easily extendable to other tests:
    • Add baseline_file_name to the test,
    • Call test_synthesis().

• OneAPI backend is currently skipped due to missing synthesis reports support.

---

Type of change

• Other: improvment of testing infrastrucutre using pytest

---

Tests

Synthesis tests are run conditionally and compared against versioned baselines if RUN_SYNTHESIS=true is set in the environment.

Checklist

• I have read the guidelines for contributing.
• I have commented my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings.
• I have installed and run pre-commit on the files I edited or added.
• I have added tests that prove my feature works.

[jmitrevs]

fpga_emu is actually just C compilation, so this needs to be changed, though to what I am not sure. The descriptions are given in https://github.com/oneapi-src/oneAPI-samples/tree/release/2025.0/DirectProgramming/C%2B%2BSYCL_FPGA/Tutorials/GettingStarted/fpga_compile#four-compilation-options and they correspond to fpga_emu, report, fpga_sim, and fpga. I would guess report unless you want to run cosim, in which case it's fpga_sim. The first two don't require Quartus, while the last two do.

[jmitrevs]

Can we test this to actually run SYNTHESIS tests?

[marco66colombo]

yes, but if the env var 'RUN_SYNTHESIS' is not set, the default is False.
I can add a line in the ci-template.yml and export RUN_SYNTHESIS=True

[jmitrevs]

I generally agree about storing the baselines in a dedicated repo. Is CERN gitlab or regular github preferred? We create the testing containers in gitlab. I don't really have a preference.