feat: benchmark suite #10804

yash-atreya · 2025-06-18T17:14:20Z

Motivation

Aggregated benchmark suite for testing performance of various commands such as test, build, coverage across multiple foundry versions and repositories.

Solution

Benchmarks reside in /benches
For each individual command, a criterion benchmark should be written e.g bench/forge_test.rs
BenchmarkProject is the utility type to clone various repos and run forge commands on them
Benchmarks are invoked via the foundry-bench binary which is a CLI that has flags to specify --versions, --repos, --benchmarks.
BenchmarkResults type aggregates the criterion results from the target/criterion/* cache.
Finally, the aggregated results file is written that looks like this
Adds a benchmarks.yml workflow that enables running benchmarks manually
Refer to the README.md on how to run the benchmarks

Currently included

forge test
forge test - fuzz only
forge build with cache
forge build with no cache
forge coverage

To be addressed in a followup

Allowing for repo specific config such as env variables.
benchmarks.toml, which allows for specifying repo and version config like this
forge build with dynamic_test_linking
Invariant benches

PR Checklist

Added Tests
Added Documentation
Breaking changes

- Automated benchmarking across multiple Foundry versions using hyperfine - Supports stable, nightly, and specific version tags (e.g., v1.0.0) - Benchmarks 5 major Foundry projects: account, v4-core, solady, morpho-blue, spark-psm - Tests forge test, forge build (no cache), and forge build (with cache) - Generates comparison tables in markdown format - Uses foundryup for version management - Exports JSON data for detailed analysis 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

- Fix relative path issue causing JSON files to fail creation - Convert benchmark directories to absolute paths using SCRIPT_DIR - Improve markdown table formatting with proper column names and alignment - Use unified table generation with string concatenation for better formatting - Increase benchmark runs from 3 to 5 for more reliable results - Use --prepare instead of --cleanup for better cache management - Remove stderr suppression to catch hyperfine errors - Update table headers to show units (seconds) for clarity 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

0xrusowsky

i think that the only "must" is support for env vars, the rest are nice-to-have features

benches/benchmark.sh

benches/repos_and_versions.sh

benches/benchmark.sh

- run forge build in parallet for forge-test bench - switch foundry versions - README specifying prereqs

…nches

- Add `get_benchmark_versions()` helper to read versions from env var - Update all benchmarks to use version helper for consistency - Add `--versions` and `--force-install` flags to shell script - Enable all three benchmarks (forge_test, build_no_cache, build_with_cache) - Improve error handling for corrupted forge installations - Remove complex workarounds in favor of clear error messages The benchmarks now support custom versions via: ./run_benchmarks.sh --versions stable,nightly,v1.2.0 🤖 Generated with Claude Code Co-Authored-By: Claude <[email protected]>

* feat: criterion benches * - setup benchmark repos in parallel - run forge build in parallet for forge-test bench - switch foundry versions - README specifying prereqs * feat: shell script to run benches * feat: ci workflow, fix script * update readme * feat: enhance benchmarking suite with version flexibility - Add `get_benchmark_versions()` helper to read versions from env var - Update all benchmarks to use version helper for consistency - Add `--versions` and `--force-install` flags to shell script - Enable all three benchmarks (forge_test, build_no_cache, build_with_cache) - Improve error handling for corrupted forge installations - Remove complex workarounds in favor of clear error messages The benchmarks now support custom versions via: ./run_benchmarks.sh --versions stable,nightly,v1.2.0 🤖 Generated with Claude Code Co-Authored-By: Claude <[email protected]> * latest bench * rm notes * remove shell based bench suite --------- Co-authored-by: Claude <[email protected]>

* main.rs * forge version is controlled by the bin * parses criterion json to collect results - writes to LATEST.md

0xrusowsky

looks like a great initial version to get the ball rolling!
for now i can't think of any more improvements than the listed ones in the follow-up section 👍

DaniPopes · 2025-07-02T18:30:55Z

.github/workflows/benchmarks.yml

+        if: github.event.inputs.pr_number != ''
+        uses: actions/github-script@v7
+        with:
+          script: |


can we move as much logic as possible away from the yaml file into separate files

benches/src/lib.rs

test-forge-regex

benches/FUZZ_BENCH.md

benches/forge_build_no_cache.rs

DaniPopes · 2025-07-02T18:46:13Z

.github/workflows/benchmarks.yml

+jobs:
+  forge-test:
+    name: Run forge_test and forge_fuzz_test benchmarks
+    runs-on: ubuntu-latest


we should not rely on github runners for accurate measurements

DaniPopes · 2025-07-02T18:47:14Z

benches/FUZZ_BENCH.md

+| Repository        | stable | nightly |
+| ----------------- | ------ | ------- |
+| ithacaxyz-account | 4.34 s | 3.69 s  |
+| solady            | 3.68 s | 2.92 s  |


this is ok for now, but walltime is extremely unreliable and should not be the only metric we use

benches/src/main.rs

DaniPopes · 2025-07-02T18:53:58Z

benches/src/lib.rs

+/// - "v0.2.0" - Specific version tag
+/// - "commit-hash" - Specific commit hash
+/// - "nightly-rev" - Nightly build with specific revision
+pub static FOUNDRY_VERSIONS: &[&str] = &["stable", "nightly"];


these are the defaults correct?

github-actions · 2025-07-04T08:09:09Z

📊 Foundry Benchmark Results

Click to view detailed benchmark results

Foundry Benchmark Results

Generated at: 2025-07-04 08:09:08 UTC

Date: 2025-07-04 08:08:57

Summary

Benchmarked 2 Foundry versions across 2 repositories.

Repositories Tested

Foundry Versions

stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
nightly: forge Version: 1.2.3-nightly (6092317 2025-07-04)

Forge Fuzz Test

Repository	stable	nightly
ithacaxyz-account	41.62 s	35.77 s
solady	38.38 s	33.33 s

Forge Test

Repository	stable	nightly
ithacaxyz-account	41.72 s	35.01 s
solady	38.99 s	34.21 s

System Information

OS: linux
CPU: 4
Rustc: rustc 1.88.0 (6b00bc388 2025-06-23)

Date: 2025-07-04 08:02:32

Summary

Benchmarked 2 Foundry versions across 2 repositories.

Repositories Tested

Foundry Versions

stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
nightly: forge Version: 1.2.3-nightly (6092317 2025-07-04)

Forge Build (With Cache)

Repository	stable	nightly
ithacaxyz-account	5.75 s	5.58 s
solady	8.13 s	8.14 s

Forge Build (No Cache)

Repository	stable	nightly
ithacaxyz-account	5.70 s	5.61 s
solady	8.06 s	8.12 s

System Information

OS: linux
CPU: 4
Rustc: rustc 1.88.0 (6b00bc388 2025-06-23)

Date: 2025-07-04 08:03:26

Summary

Benchmarked 2 Foundry versions across 1 repositories.

Repositories Tested

ithacaxyz/account

Foundry Versions

stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
nightly: forge Version: 1.2.3-nightly (6092317 2025-07-04)

Forge Coverage

Repository	stable	nightly
ithacaxyz-account	33.46 s	33.47 s

System Information

OS: linux
CPU: 4
Rustc: rustc 1.88.0 (6b00bc388 2025-06-23)

🤖 This comment was automatically generated by the Foundry Benchmarks workflow.

To run benchmarks manually: Go to Actions → "Run workflow"

yash-atreya · 2025-07-04T08:27:23Z

Benchmarks ran in ci, but took super long and do not indicate accurate results. Probably because of github runner? @DaniPopes

github-actions · 2025-07-04T10:01:10Z

📊 Foundry Benchmark Results

Click to view detailed benchmark results

Foundry Benchmark Results

Generated at: 2025-07-04 10:01:09 UTC

Date: 2025-07-04 10:00:55

Summary

Benchmarked 2 Foundry versions across 2 repositories.

Repositories Tested

Foundry Versions

stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
nightly: forge Version: 1.2.3-nightly (6092317 2025-07-04)

Forge Fuzz Test

Repository	stable	nightly
ithacaxyz-account	40.42 s	35.22 s
solady	36.40 s	32.22 s

Forge Test

Repository	stable	nightly
ithacaxyz-account	40.30 s	34.61 s
solady	39.02 s	33.67 s

System Information

OS: linux
CPU: 4
Rustc: rustc 1.88.0 (6b00bc388 2025-06-23)

Date: 2025-07-04 09:54:36

Summary

Benchmarked 2 Foundry versions across 2 repositories.

Repositories Tested

Foundry Versions

stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
nightly: forge Version: 1.2.3-nightly (6092317 2025-07-04)

Forge Build (No Cache)

Repository	stable	nightly
ithacaxyz-account	5.74 s	5.62 s
solady	8.13 s	8.17 s

Forge Build (With Cache)

Repository	stable	nightly
ithacaxyz-account	5.73 s	5.66 s
solady	8.17 s	8.19 s

System Information

OS: linux
CPU: 4
Rustc: rustc 1.88.0 (6b00bc388 2025-06-23)

Date: 2025-07-04 09:54:42

Summary

Benchmarked 2 Foundry versions across 1 repositories.

Repositories Tested

ithacaxyz/account

Foundry Versions

stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
nightly: forge Version: 1.2.3-nightly (6092317 2025-07-04)

Forge Coverage

Repository	stable	nightly
ithacaxyz-account	34.96 s	35.31 s

System Information

OS: linux
CPU: 4
Rustc: rustc 1.88.0 (6b00bc388 2025-06-23)

🤖 This comment was automatically generated by the Foundry Benchmarks workflow.

To run benchmarks manually: Go to Actions → "Run workflow"

yash-atreya and others added 7 commits June 10, 2025 16:55

parallel benchmarking

a8e28b1

refac: mv to benches/ dir

0a569d7

feat: criterion benches

09bfb57

fix: install foundry versions at once

8a673c9

nit

c1e52dd

github-project-automation bot added this to Foundry Jun 18, 2025

yash-atreya mentioned this pull request Jun 18, 2025

feat: benches using criterion #10805

Merged

3 tasks

Merge branch 'master' into yash/foundry-benchmarking-suite

01f1af6

0xrusowsky reviewed Jun 22, 2025

View reviewed changes

benches/benchmark.sh Outdated Show resolved Hide resolved

benches/repos_and_versions.sh Outdated Show resolved Hide resolved

benches/repos_and_versions.sh Outdated Show resolved Hide resolved

benches/benchmark.sh Outdated Show resolved Hide resolved

yash-atreya and others added 19 commits June 25, 2025 16:56

- setup benchmark repos in parallel

0dc8187

- run forge build in parallet for forge-test bench - switch foundry versions - README specifying prereqs

feat: shell script to run benches

9f13124

feat: ci workflow, fix script

7d1d85a

Merge branch 'yash/foundry-benchmarking-suite' into yash/criterion-be…

1a1587a

…nches

update readme

b667106

latest bench

fcba242

rm notes

858d8d9

remove shell based bench suite

44835cf

unified benchmarker -

792c592

* main.rs * forge version is controlled by the bin * parses criterion json to collect results - writes to LATEST.md

parallel bench

b18141d

refac

35d9861

refac benchmark results table generation

2392590

cleanup main.rs

4f896ae

rm dep

7edb40e

cleanup main.rs

d0a1525

deser estimate

03b54fd

nit

6c82d2f

yash-atreya added 4 commits July 1, 2025 16:52

fmt

afcf3ed

license

ed82d8a

coverage bench

4dac5c3

nits

3887370

yash-atreya marked this pull request as ready for review July 1, 2025 12:20

yash-atreya requested review from DaniPopes, klkvr, mattsse, grandizzy and zerosnacks as code owners July 1, 2025 12:20

yash-atreya added 2 commits July 1, 2025 18:31

clippy

cbd17d8

clippy

4fa2315

yash-atreya requested a review from 0xrusowsky July 1, 2025 13:22

0xrusowsky previously approved these changes Jul 1, 2025

View reviewed changes

separate benches into different jobs in CI

afc236b

yash-atreya dismissed 0xrusowsky’s stale review via afc236b July 2, 2025 06:50

DaniPopes requested changes Jul 2, 2025

View reviewed changes

yash-atreya added 9 commits July 3, 2025 13:03

remove criterion

3c1c8cb

feat: hyperfine setup in foundry-bench

51d10b5

forge version details: hash and date

a79409d

run benches again - run cov with --ir-min

354a8fe

del

3666090

bench in separate ci jobs

fcd2d82

move combine bench results logic to scripts

51225d6

setup foundryup in ci

f23b52b

setup foundryup fix

09a174d

clippy

3bbc762

feat: benchmark suite #10804

Are you sure you want to change the base?

feat: benchmark suite #10804

Uh oh!

Conversation

yash-atreya commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Solution

PR Checklist

Uh oh!

0xrusowsky left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

0xrusowsky left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DaniPopes Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DaniPopes Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

DaniPopes Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

DaniPopes Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

yash-atreya Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 4, 2025

📊 Foundry Benchmark Results

Foundry Benchmark Results

Summary

Repositories Tested

Foundry Versions

Forge Fuzz Test

Forge Test

System Information

Summary

Repositories Tested

Foundry Versions

Forge Build (With Cache)

Forge Build (No Cache)

System Information

Summary

Repositories Tested

Foundry Versions

Forge Coverage

System Information

Uh oh!

yash-atreya commented Jul 4, 2025

Uh oh!

github-actions bot commented Jul 4, 2025

📊 Foundry Benchmark Results

Foundry Benchmark Results

Summary

Repositories Tested

Foundry Versions

Forge Fuzz Test

Forge Test

System Information

Summary

Repositories Tested

Foundry Versions

Forge Build (No Cache)

Forge Build (With Cache)

yash-atreya commented Jun 18, 2025 •

edited

Loading

0xrusowsky left a comment •

edited

Loading