Skip to content

feat: benchmark suite #10804

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 80 commits into from
Jul 11, 2025
Merged

feat: benchmark suite #10804

merged 80 commits into from
Jul 11, 2025

Conversation

yash-atreya
Copy link
Member

@yash-atreya yash-atreya commented Jun 18, 2025

Motivation

Closes #10548

Aggregated benchmark suite for testing performance of various commands such as test, build, coverage across multiple foundry versions and repositories.

Solution

  • Benchmarks reside in /benches
  • For each individual command, a criterion benchmark should be written e.g bench/forge_test.rs
  • BenchmarkProject is the utility type to clone various repos and run forge commands on them
  • Benchmarks are invoked via the foundry-bench binary which is a CLI that has flags to specify --versions, --repos, --benchmarks.
  • BenchmarkResults type aggregates the criterion results from the target/criterion/* cache.
  • Finally, the aggregated results file is written that looks like this
  • Adds a benchmarks.yml workflow that enables running benchmarks manually
  • Refer to the README.md on how to run the benchmarks

Currently included

  • forge test
  • forge test - fuzz only
  • forge build with cache
  • forge build with no cache
  • forge coverage

To be addressed in a followup

  • Allowing for repo specific config such as env variables.
  • benchmarks.toml, which allows for specifying repo and version config like this
  • forge build with dynamic_test_linking
  • forge test --isolate
  • Invariant benches

PR Checklist

  • Added Tests
  • Added Documentation
  • Breaking changes
Siteproxy
yash-atreya and others added 7 commits June 10, 2025 16:55
- Automated benchmarking across multiple Foundry versions using hyperfine
- Supports stable, nightly, and specific version tags (e.g., v1.0.0)
- Benchmarks 5 major Foundry projects: account, v4-core, solady, morpho-blue, spark-psm
- Tests forge test, forge build (no cache), and forge build (with cache)
- Generates comparison tables in markdown format
- Uses foundryup for version management
- Exports JSON data for detailed analysis

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Fix relative path issue causing JSON files to fail creation
- Convert benchmark directories to absolute paths using SCRIPT_DIR
- Improve markdown table formatting with proper column names and alignment
- Use unified table generation with string concatenation for better formatting
- Increase benchmark runs from 3 to 5 for more reliable results
- Use --prepare instead of --cleanup for better cache management
- Remove stderr suppression to catch hyperfine errors
- Update table headers to show units (seconds) for clarity

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Copy link
Contributor

@0xrusowsky 0xrusowsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think that the only "must" is support for env vars, the rest are nice-to-have features

yash-atreya and others added 19 commits June 25, 2025 16:56
- run forge build in parallet for forge-test bench
- switch foundry versions
- README specifying prereqs
…nches
- Add `get_benchmark_versions()` helper to read versions from env var
- Update all benchmarks to use version helper for consistency
- Add `--versions` and `--force-install` flags to shell script
- Enable all three benchmarks (forge_test, build_no_cache, build_with_cache)
- Improve error handling for corrupted forge installations
- Remove complex workarounds in favor of clear error messages

The benchmarks now support custom versions via:
  ./run_benchmarks.sh --versions stable,nightly,v1.2.0

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
* feat: criterion benches

* - setup benchmark repos in parallel
- run forge build in parallet for forge-test bench
- switch foundry versions
- README specifying prereqs

* feat: shell script to run benches

* feat: ci workflow, fix script

* update readme

* feat: enhance benchmarking suite with version flexibility

- Add `get_benchmark_versions()` helper to read versions from env var
- Update all benchmarks to use version helper for consistency
- Add `--versions` and `--force-install` flags to shell script
- Enable all three benchmarks (forge_test, build_no_cache, build_with_cache)
- Improve error handling for corrupted forge installations
- Remove complex workarounds in favor of clear error messages

The benchmarks now support custom versions via:
  ./run_benchmarks.sh --versions stable,nightly,v1.2.0

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>

* latest bench

* rm notes

* remove shell based bench suite

---------

Co-authored-by: Claude <noreply@anthropic.com>
* main.rs
* forge version is controlled by the bin
* parses criterion json to collect results - writes to LATEST.md
Copy link
Contributor

📊 Foundry Benchmark Results

📈 View all benchmark results

Foundry Benchmark Results

Generated at: 2025-07-11 05:48:17 UTC

Date: 2025-07-11 05:33:07

Summary

Benchmarked 2 Foundry versions across 2 repositories.

Repositories Tested

  1. ithacaxyz/account
  2. Vectorized/solady

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.2.3-nightly (4f1d965 2025-07-10)

Forge Test

Repository stable nightly
ithacaxyz-account 7.61 s 5.54 s
solady 7.88 s 6.69 s

Forge Fuzz Test

Repository stable nightly
ithacaxyz-account 7.48 s 5.63 s
solady 7.97 s 6.86 s

System Information

  • OS: linux
  • CPU: 8
  • Rustc: unknown

Date: 2025-07-11 05:35:35

Summary

Benchmarked 2 Foundry versions across 2 repositories.

Repositories Tested

  1. ithacaxyz/account
  2. Vectorized/solady

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.2.3-nightly (4f1d965 2025-07-10)

Forge Build (With Cache)

Repository stable nightly
ithacaxyz-account 2.63 s 2.55 s
solady 4.07 s 4.00 s

Forge Build (No Cache)

Repository stable nightly
ithacaxyz-account 2.58 s 2.56 s
solady 4.07 s 3.96 s

System Information

  • OS: linux
  • CPU: 8
  • Rustc: unknown

Date: 2025-07-11 05:38:31

Summary

Benchmarked 2 Foundry versions across 1 repositories.

Repositories Tested

  1. ithacaxyz/account

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.2.3-nightly (4f1d965 2025-07-10)

Forge Coverage

Repository stable nightly
ithacaxyz-account 21.64 s 21.46 s

System Information

  • OS: linux
  • CPU: 8
  • Rustc: unknown

🤖 This comment was automatically generated by the Foundry Benchmarks workflow.

To run benchmarks manually: Go to Actions → "Run workflow"

Copy link
Contributor

📊 Foundry Benchmark Results

📈 View all benchmark results

Foundry Benchmark Results

Generated at: 2025-07-11 05:48:40 UTC

Date: 2025-07-11 05:47:53

Summary

Benchmarked 2 Foundry versions across 2 repositories.

Repositories Tested

  1. ithacaxyz/account
  2. Vectorized/solady

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.2.3-nightly (4f1d965 2025-07-10)

Forge Test

Repository stable nightly
ithacaxyz-account 6.86 s 5.64 s
solady 7.81 s 6.28 s

Forge Fuzz Test

Repository stable nightly
ithacaxyz-account 7.17 s 5.71 s
solady 7.59 s 6.73 s

System Information

  • OS: linux
  • CPU: 8
  • Rustc: unknown

Date: 2025-07-11 05:41:02

Summary

Benchmarked 2 Foundry versions across 2 repositories.

Repositories Tested

  1. ithacaxyz/account
  2. Vectorized/solady

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.2.3-nightly (4f1d965 2025-07-10)

Forge Build (No Cache)

Repository stable nightly
ithacaxyz-account 2.58 s 2.59 s
solady 3.97 s 3.96 s

Forge Build (With Cache)

Repository stable nightly
ithacaxyz-account 2.55 s 2.61 s
solady 4.02 s 4.00 s

System Information

  • OS: linux
  • CPU: 8
  • Rustc: unknown

Date: 2025-07-11 05:43:54

Summary

Benchmarked 2 Foundry versions across 1 repositories.

Repositories Tested

  1. ithacaxyz/account

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.2.3-nightly (4f1d965 2025-07-10)

Forge Coverage

Repository stable nightly
ithacaxyz-account 21.44 s 20.17 s

System Information

  • OS: linux
  • CPU: 8
  • Rustc: unknown

🤖 This comment was automatically generated by the Foundry Benchmarks workflow.

To run benchmarks manually: Go to Actions → "Run workflow"

fix
Copy link
Contributor

📊 Foundry Benchmark Results

Generated at: 2025-07-11 08:33:33 UTC

Forge Test

Repositories Tested

  1. ithacaxyz/account
  2. Vectorized/solady

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.2.3-nightly (d592b3e 2025-07-11)
Repository stable nightly
ithacaxyz-account 7.02 s 5.38 s
solady 7.83 s 6.27 s
📈 View all benchmark results

Forge Build

Repositories Tested

  1. ithacaxyz/account
  2. Vectorized/solady

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.2.3-nightly (d592b3e 2025-07-11)

No Cache

Repository stable nightly
ithacaxyz-account 2.61 s 2.57 s
solady 3.93 s 3.99 s

With Cache

Repository stable nightly
ithacaxyz-account 2.58 s 2.61 s
solady 4.00 s 4.04 s

Forge Coverage

Repositories Tested

  1. ithacaxyz/account

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.2.3-nightly (d592b3e 2025-07-11)
Repository stable nightly
ithacaxyz-account 21.98 s 21.30 s

🤖 This comment was automatically generated by the Foundry Benchmarks workflow.

To run benchmarks manually: Go to Actions → "Run workflow"

yash-atreya and others added 4 commits July 11, 2025 14:12
fix
🤖 Generated with [Foundry Benchmarks](https://github.com/foundry-rs/foundry/actions)

Co-Authored-By: github-actions <github-actions@github.com>
Copy link
Contributor

📊 Foundry Benchmark Results

Generated at: 2025-07-11 10:28:32 UTC

Forge Test

Repositories Tested

  1. ithacaxyz/account
  2. Vectorized/solady

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.2.3-nightly (d592b3e 2025-07-11)
Repository stable nightly
ithacaxyz-account 7.46 s 6.13 s
solady 7.84 s 6.55 s
📈 View all benchmark results

Forge Build

Repositories Tested

  1. ithacaxyz/account
  2. Vectorized/solady

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.2.3-nightly (d592b3e 2025-07-11)

No Cache

Repository stable nightly
ithacaxyz-account 2.61 s 2.58 s
solady 4.06 s 3.99 s

With Cache

Repository stable nightly
ithacaxyz-account 2.63 s 2.63 s
solady 4.02 s 4.09 s

Forge Coverage

Repositories Tested

  1. ithacaxyz/account

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.2.3-nightly (d592b3e 2025-07-11)
Repository stable nightly
ithacaxyz-account 21.52 s 21.63 s

🤖 This comment was automatically generated by the Foundry Benchmarks workflow.

To run benchmarks manually: Go to Actions → "Run workflow"

yash-atreya and others added 2 commits July 11, 2025 16:08
fix
🤖 Generated with [Foundry Benchmarks](https://github.com/foundry-rs/foundry/actions)

Co-Authored-By: github-actions <github-actions@github.com>
Copy link
Contributor

📊 Foundry Benchmark Results

Generated at: 2025-07-11 10:49:57 UTC

Forge Test

Repositories Tested

  1. ithacaxyz/account
  2. Vectorized/solady

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.2.3-nightly (d592b3e 2025-07-11)
Repository stable nightly
ithacaxyz-account 7.21 s 5.94 s
solady 7.88 s 6.57 s

Forge Fuzz Test

Repository stable nightly
ithacaxyz-account 7.47 s 6.13 s
solady 7.84 s 6.84 s
📈 View all benchmark results

Forge Build

Repositories Tested

  1. ithacaxyz/account
  2. Vectorized/solady

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.2.3-nightly (d592b3e 2025-07-11)

No Cache

Repository stable nightly
ithacaxyz-account 2.60 s 2.66 s
solady 3.93 s 4.01 s

With Cache

Repository stable nightly
ithacaxyz-account 2.62 s 2.57 s
solady 4.04 s 4.00 s

Forge Coverage

Repositories Tested

  1. ithacaxyz/account

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.2.3-nightly (d592b3e 2025-07-11)
Repository stable nightly
ithacaxyz-account 21.42 s 21.46 s

System Information

  • OS: linux
  • CPU: 8
  • Rustc: unknown

🤖 This comment was automatically generated by the Foundry Benchmarks workflow.

To run benchmarks manually: Go to Actions → "Run workflow"

mattsse
mattsse previously approved these changes Jul 11, 2025
Copy link
Member

@mattsse mattsse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@grandizzy grandizzy self-requested a review July 11, 2025 11:01
grandizzy
grandizzy previously approved these changes Jul 11, 2025
Copy link
Collaborator

@grandizzy grandizzy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@yash-atreya yash-atreya enabled auto-merge (squash) July 11, 2025 11:36
yash-atreya and others added 2 commits July 11, 2025 17:07

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
🤖 Generated with [Foundry Benchmarks](https://github.com/foundry-rs/foundry/actions)

Co-Authored-By: github-actions <github-actions@github.com>
Copy link
Contributor

📊 Foundry Benchmark Results

Generated at: 2025-07-11 11:48:21 UTC

Forge Test

Repositories Tested

  1. ithacaxyz/account
  2. Vectorized/solady

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.2.3-nightly (d592b3e 2025-07-11)
Repository stable nightly
ithacaxyz-account 6.86 s 6.03 s
solady 7.70 s 6.94 s

Forge Fuzz Test

Repository stable nightly
ithacaxyz-account 6.65 s 6.28 s
solady 7.87 s 6.98 s
📈 View all benchmark results

Forge Build

Repositories Tested

  1. ithacaxyz/account
  2. Vectorized/solady

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.2.3-nightly (d592b3e 2025-07-11)

No Cache

Repository stable nightly
ithacaxyz-account 2.57 s 2.61 s
solady 3.95 s 4.08 s

With Cache

Repository stable nightly
ithacaxyz-account 2.58 s 2.60 s
solady 4.04 s 4.07 s

Forge Coverage

Repositories Tested

  1. ithacaxyz/account

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.2.3-nightly (d592b3e 2025-07-11)
Repository stable nightly
ithacaxyz-account 21.49 s 20.85 s

System Information

  • OS: linux
  • CPU: 8
  • Rustc: unknown

🤖 This comment was automatically generated by the Foundry Benchmarks workflow.

To run benchmarks manually: Go to Actions → "Run workflow"

@github-actions github-actions bot dismissed stale reviews from mattsse and grandizzy via 81cef23 July 11, 2025 11:48
@grandizzy grandizzy self-requested a review July 11, 2025 12:05
@yash-atreya yash-atreya merged commit aa3cc0f into master Jul 11, 2025
22 checks passed
@yash-atreya yash-atreya deleted the yash/foundry-benchmarking-suite branch July 11, 2025 12:18
@github-project-automation github-project-automation bot moved this to Done in Foundry Jul 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

feat: benchmark suite and update README with latest benchmarks
6 participants