-
Notifications
You must be signed in to change notification settings - Fork 2k
feat: benchmark suite #10804
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: benchmark suite #10804
Conversation
- Automated benchmarking across multiple Foundry versions using hyperfine - Supports stable, nightly, and specific version tags (e.g., v1.0.0) - Benchmarks 5 major Foundry projects: account, v4-core, solady, morpho-blue, spark-psm - Tests forge test, forge build (no cache), and forge build (with cache) - Generates comparison tables in markdown format - Uses foundryup for version management - Exports JSON data for detailed analysis 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Fix relative path issue causing JSON files to fail creation - Convert benchmark directories to absolute paths using SCRIPT_DIR - Improve markdown table formatting with proper column names and alignment - Use unified table generation with string concatenation for better formatting - Increase benchmark runs from 3 to 5 for more reliable results - Use --prepare instead of --cleanup for better cache management - Remove stderr suppression to catch hyperfine errors - Update table headers to show units (seconds) for clarity 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i think that the only "must" is support for env vars, the rest are nice-to-have features
- run forge build in parallet for forge-test bench - switch foundry versions - README specifying prereqs
…nches
- Add `get_benchmark_versions()` helper to read versions from env var - Update all benchmarks to use version helper for consistency - Add `--versions` and `--force-install` flags to shell script - Enable all three benchmarks (forge_test, build_no_cache, build_with_cache) - Improve error handling for corrupted forge installations - Remove complex workarounds in favor of clear error messages The benchmarks now support custom versions via: ./run_benchmarks.sh --versions stable,nightly,v1.2.0 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com>
* feat: criterion benches * - setup benchmark repos in parallel - run forge build in parallet for forge-test bench - switch foundry versions - README specifying prereqs * feat: shell script to run benches * feat: ci workflow, fix script * update readme * feat: enhance benchmarking suite with version flexibility - Add `get_benchmark_versions()` helper to read versions from env var - Update all benchmarks to use version helper for consistency - Add `--versions` and `--force-install` flags to shell script - Enable all three benchmarks (forge_test, build_no_cache, build_with_cache) - Improve error handling for corrupted forge installations - Remove complex workarounds in favor of clear error messages The benchmarks now support custom versions via: ./run_benchmarks.sh --versions stable,nightly,v1.2.0 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> * latest bench * rm notes * remove shell based bench suite --------- Co-authored-by: Claude <noreply@anthropic.com>
* main.rs * forge version is controlled by the bin * parses criterion json to collect results - writes to LATEST.md
📊 Foundry Benchmark Results📈 View all benchmark resultsFoundry Benchmark ResultsGenerated at: 2025-07-11 05:48:17 UTC Date: 2025-07-11 05:33:07 SummaryBenchmarked 2 Foundry versions across 2 repositories. Repositories TestedFoundry Versions
Forge Test
Forge Fuzz Test
System Information
Date: 2025-07-11 05:35:35 SummaryBenchmarked 2 Foundry versions across 2 repositories. Repositories TestedFoundry Versions
Forge Build (With Cache)
Forge Build (No Cache)
System Information
Date: 2025-07-11 05:38:31 SummaryBenchmarked 2 Foundry versions across 1 repositories. Repositories TestedFoundry Versions
Forge Coverage
System Information
🤖 This comment was automatically generated by the Foundry Benchmarks workflow. To run benchmarks manually: Go to Actions → "Run workflow" |
📊 Foundry Benchmark Results📈 View all benchmark resultsFoundry Benchmark ResultsGenerated at: 2025-07-11 05:48:40 UTC Date: 2025-07-11 05:47:53 SummaryBenchmarked 2 Foundry versions across 2 repositories. Repositories TestedFoundry Versions
Forge Test
Forge Fuzz Test
System Information
Date: 2025-07-11 05:41:02 SummaryBenchmarked 2 Foundry versions across 2 repositories. Repositories TestedFoundry Versions
Forge Build (No Cache)
Forge Build (With Cache)
System Information
Date: 2025-07-11 05:43:54 SummaryBenchmarked 2 Foundry versions across 1 repositories. Repositories TestedFoundry Versions
Forge Coverage
System Information
🤖 This comment was automatically generated by the Foundry Benchmarks workflow. To run benchmarks manually: Go to Actions → "Run workflow" |
📊 Foundry Benchmark ResultsGenerated at: 2025-07-11 08:33:33 UTC Forge TestRepositories TestedFoundry Versions
📈 View all benchmark resultsForge BuildRepositories TestedFoundry Versions
No Cache
With Cache
Forge CoverageRepositories TestedFoundry Versions
🤖 This comment was automatically generated by the Foundry Benchmarks workflow. To run benchmarks manually: Go to Actions → "Run workflow" |
🤖 Generated with [Foundry Benchmarks](https://github.com/foundry-rs/foundry/actions) Co-Authored-By: github-actions <github-actions@github.com>
📊 Foundry Benchmark ResultsGenerated at: 2025-07-11 10:28:32 UTC Forge TestRepositories TestedFoundry Versions
📈 View all benchmark resultsForge BuildRepositories TestedFoundry Versions
No Cache
With Cache
Forge CoverageRepositories TestedFoundry Versions
🤖 This comment was automatically generated by the Foundry Benchmarks workflow. To run benchmarks manually: Go to Actions → "Run workflow" |
🤖 Generated with [Foundry Benchmarks](https://github.com/foundry-rs/foundry/actions) Co-Authored-By: github-actions <github-actions@github.com>
📊 Foundry Benchmark ResultsGenerated at: 2025-07-11 10:49:57 UTC Forge TestRepositories TestedFoundry Versions
Forge Fuzz Test
📈 View all benchmark resultsForge BuildRepositories TestedFoundry Versions
No Cache
With Cache
Forge CoverageRepositories TestedFoundry Versions
System Information
🤖 This comment was automatically generated by the Foundry Benchmarks workflow. To run benchmarks manually: Go to Actions → "Run workflow" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🚀
🤖 Generated with [Foundry Benchmarks](https://github.com/foundry-rs/foundry/actions) Co-Authored-By: github-actions <github-actions@github.com>
📊 Foundry Benchmark ResultsGenerated at: 2025-07-11 11:48:21 UTC Forge TestRepositories TestedFoundry Versions
Forge Fuzz Test
📈 View all benchmark resultsForge BuildRepositories TestedFoundry Versions
No Cache
With Cache
Forge CoverageRepositories TestedFoundry Versions
System Information
🤖 This comment was automatically generated by the Foundry Benchmarks workflow. To run benchmarks manually: Go to Actions → "Run workflow" |
81cef23
Motivation
Closes #10548
Aggregated benchmark suite for testing performance of various commands such as test, build, coverage across multiple foundry versions and repositories.
Solution
/benches
bench/forge_test.rs
BenchmarkProject
is the utility type to clone various repos and run forge commands on themfoundry-bench
binary which is a CLI that has flags to specify--versions
,--repos
,--benchmarks
.BenchmarkResults
type aggregates the criterion results from thetarget/criterion/*
cache.benchmarks.yml
workflow that enables running benchmarks manuallyCurrently included
To be addressed in a followup
dynamic_test_linking
PR Checklist