AgentForge

One file in. A better agent out.

AgentForge is a self-improving AI agent optimization platform written in Rust. Feed it a single agent file — a declarative spec describing your AI agent's system prompt, tools, output schemas, and behavioral constraints — and it autonomously generates test scenarios, runs the agent, scores every execution trace, and iterates on the specification until it converges on a measurably better version.

Table of Contents

• Overview
• Architecture
• Project Structure
• Quick Start
• LLM Providers
• Agent File Format
• CLI Usage
• REST API
• Scoring Dimensions
• Promotion Gatekeeper
• Multi-Agent Testing
• Benchmark Suites
• Configuration
• Running Tests
• GitHub Actions Marketplace
• CI/CD Integration
• Contributing
• Roadmap

Overview

AI agent development has a painful quality gap: teams ship prompts and tool definitions with little systematic testing, and improvements are made based on anecdote rather than measurement. AgentForge removes the manual burden by orchestrating a fully automated improvement loop:

parse → generate tests → run → trace → score → optimize → gate → promote

Humans set the quality bar. The platform handles the repetitive evaluation and iteration work.

Core Features (MVP)

Architecture

┌──────────────────────────────────────────────────────────────────────┐
│                         AGENTFORGE PLATFORM                          │
│                                                                      │
│  INPUT: Agent File (YAML / JSON / MD)                                │
│         system_prompt · tools[] · output_schema                      │
│         constraints[] · model  · sampling_config                     │
│                       │                                              │
│                       ▼                                              │
│  ┌────────────────────────────────┐                                  │
│  │  F-01: AGENT LOADER            │                                  │
│  │  Parser → Schema Validator     │                                  │
│  │  → Version Store (SHA-based)   │                                  │
│  └────────────────┬───────────────┘                                  │
│                   ▼                                                  │
│  ┌───────────────────────────────┐                                   │
│  │  F-02: SCENARIO GENERATOR     │                                   │
│  │  Schema-derived (50%)         │                                   │
│  │  Adversarial    (30%)         │                                   │
│  │  Domain-seeded  (20%)         │                                   │
│  └────────────────┬──────────────┘                                   │
│                   ▼                                                  │
│  ┌────────────────────────────────────────────────────────────┐      │
│  │  F-03: AGENT RUNNER (parallel workers, full trace capture) │      │
│  └────────────────────────────────┬───────────────────────────┘      │
│                                   ▼                                  │
│  ┌────────────────────────────────────────────────────────────┐      │
│  │  F-04: TRACE ANALYZER & SCORER                             │      │
│  │  Deterministic assertions + LLM-as-judge                   │      │
│  │  Weighted aggregate score + Failure cluster report         │      │
│  └─────────────────────────────────┬──────────────────────────┘      │
│                                    ▼                                 │
│  ┌──────────────────────────────────────┐                            │
│  │  F-05: OPTIMIZER                     │                            │
│  │  Prompt rewrite · Tool desc rewrite  │                            │
│  │  Schema tighten · Example inject     │                            │
│  │  → 5–20 Candidate Variants           │                            │
│  └──────────────────────┬───────────────┘                            │
│                         ▼                                            │
│  ┌─────────────────────────────────┐                                 │
│  │  F-06: PROMOTION GATEKEEPER     │                                 │
│  │  Score Gate (+3% over champion) │                                 │
│  │  Regression Gate (≥99% pass)    │                                 │
│  │  Stability Gate (3 seeds)       │                                 │
│  └─────────────────────┬───────────┘                                 │
│                        ▼                                             │
│  ┌───────────────────────────────────────┐                           │
│  │  PROMOTED AGENT FILE                  │                           │
│  │  (versioned, diffed, changelog)       │                           │
│  └───────────────────────────────────────┘                           │
│                                                                      │
│  F-07: REST API   │  F-08: CLI / GitHub Actions                      │
└──────────────────────────────────────────────────────────────────────┘

Project Structure

This is a Cargo workspace with 16 crates:

agentforge/
├── Cargo.toml                  # Workspace root
├── Cargo.lock
├── docker-compose.yml          # PostgreSQL for local dev
├── Dockerfile
├── .env.example                # Environment variable template
├── migrations/                 # SQLx database migrations
│   ├── 001_agent_versions.sql
│   ├── 002_eval_runs.sql
│   ├── 003_scenarios.sql
│   ├── 004_traces.sql
│   ├── 005_shadow_runs.sql
│   ├── 006_finetune_exports.sql
│   ├── 007_benchmarks.sql
│   ├── 008_trace_cost.sql
│   ├── 009_failure_cluster_api_error.sql
│   └── 010_opt_tracking.sql
├── fixtures/
│   ├── customer-support-agent.yaml     # Native YAML example
│   └── agentforge-evaluator.agent.md   # Copilot .agent.md example
└── crates/
    ├── agentforge-core/        # Shared types, errors, traits (AgentFile, EvalRun, Trace, Scenario…)
    ├── agentforge-parser/      # Agent file parsing (YAML, JSON, Markdown/Copilot frontmatter)
    ├── agentforge-scenarios/   # Scenario generation (schema-derived, adversarial, domain-seeded)
    ├── agentforge-runner/      # Parallel agent execution + full trace capture
    ├── agentforge-scorer/      # Deterministic assertions + LLM-as-judge scoring
    ├── agentforge-optimizer/   # Variant generation + self-improvement loop
    ├── agentforge-gatekeeper/  # Three-gate promotion logic
    ├── agentforge-db/          # PostgreSQL repository layer (SQLx)
    ├── agentforge-api/         # REST API (Axum 0.8)
    ├── agentforge-cli/         # CLI binary (Clap 4)
    ├── agentforge-benchmarks/  # Standard benchmark comparison suite
    ├── agentforge-finetune/    # Fine-tune dataset exporter (JSONL)
    ├── agentforge-multiagent/  # Multi-agent composition testing
    ├── agentforge-observability/ # OTLP trace export hooks
    ├── agentforge-online-eval/ # Shadow-mode real-traffic comparison
    └── agentforge-redteam/     # Adversarial safety red-team probes

Quick Start

Prerequisites

• Rust 1.83+ (install via rustup)
• Docker + Docker Compose
• One of: OpenAI API key, Anthropic API key, NVIDIA NIM API key, or Ollama running locally (no key required)
• Node.js 20+ and npm (only required if developing the web/ dashboard locally; production deployment uses the pre-built Docker image)

1. Clone and start infrastructure

git clone https://github.com/bhavinkotak/agentforge.git
cd agentforge

# Start PostgreSQL and Redis
docker-compose up -d

# Copy and configure environment
cp .env.example .env

Edit .env and add credentials for your chosen LLM provider:

OpenAI (default)

OPENAI_API_KEY=sk-...
AGENTFORGE_JUDGE_PROVIDER=openai
AGENTFORGE_JUDGE_MODEL=gpt-4o

Anthropic

ANTHROPIC_API_KEY=sk-ant-...
AGENTFORGE_JUDGE_PROVIDER=anthropic
AGENTFORGE_JUDGE_MODEL=claude-3-5-haiku-20241022

NVIDIA NIM (free tier at build.nvidia.com — no credit card for free models)

NVIDIA_API_KEY=nvapi-...
AGENTFORGE_JUDGE_PROVIDER=nvidia
AGENTFORGE_NVIDIA_MODEL=meta/llama-3.1-8b-instruct
AGENTFORGE_JUDGE_MODEL=meta/llama-3.1-8b-instruct

Ollama (fully local, no API key needed)

# Install Ollama: https://ollama.com/download
ollama serve
ollama pull llama3.2:3b     # or any model you prefer

# In .env:
AGENTFORGE_JUDGE_PROVIDER=ollama
AGENTFORGE_JUDGE_MODEL=llama3.2:3b
AGENTFORGE_OLLAMA_MODEL=llama3.2:3b
AGENTFORGE_OLLAMA_BASE_URL=http://localhost:11434/v1

Running inside Docker with Ollama on the host? Set AGENTFORGE_OLLAMA_BASE_URL=http://host.docker.internal:11434/v1 so the container can reach the Ollama process running on your machine.

2. Run database migrations

export DATABASE_URL="postgres://agentforge:agentforge@localhost:5432/agentforge"
cargo install sqlx-cli --no-default-features --features postgres
sqlx migrate run

3. Build and run the API server

cargo build --release
DATABASE_URL=$DATABASE_URL ./target/release/agentforge-api
# Server starts on http://127.0.0.1:8080

4. Run your first eval via CLI

# Run a full evaluation cycle
./target/release/agentforge run \
  --agent fixtures/customer-support-agent.yaml \
  --scenarios 50

# Show a scorecard diff between two versions
./target/release/agentforge diff <version-id-1> <version-id-2>

# Promote the winning version (pass the run-id returned by `agentforge run`)
./target/release/agentforge promote <run-id>

LLM Providers

AgentForge uses two independent LLM roles: the agent (the model being evaluated) and the judge (scores traces). Both are configured separately so you can mix providers — e.g., evaluate an Ollama agent with an OpenAI judge, or evaluate a NVIDIA NIM agent with an Anthropic judge. The API enforces that the two providers differ to prevent circular bias.

Set AGENTFORGE_JUDGE_PROVIDER to route the judge (scorer + optimizer) to the correct backend. Set provider on a POST /api/v1/runs request (or --provider via CLI) to route the agent itself.

OpenAI

OPENAI_API_KEY=sk-...
AGENTFORGE_JUDGE_PROVIDER=openai   # default
AGENTFORGE_JUDGE_MODEL=gpt-4o      # default

The AGENTFORGE_JUDGE_BASE_URL variable overrides the base URL for any OpenAI-compatible endpoint (useful for Azure OpenAI, LM Studio, or other proxies):

AGENTFORGE_JUDGE_BASE_URL=https://myazure.openai.azure.com/openai/deployments/gpt-4o
AGENTFORGE_JUDGE_API_KEY=<azure-key>

Anthropic

ANTHROPIC_API_KEY=sk-ant-...
AGENTFORGE_JUDGE_PROVIDER=anthropic
AGENTFORGE_JUDGE_MODEL=claude-3-5-haiku-20241022

NVIDIA NIM

Sign up for a free API key at build.nvidia.com. Many models are available with no credits required.

NVIDIA_API_KEY=nvapi-...
AGENTFORGE_JUDGE_PROVIDER=nvidia
AGENTFORGE_NVIDIA_MODEL=meta/llama-3.1-8b-instruct   # model for agent runs
AGENTFORGE_JUDGE_MODEL=meta/llama-3.1-8b-instruct    # model for judge/scorer

Free-tier models (no credits required):

NVIDIA NIM enforces single tool calls per request (parallel_tool_calls=false). The base URL is always https://integrate.api.nvidia.com/v1 — it cannot be overridden.

Ollama (fully local)

Install Ollama, then pull a model and start the server:

ollama pull llama3.2:3b   # ~2 GB
ollama serve              # starts on http://localhost:11434

Configure AgentForge to use it:

AGENTFORGE_JUDGE_PROVIDER=ollama
AGENTFORGE_JUDGE_MODEL=llama3.2:3b
AGENTFORGE_OLLAMA_MODEL=llama3.2:3b          # model used for agent runs
AGENTFORGE_OLLAMA_BASE_URL=http://localhost:11434/v1

Ollama requires no API key. At runtime, any model field in an agent YAML file is transparently overridden with AGENTFORGE_OLLAMA_MODEL, so cloud-targeted agent files work without modification.

Docker + Ollama on host: replace localhost with host.docker.internal:

AGENTFORGE_OLLAMA_BASE_URL=http://host.docker.internal:11434/v1

AWS Bedrock

AWS Bedrock provides managed access to foundation models from Anthropic, Meta, Mistral, and others. Because Bedrock uses SigV4 request signing rather than API keys, it requires a self-hosted runner (GitHub-hosted runners do not have AWS credentials).

Prerequisites:

• An AWS account with Bedrock access enabled for the model you want to use.
• IAM credentials with the bedrock:InvokeModel permission.
• A self-hosted GitHub Actions runner (or local/cloud VM) with AWS credentials available via environment variables or an instance role.

CLI

AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...
AWS_REGION=us-east-1

./target/release/agentforge run \
  --agent fixtures/customer-support-agent.yaml \
  --provider bedrock \
  --judge-provider openai    # judge must be a different provider

Environment variables

# AWS credentials (can also be supplied via instance role / IRSA — no env vars needed in that case)
AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...
AWS_SESSION_TOKEN=...        # only required for temporary credentials
AWS_REGION=us-east-1

# Bedrock model for agent runs (defaults to anthropic.claude-3-haiku-20240307-v1:0)
AGENTFORGE_BEDROCK_MODEL=anthropic.claude-3-5-sonnet-20241022-v2:0

Supported Bedrock model IDs

The Bedrock base URL (https://bedrock-runtime.<region>.amazonaws.com) is derived automatically from AWS_REGION. It cannot be overridden. Cross-region inference profiles are supported by setting AWS_REGION to the inference profile region.

Agent File Format

AgentForge accepts agent files in the following formats:

• AgentForge native YAML (recommended)
• GitHub Copilot .agent.md — YAML frontmatter + Markdown system prompt body
• OpenAI Assistants API JSON
• Anthropic Claude system prompt + tool block JSON
• LangChain / LangGraph agent YAML
• CrewAI agent definition YAML

GitHub Copilot .agent.md Format

Compatible with agents from github/awesome-copilot. The frontmatter holds metadata; the Markdown body becomes the system prompt.

---
name: 'Code Review Expert'
description: 'Specialist in reviewing code for security and maintainability'
model: GPT-4.1
tools: ['read', 'search/codebase', 'github/*']
---

# Code Review Expert

You are an expert code reviewer specializing in security, performance,
and maintainability.

## Review Focus Areas

- **Security**: Check for injection vulnerabilities and data exposure
- **Performance**: Identify N+1 queries and unnecessary allocations
- **Maintainability**: Evaluate clarity and SOLID principles

Frontmatter fields:

Copilot tool capability references are mapped to ToolDefinition entries so AgentForge can reason about them during scenario generation and scoring.

AgentForge Native YAML Schema

# agent.yaml — AgentForge native schema v1
agentforge_schema_version: "1"
name: "customer-support-agent"
version: "2.1.0"

model:
  provider: openai          # openai | anthropic | nvidia_nim | ollama | bedrock
  model_id: gpt-4o
  temperature: 0.2
  max_tokens: 2048

system_prompt: |
  You are a helpful customer support agent for Acme Corp.
  Always greet the user by name if known.
  Never share pricing without verifying entitlement first.

tools:
  - name: get_order_status
    description: "Retrieve status of a customer order by order ID."
    parameters:
      type: object
      properties:
        order_id:
          type: string
          description: "The order identifier, format: ORD-XXXXXXXX"
      required: [order_id]

output_schema:
  type: object
  properties:
    response:
      type: string
    action_taken:
      type: string
      enum: [escalate, resolved, needs_followup, no_action]
    confidence:
      type: number
      minimum: 0
      maximum: 1
  required: [response, action_taken]

constraints:
  - "Never mention competitor products."
  - "Do not provide refunds without running check_refund_eligibility first."
  - "Always confirm order ID before calling get_order_status."

eval_hints:
  domain: customer_support
  typical_turns: 3
  critical_tools: [get_order_status, check_refund_eligibility]
  pass_threshold: 0.85    # minimum aggregate score to promote
  scenario_count: 200

CLI Usage

agentforge <COMMAND> [OPTIONS]

Commands:
  run      Run a full eval cycle (parse → generate → run → score → optimize → gate)
  diff     Show scorecard diff between two agent versions
             Usage: agentforge diff <version-id-1> <version-id-2>
  promote  Promote a candidate version to champion
             Usage: agentforge promote <run-id>
  help     Print help

Options for `run`:
  --agent <FILE>               Path to agent YAML/JSON file (required)
  --scenarios <N>              Number of scenarios to generate (default: 100)
  --concurrency <N>            Parallel workers (default: 10)
  --seed <N>                   Random seed for reproducibility (default: 42)
  --provider <NAME>            Agent LLM provider: openai | anthropic | nvidia | ollama | bedrock (default: openai)
                               (ollama requires `ollama serve` running locally; bedrock requires AWS credentials and a self-hosted runner)
  --judge-provider <NAME>      Judge LLM provider (must differ from --provider; default: anthropic)
  --threshold <F>              Pass threshold 0.0–1.0 (default: 0.85)
  --output-json <FILE>         Write full scorecard JSON to FILE (used by the GitHub Action)
  --dry-run                    Validate agent file and preview scenario count; no LLM calls made
  --max-cost <USD>             Abort if estimated cost exceeds USD (e.g. --max-cost 5.00); 0 = no cap
  --agent-format <FMT>         Override format detection: native_yaml | openai_json | anthropic_json | langchain_yaml | crewai_yaml | copilot_agent_md
  --weight-task <F>            Override task-completion weight (default: 0.35)
  --weight-tool <F>            Override tool-selection weight (default: 0.20)
  --weight-args <F>            Override argument-correctness weight (default: 0.20)
  --weight-schema <F>          Override schema-compliance weight (default: 0.15)
  --weight-instr <F>           Override instruction-adherence weight (default: 0.07)
  --weight-path <F>            Override path-efficiency weight (default: 0.03)
  --red-team                   Append adversarial red-team probes to standard scenarios
  --cost-optimize              After eval, recommend cheaper model alternatives
  --watch                      Re-run the evaluation automatically when the agent file is saved.
                               Polls for file changes every 500 ms. Press Ctrl-C to stop.

Exit codes:
  0  — All gates passed, version promoted (or no promotion needed)
  1  — Gatekeeper blocked promotion
  2  — Error (parse failure, DB connection, etc.)

REST API

The API server runs on http://0.0.0.0:8080 by default.

An OpenAPI 3.1 spec is available at docs/openapi.yaml and can be loaded into Swagger UI, Postman, or any OpenAPI-compatible tool.

Endpoints

Concurrency limit on POST /runs: to prevent accidental LLM cost floods, the server rejects new eval-run requests with HTTP 429 when AGENTFORGE_MAX_CONCURRENT_RUNS active background tasks are already running (default: 10). For high-throughput CI, raise this value in your deployment env. For per-client rate limiting, place a reverse proxy (nginx / Cloudflare) in front of the API.

Scenario count limit on POST /runs: scenario_count must not exceed AGENTFORGE_MAX_SCENARIOS (default: 2000). Requests that exceed this value are rejected with HTTP 400.

auto_optimize on POST /runs: set "auto_optimize": true to enable the iterative self-improvement loop. After the eval completes, if the aggregate score is below the threshold (default: 0.92), the optimizer enters a round-based loop. Each round it generates candidate variants (prompt rewrites, tool-description rewrites, example injections, constraint tightenings), quick-evaluates each on a 25-scenario subset, and saves the best-performing variant if it improves by more than 1 percentage point. The loop terminates with a status of converged (score ≥ threshold), no_improvement (no variant helped), max_iterations (hit max_opt_iterations, default 5), or failed (unrecoverable error). The parent SHA is recorded on every saved variant for full lineage tracking. The opt_status, opt_rounds, opt_best_score, and opt_best_agent_id fields on the run record reflect the final loop state.

Authentication: set AGENTFORGE_API_KEY to require a Bearer token on all /api/v1/* endpoints. Requests without a valid Authorization: Bearer <key> header will receive HTTP 401. The /health endpoint is always unauthenticated. When the env var is unset the server runs in unauthenticated development mode.

Example: Start an eval run

# Upload agent file (YAML or JSON; Content-Type should match the file format)
curl -X POST http://localhost:8080/api/v1/agents \
  -H "Content-Type: application/yaml" \
  --data-binary @fixtures/customer-support-agent.yaml

# Start eval run (all fields except agent_id are optional)
# Add -H "Authorization: Bearer $AGENTFORGE_API_KEY" when authentication is enabled
curl -X POST http://localhost:8080/api/v1/runs \
  -H "Content-Type: application/json" \
  -d '{
    "agent_id": "550e8400-e29b-41d4-a716-446655440000",
    "scenario_count": 100,
    "concurrency": 10,
    "seed": 42,
    "threshold": 0.92,
    "provider": "openai",
    "judge_provider": "anthropic",
    "auto_optimize": true,
    "max_opt_iterations": 5
  }'

# Poll for run results (replace with the run UUID returned above)
curl http://localhost:8080/api/v1/runs/7dc53df0-c5fa-4f6c-b6b5-8d2d19afe5b1

Example: Shadow run (online eval)

Shadow runs let you compare a candidate agent against the current champion on a sample of real traffic without touching your production path. The champion and candidate run in parallel on the same sampled scenarios; a comparison_result summarising win/lose/tie counts and per-dimension deltas is saved when the run completes.

# Start a shadow run with 10 % traffic routed to the candidate
curl -X POST http://localhost:8080/api/v1/shadow-runs \
  -H "Content-Type: application/json" \
  -d '{
    "champion_agent_id": "550e8400-e29b-41d4-a716-446655440000",
    "candidate_agent_id": "6ba7b810-9dad-11d1-80b4-00c04fd430c8",
    "traffic_percent": 10
  }'

# Poll for results
curl http://localhost:8080/api/v1/shadow-runs/<shadow-run-id>

Example: Export fine-tune data

After a completed eval run, export passing trace pairs as a JSONL file for supervised fine-tuning. The three supported formats map to OpenAI, Anthropic, and HuggingFace datasets-compatible schemas.

# Enqueue an export job (format defaults to "openai")
curl -X POST http://localhost:8080/api/v1/exports/finetune \
  -H "Content-Type: application/json" \
  -d '{
    "run_id": "7dc53df0-c5fa-4f6c-b6b5-8d2d19afe5b1",
    "format": "openai"
  }'

# Poll until status == "complete", then read file_path for the JSONL location
curl http://localhost:8080/api/v1/exports/finetune/<export-id>

Supported format values: "openai" (chat completion pairs), "anthropic" (messages API pairs), "huggingface" (HuggingFace datasets-compatible JSONL).

Example: Run a standard benchmark

Benchmark runs evaluate an agent against a fixed task set from a public benchmark suite and return an accuracy score plus a percentile rank against the baseline public leaderboard.

# Start a GAIA benchmark run
curl -X POST http://localhost:8080/api/v1/benchmarks \
  -H "Content-Type: application/json" \
  -d '{
    "agent_id": "550e8400-e29b-41d4-a716-446655440000",
    "suite": "gaia"
  }'

# Poll for results (accuracy + percentile_rank populated when complete)
curl http://localhost:8080/api/v1/benchmarks/<benchmark-run-id>

Supported suite values: "gaia", "agentbench", "webarena". See Benchmark Suites for details.

Scoring Dimensions

Every execution trace is scored across six dimensions:

Weights are configurable via environment variables (AGENTFORGE_WEIGHT_TASK, AGENTFORGE_WEIGHT_TOOL, AGENTFORGE_WEIGHT_ARGS, AGENTFORGE_WEIGHT_SCHEMA, AGENTFORGE_WEIGHT_INSTR, AGENTFORGE_WEIGHT_PATH) or via per-run CLI flags (--weight-task, --weight-tool, etc.). The judge LLM must use a different provider from the agent to prevent circular bias — enforcement is at the provider level (e.g., openai vs anthropic), not the individual model ID. The API returns HTTP 400 if provider and judge_provider are the same string.

Failure Clusters

Traces are automatically grouped into failure clusters:

Promotion Gatekeeper

A candidate variant must clear all three gates to be promoted:

1. Score Gate — Aggregate score must exceed the current champion by at least +3% (configurable via AGENTFORGE_SCORE_GATE_DELTA).

2. Regression Gate — Must pass ≥ 99% of the scenarios the current champion passes (configurable via AGENTFORGE_REGRESSION_GATE_THRESHOLD). Prevents "robbing Peter to pay Paul" improvements.

3. Stability Gate — Must be evaluated on at least 3 independent random seeds before comparison, to account for LLM non-determinism (configurable via AGENTFORGE_STABILITY_SEEDS).

If multiple candidates pass all gates, the one with the highest aggregate score is promoted. Promotion creates a new versioned agent file with an auto-generated changelog entry.

First run (no champion): When no champion exists yet, all three gates are automatically waived and the candidate is promoted unconditionally. This bootstraps the system on the first evaluation.

Configuration

All configuration is via environment variables. See .env.example for the full list.

REDIS_URL vs AGENTFORGE_*: Redis uses the bare REDIS_URL key (compatible with most hosting platforms). All other app-level settings use the AGENTFORGE_ prefix.

AGENTFORGE_HOST default: The server binds to 127.0.0.1 by default for security (localhost only). Set to 0.0.0.0 to accept external connections — docker-compose overrides this automatically for container networking.

Running Tests

# Start PostgreSQL first
docker-compose up -d postgres

# Run all tests
DATABASE_URL="postgres://agentforge:agentforge@localhost:5432/agentforge" \
  cargo test --workspace

# Run tests for a specific crate
cargo test -p agentforge-scorer
cargo test -p agentforge-runner
cargo test -p agentforge-gatekeeper

# Run with output
cargo test --workspace -- --nocapture

The test suite covers:

• Agent file parsing (all 6 formats)
• Scenario generation (schema-derived, adversarial, domain-seeded)
• Runner execution with mocked LLM
• Scoring logic (all 6 dimensions)
• Optimizer variant generation
• Gatekeeper promotion logic
• REST API integration tests
• Database repository tests

GitHub Actions Marketplace

AgentForge is published as a reusable GitHub Action. No Rust toolchain, database, or build step required in your workflow — the action downloads a pre-built binary from the latest release automatically.

- uses: bhavinkotak/agentforge@v1
  with:
    agent_file: './agents/my-agent.yaml'
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
    DATABASE_URL: ${{ secrets.AGENTFORGE_DATABASE_URL }}

Inputs

Outputs

Use Cases

Block a merge when agent quality drops

Run a full eval cycle on every PR that touches agent files. Fail the check if the score falls below threshold — preventing regressions from being merged.

name: Agent Quality Gate
on:
  pull_request:
    paths: ['agents/**', '*.agent.md']

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: bhavinkotak/agentforge@v1
        id: eval
        with:
          agent_file: './agents/customer-support-agent.yaml'
          scenarios: '200'
          threshold: '0.85'
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          DATABASE_URL: ${{ secrets.AGENTFORGE_DATABASE_URL }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}   # enables PR comment
      - name: Report scorecard
        if: always()
        run: |
          echo "Pass rate: ${{ steps.eval.outputs.pass_rate }}"
          echo "Aggregate score: ${{ steps.eval.outputs.aggregate_score }}"
          echo "Promoted: ${{ steps.eval.outputs.promoted }}"

Nightly improvement loop

Run AgentForge on a schedule to continuously generate, evaluate, and auto-promote improved agent variants.

name: Nightly Agent Improvement
on:
  schedule:
    - cron: '0 2 * * *'   # 02:00 UTC every night

jobs:
  improve:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: bhavinkotak/agentforge@v1
        with:
          agent_file: './agents/customer-support-agent.yaml'
          scenarios: '500'
          concurrency: '20'
          threshold: '0.88'
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          DATABASE_URL: ${{ secrets.AGENTFORGE_DATABASE_URL }}

Evaluate a GitHub Copilot .agent.md file

AgentForge natively parses Copilot agent files — just point agent_file at the .agent.md.

- uses: bhavinkotak/agentforge@v1
  with:
    agent_file: '.github/agents/code-review.agent.md'
    scenarios: '100'
    provider: 'openai'
    judge_provider: 'anthropic'
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
    DATABASE_URL: ${{ secrets.AGENTFORGE_DATABASE_URL }}

Tip: Pin to a specific version for reproducibility: bhavinkotak/agentforge@v1.2.3

CI/CD Integration

Self-hosted / custom CI

If you prefer to build from source (e.g., air-gapped environments), use the CLI directly:

# .github/workflows/agent-eval.yml
name: Agent Evaluation

on:
  push:
    paths: ['agents/**']
  pull_request:
    paths: ['agents/**']

jobs:
  evaluate:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_USER: agentforge
          POSTGRES_PASSWORD: agentforge
          POSTGRES_DB: agentforge
        ports:
          - 5432:5432
      redis:
        image: redis:7-alpine
        ports:
          - 6379:6379

    steps:
      - uses: actions/checkout@v4

      - name: Install Rust
        uses: dtolnay/rust-toolchain@stable

      - name: Build AgentForge CLI
        run: cargo build --release -p agentforge-cli

      - name: Run Migrations
        env:
          DATABASE_URL: postgres://agentforge:agentforge@localhost:5432/agentforge
        run: cargo sqlx migrate run

      - name: Run AgentForge Evaluation
        env:
          DATABASE_URL: postgres://agentforge:agentforge@localhost:5432/agentforge
          REDIS_URL: redis://localhost:6379
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          ./target/release/agentforge run \
            --agent ./agents/customer-support-agent.yaml \
            --scenarios 200 \
            --threshold 0.85

Exit codes: 0 = passed/promoted, 1 = gatekeeper blocked, 2 = error.

Contributing

Contributions are welcome! Please read CONTRIBUTING.md for the full process. In short:

1. Open an issue first for significant changes so we can discuss the approach.
2. Fork and branch — create a feature branch from main.
3. Follow code conventions — cargo fmt and cargo clippy --all-targets must pass.
4. Add tests — all new behaviour must be covered.
5. Open a PR — all PRs require approval from the project maintainer (@bhavinkotak) before merging.

See CONTRIBUTING.md for dev environment setup, commit message conventions, and the full review checklist.

Technical Stack

Roadmap

v2 (Post-MVP)

Web dashboard (web/) is already included in this repo and served via the Docker Compose stack on port 3000.

Multi-Agent Testing

The agentforge-multiagent crate lets you evaluate composed agent teams described as directed graphs. Each node in the graph is a full AgentFile; edges thread the output of one node into the input context of the next.

Graph spec (YAML)

# multi-agent-graph.yaml
id: "customer-triage-graph"
nodes:
  - id: "classifier"
    role: "Classify the incoming request"
    agent:
      name: "request-classifier"
      model:
        provider: openai
        model_id: gpt-4o-mini
      system_prompt: |
        Classify the customer request into one of: billing, technical, general.
        Output JSON: {"category": "<category>", "confidence": <0-1>}
      output_schema:
        type: object
        properties:
          category: { type: string, enum: [billing, technical, general] }
          confidence: { type: number }
        required: [category, confidence]

  - id: "resolver"
    role: "Resolve the classified request"
    agent:
      name: "request-resolver"
      model:
        provider: openai
        model_id: gpt-4o
      system_prompt: |
        You are a resolution specialist. The classifier node output is available
        in your context under the key "classifier". Use it to route appropriately.

edges:
  - from: "classifier"
    to: "resolver"
    input_key: "classifier"   # key injected into resolver's scenario context

API

Pass the graph spec to POST /api/v1/agents with Content-Type: application/yaml. The API will validate the graph topology (cycle detection included) and register each node agent. Then start a run as normal; the runner automatically executes nodes in topological order and threads outputs between them.

Scoring

Each node is scored independently across the six standard dimensions. A composite_score (unweighted mean of node scores) and a pass_rate across all node traces are reported in the MultiAgentScorecard returned by GET /api/v1/runs/:id/scorecard.

Benchmark Suites

The agentforge-benchmarks crate evaluates an agent against three public benchmark suites. Start a benchmark run via POST /api/v1/benchmarks and poll GET /api/v1/benchmarks/:id for results.

Supported suites

Accuracy and percentile rank

• accuracy — fraction of tasks answered correctly (correct / total_tasks).
• percentile_rank — where your score falls relative to submissions tracked by AgentForge's built-in normalizer. A rank of 0.75 means the agent outperforms 75 % of tracked runs for that suite.

Example output

{
  "id": "b1e3...",
  "agent_id": "550e...",
  "suite": "gaia",
  "status": "complete",
  "total_tasks": 165,
  "correct": 132,
  "accuracy": 0.8,
  "percentile_rank": 0.82,
  "started_at": "2026-05-25T02:00:00Z",
  "completed_at": "2026-05-25T02:18:43Z"
}

Web Dashboard

The React/TypeScript dashboard is served on port 3000 (Docker) or npm run dev (port 5173) from the web/ directory.

Agents List

Agents are grouped by name so every agent family appears as a single row:

• Latest Version column shows the most recently created version.
• Versions count badge (stack icon) shows how many versions exist for that name.
• Click a row to open the Agent Detail page.

Agent Detail

• Summary panel — model, status, system prompt, tools, constraints.
• Score History — chart of aggregate scores over time.
• Version History — card listing all versions for the same agent name, sorted by semver ascending. The current version is highlighted with an indigo background and a current badge. Click any row to navigate to that version. Click the diff icon to compare system prompts.

Diff Viewer

The diff viewer uses a real LCS-based unified diff algorithm:

• Removed lines are shown in red.
• Added lines are shown in green.
• Unchanged context lines are shown in gray.

Trigger a diff from the Runs page (scorecard diff between two versions) or from the Version History card (system-prompt diff between any two agent versions).

Privacy

AgentForge is a self-hosted platform. It sends no telemetry, analytics, or usage data to any external service. All evaluation data, agent files, and traces remain within your own infrastructure.

License

MIT