Running Gemmaclaw Benchmarks

Gemmaclaw includes a built-in E2E agentic benchmark harness that evaluates Gemma models as AI agents with real tool use. The harness dispatches the full agent task suite, captures full conversations including tool calls, and saves structured results ready for PR submission.

Each task runs in an isolated environment with mock tools (email, calendar, tasks, contacts). Results are saved after every task, so an interrupted run can resume without losing completed tests.

Quick Start

# 1. Set up gemmaclaw (auto-detects hardware, installs backend, pulls model)
gemmaclaw setup

# 2. List all benchmark tasks
pnpm benchmark agent list
pnpm benchmark agent list --suite expanded

# 3. Run the full agentic task suite (model auto-selected from your hardware)
pnpm benchmark agent
pnpm benchmark agent --suite expanded

# 4. Run with a specific model
pnpm benchmark agent --model gemma4:31b --quant Q4_K_M --thinking high

# 5. Resume a run, rerun one task, or rerun failed tasks only
pnpm benchmark agent --run-id q4-rtx3090-v1
pnpm benchmark agent --run-id q4-rtx3090-v1 --task email_triage --rerun
pnpm benchmark agent --run-id q4-rtx3090-v1 --rerun-failed

# 6. Rebuild aggregate results from saved per-task artifacts
pnpm benchmark agent --run-id q4-rtx3090-v1 --assemble

# 7. Mock mode: test the harness without a real model (instant)
pnpm benchmark agent --mock --run-id smoke

# 8. Sample the 200-variation template suite before a full sweep
pnpm benchmark agent --suite variants --sample-per-template 2 --sample-seed gemini-flash-smoke-20260513 --backend google-gemini-cli --model gemini-3-flash-preview --run-id variants-gemini-flash-sample --idle-timeout 10 --no-activity-timeout 120 --hard-cap 600

Backends

The benchmark supports two inference backends:

pnpm benchmark agent --model gemma4:e4b
pnpm benchmark agent --model gemma4:31b --ollama-url http://192.168.1.50:11434

# Start llama-server (requires a GGUF model file)
llama-server -m /path/to/model.gguf --port 8080 --n-gpu-layers 99

# Run benchmark against it
pnpm benchmark agent --model gemma3:1b --backend llama-cpp --llama-cpp-url http://127.0.0.1:8080

Benchmark Test Suites

Use --suite to choose which task family to run. The default suite is the published comparison baseline. Expanded suites broaden coverage but need reference validation before their model results are published.

Template Variation Suite

The benchmark now includes 29400 generated tests by turning every expanded Gemmaclaw task into a reusable template with 200 controlled variants underneath it. A template defines the skill being measured, fixture schema, expected behavior, and grading rubric. Variants alter role, context, distractors, wording, output framing, and artifact requirements while preserving the same core capability target.

For harness validation, use a deterministic sample before running the full 29400-case suite. The standard smoke path is 2 variants per template, which gives 294 tasks across all 147 templates and still exercises every template family.

pnpm benchmark agent list --suite variants --sample-per-template 2 --sample-seed gemini-flash-smoke-20260513
pnpm benchmark agent --suite variants --sample-per-template 2 --sample-seed gemini-flash-smoke-20260513 --backend google-gemini-cli --model gemini-3-flash-preview --run-id variants-gemini-flash-sample --idle-timeout 10 --no-activity-timeout 120 --hard-cap 600

Generated variants are not publishable by default. They must pass reference e2e validation, harness-bug review, clean model runs, judge evaluation, and site QA before they appear as comparable results.

Configuration Options

The Agent Task Suite

Tasks evaluate Gemma models as AI agents. Each task sends a natural language request, the agent decides which tools to call, interprets results, and takes follow-up actions. The full conversation is captured for review.

How It Works

1. Hardware detection: The harness uses the same model catalog as gemmaclaw setup to auto-select the best model for your hardware. Override with --model if desired.
2. Seed mock tools: Before each task, a realistic workspace is created with emails, calendar events, contacts, and tasks. Professional/workplace themed.
3. Isolated environment: Each task runs in a fresh gemmaclaw home directory. No state leaks between tasks.
4. Dispatch task: The task prompt is sent via gemmaclaw agent --local. The agent reads emails, checks calendars, creates tasks, sends emails using mock tools.
5. Capture conversation: The full agent loop is recorded: every tool call, tool result, thinking block, and follow-up action.
6. Save per-task results: After each task, the harness writes tasks/<task-id>/result.json, a transcript, and copied session/trajectory logs when available.
7. Resume or rerun: A later command with the same --run-id reuses matching task artifacts. Add --rerun for a selected task or --rerun-failed for only failures.
8. Evaluation (separate step): Results are reviewed against grading criteria. Scores are added to the evaluation files and published to the site.

Results Directory

benchmark-results/
  runs/<model>__<quant>__<timestamp>/
    manifest.json        # Run id, task list, config hash, metadata
    metadata.json        # Hardware, model, quant, config, git SHA
    results.json         # Per-task conversations, tool calls, stats
    tasks/
      <task-id>/
        result.json      # Atomic per-task artifact used for resume
        transcript.txt   # Human-readable transcript for this task
        session.jsonl    # Raw OpenClaw session log, when available
        trajectory.jsonl # Raw OpenClaw trajectory log, when available
    transcripts/         # Human-readable per-task transcripts
    RESULTS.md           # Markdown summary
  evaluations/<model>__<quant>__<timestamp>/
    <task-id>.json       # Grading criteria + evaluation scores

Crash Recovery and Targeted Reruns

The harness treats each task as an independent artifact. If a process dies halfway through a suite run, keep the same --run-id and run the command again. Completed task artifacts with the same config hash are skipped, while missing tasks continue. If a task has a harness error or suspicious transcript, rerun just that task with --task <id> --rerun. If several tasks failed, use --rerun-failed.

# First full Q4 run
pnpm benchmark agent --model gemma4:31b --quant Q4_K_M --thinking high --run-id q4-rtx3090-v1

# Resume after crash
pnpm benchmark agent --model gemma4:31b --quant Q4_K_M --thinking high --run-id q4-rtx3090-v1

# Rerun a suspicious task only
pnpm benchmark agent --model gemma4:31b --quant Q4_K_M --thinking high --run-id q4-rtx3090-v1 --task calendar_create --rerun

# Rebuild aggregate outputs
pnpm benchmark agent --run-id q4-rtx3090-v1 --assemble

Publishing Requirements

Publish only post-template results that have been inspected and evaluated. The benchmark page supports a model-level view, clickable task rows, and a full transcript viewer. Tool calls, tool results, and thinking blocks are shown inline with the conversation and collapsed by default so readers can inspect the evidence without losing the dialogue flow.

Metadata Captured

• Hardware: GPU name, VRAM, CPU model, core count, total RAM
• Model: Name, parameter count, quantization level, format (from Ollama API)
• Config: Backend type, thinking level, context length, URLs
• Environment: Git SHA, Node.js version, OS/platform, timestamps

Submitting Results

1. Run the benchmark on your hardware
2. Check the results in benchmark-results/runs/
3. Open a PR adding your results directory to the gemmaclaw repo
4. Results will be reviewed, evaluated, and published to the site

Smoke Test

After making changes to the benchmark harness, run the smoke test to verify everything works:

# Mock only (instant, no model needed)
bash scripts/benchmark/smoke-test.sh

# Full test: mock + Ollama + llama.cpp
bash scripts/benchmark/smoke-test.sh --real

Prompt-Response Mode (Legacy)

The original prompt-response benchmark is still available. It sends isolated prompts to the backend and checks text output (no agent loop, no tool calling).

pnpm benchmark --local --model gemma4:31b
pnpm benchmark --mock