Guide · · 8 min read

How to Benchmark AI Coding Agents on Your Own Codebase

What SWE-bench and Terminal-Bench measure, and how to run a fair comparison on your own codebase.

By Vibe Coding Editorial Team

AI coding agent benchmark illustrated by two Git branches starting at the same commit and ending at matching test checks

How should you benchmark an AI coding agent?

Run the same representative tasks from the same repository commits, with frozen acceptance checks and equal time and intervention rules. Report accepted tasks, human review and rework time, and total cost per accepted result. Use public benchmarks to shortlist candidates and use your repository pilot to choose a workflow.

Jump to the benchmark scorecard

What SWE-bench and Terminal-Bench measure

A benchmark score describes a particular run. Check the dataset split and version, the model, and the agent software that supplies tools and instructions. Compare the time budget, retry rules and scoring procedure before interpreting a difference.

BenchmarkWhat it measuresWhen to use itWhat to check
SWE-bench VerifiedResolve GitHub issues and pass repository tests. Verified is a human-filtered set of 500 tasks.Shortlist candidates for issue-to-patch work.OpenAI’s audit flagged flawed tests in a selected subset and evidence of training exposure.
SWE-bench ProResolve longer software engineering tasks. Scale describes public, held-out open-source and private subsets.Check longer repository tasks and confirm which split was scored.A later OpenAI audit found problems such as hidden requirements and tests that accepted incomplete fixes.
Terminal-BenchComplete tasks inside terminal environments. The leaderboard identifies both the model and the agent.Shortlist agents for command-line work.Check the exact benchmark version and agent setup. An IDE workflow or code review cycle may involve different work.

OpenAI documented test-design and contamination problems in its February 2026 Verified audit, then withdrew its recommendation to switch to Pro after auditing Pro in July 2026. The benchmark designers and the audit authors published different evaluations of the datasets. Read both when a vendor relies on these scores.

Productivity takes a different measurement. METR’s early-2025 randomized study assigned 246 tasks from 16 experienced open-source developers to conditions that allowed or disallowed AI. The researchers estimated that allowing AI increased completion time by 19% in that setting. Use a manual baseline in your own pilot if the decision depends on time saved.

How to run an AI coding agent evaluation on your repository

1. Pick a decision and a small task set

Choose the workflow you need to staff, such as fixing bugs in a TypeScript codebase. Start with 8 to 12 tasks spanning bug fixes, feature changes, refactors and test writing. Use recent private issues cleared for the tool, or scrubbed equivalents. Avoid publishing proprietary task material.

2. Write acceptance checks before any run

Pin a base commit for each task. Record the expected behavior, allowed files and the regression commands. Keep independent grader files outside the agent’s editing scope. Confirm a bug test fails before the fix and existing checks pass on the base commit. For a refactor, define the observable behavior that must be preserved.

3. Run each candidate from a clean copy

Use the same task brief, base commit, test fixtures, time cap and rules for human help. Record the model, agent version, reasoning settings, tool permissions, network access and configured project instructions. If one product requires a different execution environment, document the difference.

4. Grade the result independently

Run the acceptance checks from a clean verifier context and have a reviewer inspect the diff against the brief. Log the terminal outcome as accepted, rejected, timeout or infrastructure error. Keep the original attempt if you later retry. Record setup failures separately and apply the same retry rule to both candidates.

5. Compare the work your team actually does

Count retries, review and corrections in the totals. If you need a productivity estimate, run comparable tasks without agent assistance under the same acceptance criteria. Rotate the order for repeated trials to reduce familiarity effects.

For example, ask each tool to add empty-state feedback to a search results page. Require the feedback after a successful empty response, preserve loading and error behavior, and check keyboard navigation. Keep acceptance tied to the observable result so several valid implementations can pass.

AI coding agent benchmark scorecard

Record one row for each attempt. A run counts as accepted when it meets the required behavior, passes regression checks and clears reviewer sign-off within the agreed rules. Use the same decision rule for both candidates.

  • Task: ID, category, base commit, brief and acceptance criteria.
  • Setup: agent and model versions, permissions, budget and time cap.
  • Outcome: acceptance, regression and review results, with links to evidence.
  • Effort: wall-clock minutes, active human minutes and tool or execution spend.

Acceptance rate = accepted attempts ÷ scored attempts. Count timeouts as scored failures. Report infrastructure exclusions separately.

Cost per accepted result = (total tool and execution spend + total active human hours × hourly cost) ÷ accepted attempts. Include spend and review time from failures. Keep wall-clock time separate from active human time.

Download the Markdown scorecard

Worked example with 10 tasks per candidate

These fictional values illustrate the calculation. Both candidates get one attempt per task, and active human time is valued at $60 per hour.

MetricCandidate ACandidate B
Accepted attempts7 of 108 of 10
Tool and execution spend$18$24
Active human time72 min48 min
Total cost$90$72
Cost per accepted result$12.86$9.00

B spends $6 more on the tool and execution, but takes 24 fewer minutes of human work. One task separates the acceptance counts. Repeat borderline runs and check the task-by-task results before treating B as the better fit across the whole repository.

How to turn results into a tool decision

Compare results by task category and write down where each agent needed help. You may choose one tool for bug fixes and keep a human-led process for broad refactors. Record the configuration and the date, then rerun a small subset after a major change to the model, agent or repository.

Use the AI tool finder to shortlist candidates, the AI coding tool comparison to compare workflows, and the AI-generated code review checklist to define review criteria. See our editorial methodology for how we document product claims.

Frequently asked questions

What is an AI coding agent benchmark?

It is a set of coding tasks, a repeatable test environment and a scoring procedure. The result applies to the model, agent, settings and tasks used in that evaluation.

What is the difference between SWE-bench and Terminal-Bench?

SWE-bench evaluates repository issue resolution with code patches and tests. Terminal-Bench evaluates tasks completed through a terminal. Compare results within the same benchmark version and configuration.

How many tasks do I need for a coding agent pilot?

Use 8 to 12 representative tasks as a starting point for a small internal pilot. This is a practical editorial recommendation. Report raw counts, cover several task types and repeat close or inconsistent runs before making a broader decision.

How do I measure the cost of a coding agent?

Add tool and execution spend to the cost of human supervision, review and rework across all scored attempts, including failures. Divide by the number of accepted attempts. If none are accepted, report the total spend and zero accepts without dividing.

Can I compare agents that use different models?

Yes, when you are evaluating the products as your team would use them. Record each configuration. If your question is which model performs better within one agent, hold the agent and other settings constant and vary the model.

Built a tool for coding or evaluation?

Submit a relevant tool for editorial consideration in the Vibe Coding directory.

Submit a tool