# AI Coding Agent Benchmark Scorecard

Vibe Coding Editorial Team | Version 1.0 | 2026-11-20

Guide: https://www.vibe-coding.cloud/blog/ai-coding-agent-benchmark#scorecard

## Evaluation plan

- Decision and target workflow:
- Task set and base commits:
- Acceptance checks, regression commands and review criteria:
- Allowed tools, permissions, context, time cap and spend cap:
- Human intervention, retry and infrastructure failure rules:
- Candidates, model and agent versions, and test dates:
- Hourly cost for active human time:

## Attempt log

Use one row per attempt. Record links to the diff, check output and reviewer notes in Evidence. Keep retries as separate rows and apply the declared scoring rule consistently.

| Task and attempt ID | Category | Base commit | Agent and model | Acceptance checks | Regression checks | Reviewer decision | Final outcome | Wall-clock minutes | Active human minutes | Tool and execution spend | Evidence and failure reason |
| --- | --- | --- | --- | --- | --- | --- | --- | ---: | ---: | ---: | --- |
| | | | | | | | | | | | |

## Summary by candidate

| Candidate | Scored attempts | Accepted attempts | Acceptance rate | Infrastructure exclusions | Total human minutes | Tool and execution spend | Total cost | Cost per accepted result |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| | | | | | | | | |

- Acceptance rate = accepted attempts / scored attempts. Include timeouts as scored failures and report infrastructure exclusions separately.
- Total cost = tool and execution spend + active human minutes / 60 × hourly cost. Count supervision, review and rework once. Include cost from failed scored attempts and disclose any excluded infrastructure costs separately.
- Cost per accepted result = total cost across scored attempts / accepted attempts. If there are zero accepts, report total cost and mark the ratio undefined.
- For a time-saving estimate, run comparable tasks without an agent using the same acceptance criteria.

## Decision

- Where did each candidate perform well?
- Which tasks needed human intervention or rework?
- Which results need a repeat run?
- Which workflow should the team adopt, and when should it retest?
