GitHub ran four frontier models through Copilot's harness. None won every task.

GitHub just published benchmarks for the layer most people ignore: the agent harness around the model. Its engineers ran four frontier models through Copilot’s own harness and compared each one against the model maker’s native tool, the one the lab ships to show off its own model. The takeaway isn’t a leaderboard.

The June 25 write-up from GitHub engineers Shibani Basava and Carlos Castro tested Claude Sonnet 4.6, Claude Opus 4.7, GPT-5.4, and GPT-5.5. Each model ran inside Copilot’s harness and, separately, inside its vendor tool: Claude Code for the two Claude models, Codex CLI for the two GPT models. The scaffolding around a model, GitHub argues, shapes task success and token cost about as much as the model choice does. Their framing: “While the model provides the raw intelligence, the harness shapes how effectively that intelligence is applied.”

What GitHub actually measured

Start with the vocabulary, because it carries the whole story. An agentic harness is the code that wraps a language model and turns it into something that can act: it decides which tools the model can call, how the planning loop runs, and what context gets packed into each request. GitHub runs one shared harness across the Copilot CLI, the Copilot app, and Copilot code review, so any efficiency gain at this layer compounds across every surface at once. That shared design is also why a clean model-versus-model test is even possible here: the scaffolding stays fixed while the intelligence inside it changes.

The eval spans five benchmarks. SWE-bench Verified is the industry standard, 500 human-validated bug-fix tasks pulled from open-source Python repositories. SWE-bench Pro raises the difficulty with multi-step engineering work. SkillsBench checks whether an agent picks and triggers the right skills. TerminalBench scores command-line workflows. Win-Hill, an internal test, runs everything inside Windows containers to see whether the results hold on a different operating system. Every run got a two-hour timeout.

Here’s how the four benchmarked models line up inside Copilot’s harness:

The four models, head to head

The pattern GitHub found is consistent. Run a Claude model inside Copilot’s harness and it burns fewer tokens than Claude Code does on the same tasks. Run a GPT model and the same holds, with one exception worth naming.

Claude Opus 4.7

Opus is the quality play. On TerminalBench 2.0, GitHub plots it in the upper right: the highest task-resolution rate of any model tested, reached at the highest cost per task. Inside Copilot’s harness it still used fewer tokens than Claude Code across every benchmark, so the scaffolding trims the bill without capping the ceiling. When a task needs deep, multi-step reasoning, this is the model GitHub’s data points to. You just pay for it. For routine edits that resolution edge rarely earns back the token spend, which is the whole argument against defaulting every request to the biggest model.

Claude Sonnet 4.6

Sonnet is the middle lane. It landed on par with Claude Code on task resolution, inside what GitHub calls the variance you’d expect from the models’ own randomness, while spending fewer tokens in every configuration. That makes it the sensible default for the bulk of agent work that doesn’t need Opus-grade reasoning. The efficiency story echoes a trap Anthropic’s own pricing keeps hitting: cheaper-per-token doesn’t always mean cheaper-per-task once a new tokenizer changes how much text a single token holds. Same token count, different mileage.

GPT-5.5

GPT-5.5 is where the cost story gets interesting. On TerminalBench, GitHub puts both GPT models on the left of the chart: strong resolution at the lowest cost per task, the best raw value in the lineup. Copilot’s harness used fewer tokens than Codex CLI almost everywhere. The exception was SWE-bench Verified, where the harness ran 4% less efficient than Codex on GPT-5.5. On that one Python bug-fixing benchmark, OpenAI’s own tool kept an edge. Everywhere else, including the harder SWE-bench Pro, Copilot came out ahead or even.

GPT-5.4

GPT-5.4 tells almost the same story with a slightly wider gap. It shared the best-value corner with GPT-5.5, but on SWE-bench Verified the Copilot harness ran 7% less efficient than Codex CLI, the largest single regression in the whole eval. Read that as a ceiling on the harness claim, not a refutation: one benchmark, one model family, where the vendor’s own tuning still wins. For teams picking a budget workhorse, 5.4 is the cheap option, with the caveat that its weakest task type is exactly the classic bug-fix benchmark most people cite.

Our pick for agentic coding

For day-to-day agentic coding, GPT-5.5 inside Copilot’s harness is the one to reach for first. It sits in the best-value corner of TerminalBench, gives up almost nothing on task resolution, and the harness keeps its token bill under Codex CLI’s on every benchmark but one. Reserve Opus 4.7 for the genuinely hard tickets, the refactors and multi-file debugging where a higher resolution rate actually saves you a second round. Sonnet 4.6 is a fine middle default if you already live in the Claude ecosystem. The deeper point GitHub is selling is that the harness is now a product surface worth optimizing, the same bet Vercel made when it turned agents into a first-class primitive in its AI SDK. Model access is table stakes. Orchestration is the differentiator.

What this means for you

If you’re on Copilot today, the practical move is to stop hard-coding one model for everything. GitHub’s own conclusion, backed by the numbers, is that no single model wins across task types: a cheaper model often reaches the same result, and a stronger one earns its keep only on harder reasoning. Copilot’s Auto mode tries to make that call for you by weighing task intent against model health. There’s also Rubber Duck, where one model family critiques another’s work to lift quality past what either hits alone. The efficiency race is spreading well past model weights, showing up in runtimes too, like the way Deno 2.9 chased a faster cold start. My read: treat these benchmarks as a nudge to turn on Auto, test your own repo against two or three models before you commit, and watch that SWE-bench Verified gap if bug-fixing is your main workload.