Is GPT-6.1 Sol Ultrafast better than Claude Opus 5.5 for coding?

Not for my real work yet. In 42 timed runs Sol Ultrafast was faster on every job, but it only cleared my pre-set rule on short made-up jobs. On real config changes it was 11% faster against a 25% bar, and on the one real feature it met fewer requirements than Opus. I kept Opus 5.5 as my default coding agent.

How much faster was Sol Ultrafast on real coding work?

On the two real config jobs, the median time to accepted work was 671.9 seconds for Sol against 754.9 seconds for Opus, about 11% faster across six runs each. One job alone was about 37% faster and the other was nearly even. On the real feature, Sol finished sooner but neither model produced work that passed.

Why didn't faster token generation make the coding agent much faster?

Because the model was not most of the clock. On the real config jobs Sol's model time had a median of 50 seconds against 120 for Opus, yet the median run took about 11 to 13 minutes in total. The agent spent the rest running commands, builds and tests, which no model speed changes.

Can an AI reviewer catch what a coding agent missed?

Not reliably in my test. An AI completion checker missed the main gap in the big feature in all 6 runs at both medium and high effort, and it raised false alarms on 23 of 36 runs that had passed everything. Two AI judges picked the same winner on only 8 of 24 pairs.

How do you test AI coding models fairly?

Replay changes you already merged, write hidden requirement checks the agent never sees, and write the switching rule down before the first run. Time each run until the work passes every check, count failures as never accepted, and repeat each job at least three times per model.

Which AI agents benefit most from a faster model like Sol Ultrafast?

In my setup, the helpers that read and report: an explorer that searches code, a librarian that looks up docs and an observer that reads images. They mostly wait on the model, so speed shows up directly. The agents that write code stay on Claude Opus 5.5.

Should I trust an AI coding agent when it says the task is done?

No. On the big feature in my test, all six runs said they were done. None passed the hidden checks, all six missed the same written requirement, and five of the six left a script-injection hole on a dashboard. Check the work against requirements the agent did not write.

October 11, 2026

Is a Faster AI Coding Model Better? How I Tested It

I ran 42 timed coding runs to see if a faster AI model could replace my default. It was quicker, but it didn't clear the bar I set for real work.

TL;DR

I kept Claude Opus 5.5 as my default coding agent. GPT-6.1 Sol in Ultrafast mode was faster on every job I gave it, but faster tokens did not turn into much faster finished work. On real config changes it was 11% faster. I had set the bar at 25% before I ran anything. On the one real feature, NEITHER model finished: all six runs missed the same requirement, and all six said they were done. The AI reviewers I hoped would catch that did not.

Key takeaways:

42 timed runs across six jobs, scored on hidden checks the agents were never given.
On real config work Sol’s model time was 50 seconds against 120 for Opus. Total time dropped only 11%, because tools and tests took most of the clock.
Sol cleared my rule only on made-up jobs that took a few minutes or less.
On the big feature, 0 of 6 runs passed, and 5 of 6 left a script-injection hole on a dashboard.
An AI completion checker caught the main gap in 0 of 6 runs and raised false alarms on 23 of 36 clean runs.
My helper agents that search, look things up and read images already run on Sol Ultrafast. The agents that write code stay on Opus.

Why did I test this at all?

Two weeks ago I wrote about which AI model I actually use for coding. That was usage: what I reached for and how long I kept it. It said nothing about whether the work was any good.

On Thursday, October 8, OpenAI started rolling out Ultrafast mode for GPT-6.1 Sol. OpenAI says it generates tokens up to eight times faster than standard speed in Codex. The announcement said “rolling out today” without a start time, and a launch report was up by 2:08 p.m. Central.

That night I switched my helper agents, the ones that search, look things up and read images, to Sol Ultrafast. The coding agent was the open question. My agents spend a lot of time waiting on a model, so it was an obvious one. If I switch my coding agent to Sol Ultrafast, do I get finished work sooner without giving up quality?

I had already run a quick test with tiny tasks of about 40 seconds each. Tiny tasks cannot tell you how a model handles a 15-minute change with real tests behind it. So I built a bigger one.

My priorities, in order: quality, then total time to finished work, then cost. This post is about the first two.

What exactly did I compare?

Claude Opus 5.5 from Anthropic, at medium effort. My default coding agent.
GPT-6.1 Sol Ultrafast from OpenAI, at medium effort. Ultrafast was about a day old.
Sol Ultrafast at high effort on two of the made-up jobs, to see if more thinking helped.

Same agent setup, same tools, same instructions, same starting code. Only the model changed.

Quick glossary

Effort: how much the model is allowed to think before it acts. Medium is my default.
Hidden checks: requirement tests the agent never sees. I score with them.
Requirement coverage: the share of a job’s requirements that pass the hidden checks.
Time to accepted: wall-clock time from start until the work passes every check. Work that never passes counts as never accepted.
Model time: the part of a run spent waiting on the model, as opposed to running commands and tests.
Silent defect: a run that reports it is done while a requirement still fails.

How did I keep the test fair?

Six jobs, three runs each

Three real jobs, each a replay of a change I had already merged. Every agent started from the commit before my change and got the same written handoff.

Real 1: a config change that delivers a credential to a documentation tool. 12 requirements.
Real 2: a config change that tightens what my agents are allowed to do. 15 requirements.
Real 3: a cross-language feature in Rust and TypeScript, built from a written spec. 19 requirements.

Three made-up Python jobs, built so I knew the right answer:

Made-up 1: build a settings-file parser from a 24-point spec, checked by 129 hidden tests.
Made-up 2: migrate a 39-module app to a new database layer without breaking any of its 224 tests.
Made-up 3: find the causes of four reported symptoms. One cause lives in vendor code and should be reported, not edited.

Six jobs, three runs each, two models: 36 runs. Add six runs of Sol at high effort and that is 42. All 42 ran on the evening of Friday, October 9, starting at about 6:20 p.m. Central, roughly 28 hours after that launch report. There were no invalid runs and no retries.

Hidden checks, and an audit

The agents worked against the visible tests that came with each job. I scored them on hidden ones.

One honest caveat. The hidden tests were not locked away at the system level. An agent that went looking could have found them. So I audited every run. None read them, none edited files outside their job, none weakened a test, and none wrote anywhere live.

Time until accepted, not tokens per second

The clock ran from the start of a run until the work passed every check. If the visible checks failed, the agent got one round of feedback. If the work never passed, it counted as never accepted, which ranks behind any finish time.

The rule came first

I wrote the switching rule down before the first run. For each kind of job, Sol had to:

match Opus on quality: no more silent defects, requirement coverage within 5 points, about as many mergeable runs, and zero integrity problems
reach accepted work with a median time at least 25% lower than Opus

I picked 25% before I saw a single number. That matters, because once you see results it is very easy to move the line.

Flow diagram of how each run was scored: the agent works on the job, visible checks run with one feedback round allowed, hidden requirement checks decide acceptance, accepted runs get a time to accepted, and failed runs count as never accepted, all judged against a rule set before any run.

Source: my test design, written before the first run on October 9, 2026.

What happened on each job?

Sol was faster on every job. The question was whether it was fast enough, at the same quality, where it counted.

Job	Opus 5.5	Sol Ultrafast	Sol high
Real 1: config	100%, 3 of 3, 11.5 min	100%, 3 of 3, 11.1 min	not run
Real 2: config	100%, 3 of 3, 18.9 min	100%, 3 of 3, 11.9 min	not run
Real 3: feature	63%, 0 of 3, 13.0 min	50%, 0 of 3, 7.2 min	not run
Made-up 1: parser	100%, 3 of 3, 3.9 min	100%, 3 of 3, 1.0 min	100%, 3 of 3, 1.4 min
Made-up 2: migration	100%, 3 of 3, 1.8 min	100%, 3 of 3, 1.2 min	not run
Made-up 3: debugging	100%, 3 of 3, 0.8 min	100%, 3 of 3, 0.6 min	100%, 3 of 3, 0.8 min

Each cell: average requirement coverage, runs accepted, median wall-clock time.

Grouped bar chart of median minutes per job for Opus 5.5 and Sol Ultrafast across six jobs, with the real feature marked as not accepted in any of three runs for either model, and a note that real config work was 11% faster against a 25% bar.

Source: my 42-run test, October 9, 2026. Median wall-clock time per job, three runs per model.

Here is how each kind of job came out against the rule:

Config (Real 1 and Real 2): same quality, 11% faster. Fail.
Feature (Real 3 and Made-up 1): neither model got Real 3 accepted, and Sol covered fewer requirements across the six feature runs, 75% against 82%. Fail.
Migration (Made-up 2): same quality, 33% faster. Pass.
Debugging (Made-up 3): same quality, 27% faster. Pass.
Sol at high effort: passed on the parser, 83 seconds against 234, but not on debugging, 46 seconds against 47.

The two passes came from made-up jobs where runs took between 25 seconds and two minutes, with three runs per model. That is a small sample on short work, so I treat those passes as provisional.

Two smaller things worth knowing. All three Opus runs of the migration needed one extra round to fix whitespace, and no Sol run did. And all nine debugging runs, Opus and Sol alike, reported the vendor bug instead of editing vendor code, which is exactly what I asked for.

Why didn’t faster tokens mean faster work?

Because the model was not most of the clock.

On the two real config jobs, Sol’s median model time was 50 seconds. Opus took 120. Less than half the model time. But the median config run took 11 to 13 minutes in total. The rest was the agent running commands, builds and tests, and no model makes those faster.

The 11% compares the median of all six config runs per model, 671.9 seconds against 754.9. It is not an average of the two job medians. Inside that, the jobs split. Real 2 alone was about 37% faster. Real 1 was close to even, 11.1 minutes against 11.5.

That split is the honest answer. Ultrafast helps when the model is the bottleneck. On my real work, it usually is not.

Where does Sol Ultrafast already work for me?

I didn’t wait for this test to use it. I run my agents through oh-my-opencode-slim, which splits the work into roles. The night Ultrafast rolled out, I switched three of those roles to Sol Ultrafast because of how fast it answers:

Explorer, at low effort. It searches the codebase and hands back a short map of what exists and where.
Librarian, at medium effort. It looks up current library docs and outside sources.
Observer, at medium effort. It reads screenshots, PDFs and diagrams and reports what it sees.

I also moved the step that summarizes a long session, so the work can keep going in a fresh context, to Sol Ultrafast.

These jobs fit what the test found. They spend most of their time waiting on the model, not on builds and tests, so a faster model shows up as a faster answer. And what they return feeds the next step. It is not finished work that I merge.

The roles that write code stay on Opus 5.5: the fixer, which makes code changes, and the designer, which builds interface work. That is the line this test drew. Sol Ultrafast for the helpers that read and report. Opus for the agents whose work ships.

I did not run the 42-run test on the helpers. Their switch rests on how much faster they answer, not on hidden checks.

What happened on the big feature?

Real 3 is the closest thing to my day-to-day work in this test. It is a Rust and TypeScript change with a written spec and 19 requirements, the kind of change I would hand to an agent and walk away from.

Neither model got it accepted. Not once in six runs.

Coverage per run: Opus 60%, 70% and 60%. Sol 60%, 60% and 30%.
The same miss, six times: the spec said, in plain words, how to reconcile two agent records that share a timestamp. All six runs skipped it.
A security hole: an agent name containing hostile text broke the HTML escaping on a dashboard and ran injected script in 5 of 6 runs. One Opus run handled it.
A broken contract: one Sol run changed a public function’s signature, so part of the hidden tests would not even compile.
Every run said it was done. All six.

That last line is the one I keep coming back to. If I had trusted the summaries, I would have merged broken work, and in five of the six cases a script-injection hole with it.

The bar was reachable. The version I shipped earlier passes the same hidden checks in 5 of 5 runs.

Sol did finish sooner, 7.2 minutes against 13.0. It got to work that did not pass faster.

Chart of the six big-feature runs showing requirement coverage of 60, 70 and 60 percent for Opus and 60, 60 and 30 percent for Sol, with markers showing all six missed the same requirement, five of six had the escaping hole, one Sol run broke a public function, and all six reported done.

Source: hidden requirement checks on the six big-feature runs, October 9, 2026.

Can AI review catch what the agents missed?

I tried two ways. Neither worked well enough to trust.

Two AI judges

Two AI judges compared runs in pairs, 24 pairs, each shown in both orders. They picked the same winner on only 8 of the 24 pairs. Just swapping which run they saw first flipped 9 of 48 verdicts.

So I did not let them decide anything. The hidden checks decided.

A completion checker

This one I really wanted to work. It is an AI agent that reads the spec and the change, runs the tests, and lists what is missing. I tested it at medium and high effort on all 42 runs, 84 audits in total.

It caught the main big-feature gap in 0 of 6 runs at both efforts. It saw the agent’s own tests passing and marked the requirement met.
It caught the dashboard escaping hole in 1 of 5 runs at medium and 4 of 5 at high.
It raised false alarms on 23 of 36 runs that had passed everything, at each effort.

I set its bar before testing it too: catch the main gap in at least 5 of 6 runs, with no more than 4 false alarms out of 36. It missed both. I did not ship it.

Chart of what AI review caught: the completion checker found the main big-feature gap in 0 of 6 runs at medium and high effort, found the escaping hole in 1 of 5 at medium and 4 of 5 at high, raised false alarms on 23 of 36 clean runs at each effort, and two AI judges agreed on 8 of 24 pairs.

Source: 84 completion-checker audits and 96 judge comparisons on the same 42 runs.

What did I decide?

Opus 5.5 at medium effort stays my default coding agent. Nothing in my setup changes: the helpers that read and report stay on Sol Ultrafast, and the agents that write code stay on Opus.

The completion checker does not ship.

Hidden, requirement-level checks are the review that actually worked. They caught what both models and both kinds of AI review missed.

What would change my mind: a larger set of real feature jobs where Sol clears the same rule, or a setup where tools and tests stop dominating the clock, so model speed becomes a bigger share of total time.

For how my agents are set up day to day, see How I run AI agents.

What this does not claim

This is not a ranking of the best coding model. It is two models, six jobs, and my kind of work.
Three runs per job is a small sample. The made-up jobs also ran much shorter than I planned.
There was only one substantial real feature, and neither model passed it.
The hidden tests were not locked away at the system level. I audited every run instead.
I did not compare costs. Price was not part of the decision.
Models and modes change quickly. Ultrafast for Sol was about a day old when I tested it. This is how both models behaved on October 9, 2026.

Try this yourself

Pick three changes you have already merged. Replay each one from the commit before it.
Write hidden checks for every requirement before you run anything.
Write your switching rule down first, including the speed number you need.
Time each run until the work is accepted, not tokens per second. Count failures as never accepted.
Run every job at least three times per model.
Never take “done” as proof. Check it against tests the agent did not write.

So, is a faster AI coding model better?

Faster at producing text, yes. Faster at producing work I would merge, only a little, on my real jobs. And on the hardest job, speed just got me to the wrong answer sooner.

I will rerun this when the next big model lands. What would you put in a test like this?

Sources

Anthropic, Introducing Claude Opus 5.5, September 22, 2026.
OpenAI, Introducing GPT-6.1 Sol.
OpenAI API docs, Ultrafast mode.
OpenAI Developer Community, Ultrafast is rolling out today for GPT-6.1 Sol in the API, Codex, and ChatGPT Work, October 8, 2026.
Runtimewire, OpenAI adds GPT-6.1 Sol Ultrafast at six times standard API prices, published October 8, 2026, 2:08 p.m. CT.
My earlier usage study, Which AI Model Is Best for Coding? What My Own Usage Says.
My own 42-run test on October 9, 2026, scored with hidden requirement checks. Raw runs stay private.