Artificial intelligence news.
Straight from the original announcements.

Fivetran study finds coding model price is a poor value guide

November 9, 2026, 17:02By the AI News Bites Editorial TeamSource: Fivetran
Fivetran study finds coding model price is a poor value guide
Image: Fivetran

A blinded Fivetran study of 39 engineers found that paying substantially more for an OpenAI coding model produced only a modest preference advantage in everyday software work. The company compared Luna, Terra, and Sol through Claude Code on its own codebase over two weeks.

Key points
  • Fivetran tested Luna, Terra, and Sol with 39 engineers.
  • Sol led both matchups, but the preference results were not decisive.
  • Median weekly spending ranged from $5.32 for Luna to $168.44 for Sol.
  • Pull request counts did not establish a productivity ranking.
  • Fivetran plans a follow-up study of planners and orchestrators.

A blinded test in everyday coding work

Fivetran conducted the study from September 14, 25, 2026, with engineers voluntarily assigned two of the three models, one per week and in random order. The models were presented through aliases, while a LiteLLM proxy removed model and provider names from responses and instructed the models not to reveal their identities.

Of the 39 participating engineers, 35 used at least one assigned model and 31 tested both. The models ran through Claude Code, the participants’ usual coding agent, and the company recorded model preference, spending, and pull request activity in normal repository work.

Fivetran said the design was intended to avoid results being shaped by brand familiarity or expectations about price. Unlike public benchmarks focused on isolated tasks, the test examined how the models performed in the company’s own repositories and tools.

Sol led preferences, but the result was not decisive

Among the 27 engineers who completed the final survey after using both assigned models, Sol was preferred in both of its matchups. Six of nine engineers preferred Sol to Luna, while seven of 10 preferred Sol to Terra. Terra edged Luna by five votes to three.

The differences were narrow. Fivetran reported that 12 of the 27 preferences were weak, including four of Sol’s seven wins over Terra. The company said none of the three splits was clearly more than chance, and all exact 95% confidence intervals crossed 50%.

Pull request activity also did not establish a productivity ranking. A typical engineer opened five pull requests during the Luna week, four during the Sol week, and two or three during the Terra week. Fivetran said those counts largely reflected whether engineers worked on a large refactor or several smaller changes.

Higher prices brought sharply higher spending

The token prices varied widely. Terra’s input rate was 10 times Luna’s, while Sol’s was 20 times Luna’s. The listed unchached input prices were $0.20 per million tokens for Luna, $2.00 for Terra, and $4.00 for Sol. Output prices were $1.20, $12.00, and $20.00 respectively.

Typical weekly spending followed the same pattern. The median engineer spent $5.32 during the Luna week, $41.42 during the Terra week, and $168.44 during the Sol week. Average spending was higher, at $9.95 for Luna, $88.85 for Terra, and $243.49 for Sol, because some engineers used the models much more heavily.

Fivetran concluded that price tier was a poor proxy for day-to-day value. According to the company, the 20-fold token price increase from Luna to Sol delivered a modest preference edge rather than a decisive win.

Task fit affected the user experience

Fivetran said heavier models could be useful for large refactors, where thorough planning paid off. For focused changes, however, engineers placed more value on fast responses and tight scope control, and heavier models sometimes produced extra steps, broader edits, or unnecessary cost.

In the study setup, each model handled the full agent loop, including planning, background calls, and subagent work, without routing routine tasks to a faster model. Fivetran said this increased cost and end-to-end execution time when an agent took more steps before returning control to an engineer. The company did not measure potential savings from a mixed-model workflow.

Fivetran plans a follow-up blinded comparison of planners and orchestrators in a non-proprietary harness. It says the next study will hold execution models, tools, and budgets constant across predefined task categories while measuring task completion, reviewed code quality, human effort, latency, and cost. For users, the findings support choosing models according to the task rather than assuming the most expensive option will provide the best coding experience.

Related news

Research
ResearchNovember 15, 2026, 23:00

OpenAI outlines lattice construction for the full Standard Model

OpenAI has proposed a candidate nonperturbative definition of the Standard Model on a spacetime lattice, with GPT-6 Astra helping develop the construction and detailed…

JetBrains asks 12 AI models for 1,200 programming hot takes
ResearchNovember 15, 2026, 18:02

JetBrains asks 12 AI models for 1,200 programming hot takes

Twelve AI models overwhelmingly attacked Python, Rust and TypeScript in a JetBrains experiment that collected 1,200 opinions about programming languages. The models also…

Fivetran Puts Data Pipeline Work Into an AI-Native Terminal Loop
For developersNovember 11, 2026, 21:06

Fivetran Puts Data Pipeline Work Into an AI-Native Terminal Loop

Fivetran is positioning its command-line interface as the data movement layer in AI-assisted pipeline workflows, linking connections, syncs and custom connectors with…