4 September 2026 · Benchmark · Vasiliy Radostev

The workhorse tier: ten fast, low-cost models measured on production workloads

Most model comparisons focus on the frontier. This one does not. It looks at the fast, inexpensive models that carry agentic automation and high-volume routine work, where the deciding factors are usually cost and latency rather than peak capability.

The capability floor has risen steadily. Models that now sit at the cheap end of the market would have counted as frontier systems a few generations ago, and for a large class of production work they are sufficient. The open question is no longer whether they are good enough, but which of them to choose.

We measured ten such models against three production workloads, scoring quality, latency and cost, then deriving the two ratios that tend to govern a deployment decision: quality per unit of cost, and quality per unit of latency. The intended reader is someone selecting a model for work that requires an LLM but does not require the most capable one available.

The workloads

Three workloads drawn from live traffic in production applications. All three are structured-output tasks of the kind that fills an agentic pipeline, rather than open-ended conversation or reasoning.

1. Recommendation normalization. Convert a product recommendation into a strict JSON record with a canonical brand name, a rank, a justification and a conditional URL. The difficulty lies in schema discipline and house style, not in generation.
2. Event extraction. Read a scraped listing page and return the events it contains as a JSON array, filtered by date, returning an empty array where the page contains none. Long and noisy input, short and precise output.
3. Company summarization. Produce a short descriptive summary of a company from enrichment data, covering market, customers and positioning.

Method

Each workload was frozen as a fixed set of 20 production traces. All ten candidates replayed the same six cases twice, producing 12 scored samples per model per workload and 120 per workload, on identical inputs. Responses were graded against a rubric derived from the workload's own production behaviour by an independent judge model (Claude Sonnet 5), which was not itself a candidate.

Cost and latency are expressed relative to the other models on the same workload, so 0.40x means the model cost 40 percent of what a typical model in this field cost for the same work. The reference point is the geometric mean of all ten candidates on that workload, recomputed per workload and then aggregated, again geometrically. No incumbent or baseline model enters the comparison: every model here is scored against its peers. Quality is an absolute score between 0 and 1 and is averaged arithmetically.

A graded example

The rubric for a workload is a list of yes/no questions derived from that workload's own instructions. Event extraction had thirteen. Three of them, verbatim:

Is the response solely a JSON array (no preamble text, no explanatory sentences, and no markdown code fences)?

Does every event included in the response have a start date on or after the "Today" date stated in the input, with no past events included?

If, after applying the future-date and public-event filters, the page contains no qualifying upcoming events, does the response return an empty array?

Each question is marked major or minor and is skipped where it does not apply. A major issue scores that sample zero, a minor issue scores it 0.5, and a clean response scores 1.0.

One case in that workload was a page listing a music festival line-up. Every performance on it had taken place about six weeks before the run date given in the prompt, so the correct output was an empty array.

Seven of the ten models returned []. Three, Grok 4.1 Fast, Nemotron 3 Super and Claude 3 Haiku, returned all fourteen events on the page. Their output was otherwise careful: valid JSON, correct titles, correct time-zone offsets on every timestamp. Each failed one check, the one on past events, and each scored zero for the sample.

That is the shape of most failures recorded here. They are rarely a matter of writing quality. A model formats the response correctly and misses one instruction, and for the pipeline consuming that response the output is unusable.

A response that does not conform to the workload's required output schema is scored zero. That is the right treatment for a deployment decision, since an unparseable response is of no use whatever its content, but it means a zero in this study can indicate either a wrong answer or an unusable format. Where the distinction matters below, we say which occurred.

Results

Model	Cost	Quality	Latency	Quality / cost	Quality / latency
GPT-5.6 Luna	0.81x	0.944	0.62x	1.16	1.52
Grok 4.1 Fast Reasoning	1.65x	0.903	1.43x	0.55	0.63
DeepSeek V4 Flash	0.59x	0.865	2.76x	1.47	0.31
MiniMax M3	1.29x	0.722	0.96x	0.56	0.75
Kimi K2.5	2.22x	0.708	0.94x	0.32	0.75
Gemma 4 31B	0.40x	0.622	1.19x	1.55	0.52
Gemma 4 26B A4B	0.46x	0.611	0.62x	1.32	0.99
Nemotron 3 Super	1.03x	0.583	0.92x	0.57	0.63
Grok 4.1 Fast	1.20x	0.569	0.55x	0.48	1.04
Claude 3 Haiku	1.95x	0.486	1.21x	0.25	0.40

Cost and latency are given as multiples of the field average, meaning the average across all ten models on the same workload. A model at 0.40x cost ran at 40 percent of what the typical model in this table charged for the same requests; one at 2.76x latency took nearly three times as long as the typical model. There is no incumbent or baseline here: the models are measured against each other. All ten were measured on all three workloads.

Findings

Highest quality: GPT-5.6 Luna, at 0.944. It was joint second-fastest at 0.62x, behind Grok 4.1 Fast at 0.55x, and cost below the field average at 0.81x. It is the clearest choice here, and the only model that beats the field average on quality, cost and latency at once.

Cheapest: Gemma 4 31B, at 0.40x the field average. Mid-field quality of 0.622 leaves it level with DeepSeek V4 Flash on quality per unit of cost, 1.55 against 1.47, a gap twelve samples per workload cannot resolve.

Gemma's 0.622 averages three workloads, one of which it could not perform at all. On recommendation normalization both Gemma variants failed all twelve cases because their output did not conform to the required response schema. Generation succeeded every time; the responses could not be parsed. On the two workloads it completed, Gemma scored 0.917 and 0.950, which is competitive. Its cost efficiency applies to those two, not to the field as a whole.

Gemma is therefore the cheapest option where it works, and DeepSeek the safer one: better quality at 0.865, all three workloads completed, about 50 percent more expensive.

Also of note

DeepSeek V4 Flash, at 0.865 quality and 0.59x cost. The best quality per unit of cost of any model that completed all three workloads, at 1.47. Undercut by latency: 2.76x the field average, the slowest model measured. That is immaterial for batch and asynchronous work and disqualifying for anything interactive.

Gemma 4 26B A4B, at 0.611 quality, 0.46x cost and 0.62x latency. Second-cheapest, joint second-fastest, and third in cost efficiency at 1.32. It carries the same schema-conformance failure as the 31B on recommendation normalization, so the same qualification applies.

The two Grok 4.1 Fast variants. These share a token price, and the aggregate scores diverge sharply: 0.903 for the reasoning variant against 0.569 for non-reasoning. The per-workload figures show the gap is not general. On company summarization the two tied exactly at 0.833; on recommendation normalization reasoning led modestly, 0.875 to 0.792. The aggregate difference comes almost entirely from event extraction, where the non-reasoning variant scored 0.083, close to total failure, while the reasoning variant scored 1.000. The reasoning variant is the safer choice because it avoids that failure, not because it is broadly better.

Cost behaves differently from what a shared token price implies. Reasoning emits more tokens, so per request it cost 3.25x more on recommendation normalization and 1.39x more on company summarization. On event extraction it cost 0.58x, less than non-reasoning, because the failing variant produced long unusable output.

Claude 3 Haiku placed last at 0.486 quality, and was the second most expensive model in the field at 1.95x. An older small model is not necessarily a cheap one, and on structured work of this kind it is well behind the current generation.

Obtaining these models

All ten were called through Vercel AI Gateway, which is why ten models from eight labs could be run against a single credential and billing surface. Aggregators such as Vercel, Fireworks, OpenRouter and Bedrock are the practical route for a comparison of this kind, and increasingly for production, since they allow a team to move between models without a new contract or a new integration.

Going directly to a lab (OpenAI, Anthropic, Google, DeepSeek) typically offers earlier access to new releases, occasionally better rates at volume, and a direct support relationship, at the cost of one integration per provider. Aggregator and direct pricing are not always identical, and we have observed the same model priced differently between a lab's own API and a gateway in both directions.

Self-hosting

Several of these models are open-weight, including DeepSeek V4 Flash and both Gemma variants, so renting GPUs and serving them directly is a viable alternative. Whether it is economic reduces to weekly request volume.

The measured absolute cost per request on the gateway, taken from the runs above:

On these workloads a low-cost hosted model runs between $0.0001 and $0.0005 per request. Gemma 4 31B averaged $0.000255 across the three. The break-even against self-hosting is:

requests per week to break even  =  weekly cost of the GPU instance
                                   ----------------------------------
                                   hosted cost per request

A 31B model fits comfortably on a single 48GB card. As of early September 2026 an L40S-class instance rents between $0.48 and $2.20 per hour depending on provider tier, with a median near $1.54, and an A100 80GB starts around $1.09. Dedicated GPU clouds run at roughly half hyperscaler rates. Running one card continuously:

So the threshold sits somewhere between roughly 300,000 and 2 million requests per week, depending on the card and the deal, before self-hosting wins on infrastructure alone. That figure ignores engineering time, idle capacity outside peak hours, autoscaling and on-call, all of which push the real threshold higher.

Self-hosting these models is usually a decision about data residency, latency control or vendor independence. It becomes a cost decision only at substantial volume.

Rates checked against provider listings on 4 September 2026 and against the Vercel AI Gateway model catalogue the same day; GPU rates in particular move quickly.

Limitations

• Twelve samples per model per workload. Sufficient to separate 0.9 from 0.5, not to separate 0.62 from 0.61. Small differences should be treated as noise.
• Three workloads, all structured output. The results say nothing about prose generation, conversation or tool use.
• Cost and latency are relative to the field on each workload, not absolute. A model at 0.40x is cheap compared to these nine peers, not cheap in general.
• A single judge model. A different judge would shift the scores, though we would expect the ordering to be broadly stable.
• Four of the ten models earned at least one zero for schema conformance rather than for answer quality: both Gemma variants (12 of 12 cases), Nemotron 3 Super (4) and Kimi K2.5 (3), all on recommendation normalization. Their quality averages understate them as answer generators and correctly state them as drop-in components for that workload.
• Measurement depth is not uniform. Eight models are scored on the screening sample throughout. DeepSeek V4 Flash on event extraction and Gemma 4 31B on company summarization cleared the cost threshold, advanced to a deeper evaluation, and are reported on that larger sample for those workloads. Both are flattered slightly as a result; on screening depth alone Gemma 4 31B scores 0.583 rather than 0.622, which would place it below Gemma 4 26B A4B.

Questions or suggestions: vasiliy@psl.com