Autonomous cost/quality engineer · production LLM systems
Metergraph shadow-tests your live production traffic against cheaper models and configurations, on your own keys, never in your request path. When quality provably holds, it opens a pull request with the evidence attached. It's the engineer who finally gets to it.
Why this exists
Model choice gets decided once per call site, at write time, with whatever model was best that week. Nobody revisits it, and the model market re-prices monthly. Three pains showed up in every discovery conversation we had:
Ground truth decays as your domain shifts. Every prompt or model change is whack-a-mole across a high-dimensional space, and nobody has a continuous pipeline keeping the eval set honest.
“Pull one lever, improve five characteristics, degrade fifteen others.”
A 30% batch-API savings sits deferred for months behind revenue P0s. Everyone knows the work is worth doing, and it still never gets done, because it's a project and projects lose to roadmaps.
“Free money. It still hasn't made the list.”
A provider feature-flag interaction starts emitting billable 400s, your harness retry-loops on them, and diagnosis takes hours. The want, verbatim: Sentry for your AI traces.
“Burning a thousand bucks an hour until you fix something.”
verbatim from customer discovery: teams spending $10k to $200k+/mo on LLMs
How it works
A one-line SDK (TypeScript & Python) or native OpenTelemetry ingestion logs every call asynchronously: cost, tokens, latency, tool usage. It fingerprints traffic into routes, the recurring call sites that become the unit of everything downstream.
Week-one value: per-route anomaly detection. Spend spikes, retry loops, silent-failure signatures, cost-per-unit drift. Each alert arrives with a diagnosis attached instead of a red line on a chart.
Metergraph replays sampled production inputs against challenger models and configurations on your own provider accounts, inside your allow-lists, residency rules, and unit-cost ceilings. It never sits in your request path.
Per-route scorecards: quality parity with confidence intervals, per-characteristic non-inferiority testing, cost and latency deltas. Everything is denominated in your unit economics (cost per answer, per resolution, per report), not percentages.
One click from scorecard to pull request, to session-sticky canary, to promotion or rollback, with real-outcome comparison while the canary runs. Eval sets get committed to your repo as code. The dashboard is the receipt, not the product.
judge reliability on this route: 0.87 agreement with your labels · bias-probed · sampling stopped at CI convergence
What it finds
Every lever below came out of discovery: work teams already know is worth doing and defer for months anyway. The same shadow machinery tests all of it.
Same route, cheaper model, proven parity. Every company we interviewed had done the manual version, and none of them enjoyed it.
Route the hard slices to a stronger model within budget. For teams who want 110% of the performance for 100% of the cost.
~30% on eligible routes. The “free money” nobody has two spare weeks for. Flagged per route, SDK support included.
Perfect the prompt on a frontier model, compile an example-rich version for a cheap one. 80 to 90% of the quality for a fraction of the cost.
Cheap model first, static validation, escalate on failure. Shadow evals measure exactly what share the small model handles at parity.
Reasoning budgets, retries, fallbacks. Half of one team's migration savings came from parameters, not the swap.
Cache-control tuning and KV-cache pricing arbitrage across providers. Invisible in dashboards, measurable from capture.
When web search flips to full-token pricing and becomes your biggest per-answer cost, you hear about it the same day.
The trust posture
SDK, not gateway. Capture is asynchronous, so an outage on our side can't touch your traffic. Canary and promotion ride a dynamic-config flag with feature-flag semantics.
Shadow replay runs on your provider accounts and respects your allow-lists, data residency, and unit-cost ceilings. A challenger not on your allow-list is a contract violation, not a test.
We test every LLM judge per route against your own labels and implicit feedback, then run automated bias probes. Scorecards show the judge's measured reliability next to its verdicts. Where no judge qualifies, we say so.
Bradley-Terry aggregation, bootstrap confidence intervals, sequential testing, per-characteristic non-inferiority. A recommendation surfaces only when the interval clears your parity threshold, and it still has to survive a real canary before promotion.
Eval sets are code, committed to your repo. Routes, traces, and scorecards are agent-queryable via MCP. Export everything. Coexists with the tooling you already built.
FAQ
No, by design. The SDK captures asynchronously, and native OpenTelemetry GenAI ingestion works if you're already instrumented. Enforcement (canary, promotion, rollback) rides a dynamic-config flag your SDK polls. It behaves like a feature flag; nothing gets proxied.
Yours. Replay runs on your own provider accounts, inside your contractual constraints. The eval tokens we spend on judging and scoring are our cost of goods: we spend eval tokens to find you a multiple in savings.
So has almost everyone we interviewed: logging tables, per-feature P&L, the works. The audit clears “we already have that” because of the scorecards. Those are shadow-tested projections with confidence intervals on your real traffic, not another spend chart.
BYO keys by default, per-route replay opt-outs, zero-retention modes, and residency-aware sampling, so EU data only touches EU endpoints, including for evals. If you need a BAA or an in-VPC deployment, tell us on the waitlist form; early partners are shaping that roadmap now.
A platform fee tiered on traced volume, with a shadow-eval budget included in each tier. Design partners get it free and white-glove while we calibrate, in exchange for feedback and a case study.
We're onboarding design partners now, in waitlist order. The spend band on the form helps us sequence. Teams paying their own token bill at $20k+/month fit the current cohort best.
The audit is the demo
Free LLM cost audit for early teams: a route-level cost breakdown plus shadow scorecards with projected savings and confidence intervals.