Now live · autonomous cost/quality engineer
It starts free and open source. One line of code tracks every LLM call by function, content-blind, in a dashboard you self-host. When you want more, the hosted engineer takes over: it shadow-tests your live traffic against cheaper models on your own keys, and opens a pull request once quality provably holds.
Why this exists
Model choice gets decided once per call site, at write time, with whatever model was best that week. Nobody revisits it, and the model market re-prices monthly. Three pains showed up in every discovery conversation we had:
Ground truth decays as your domain shifts. Every prompt or model change is whack-a-mole across a high-dimensional space, and nobody has a continuous pipeline keeping the eval set honest.
“Pull one lever, improve five characteristics, degrade fifteen others.”
A 30% batch-API savings sits deferred for months behind revenue P0s. Everyone knows the work is worth doing, and it still never gets done, because it's a project and projects lose to roadmaps.
“Free money. It still hasn't made the list.”
A provider feature-flag interaction starts emitting billable 400s, your harness retry-loops on them, and diagnosis takes hours. The want, verbatim: Sentry for your AI traces.
“Burning a thousand bucks an hour until you fix something.”
verbatim from customer discovery: teams spending $10k to $200k+/mo on LLMs
Free & open source
The foundation is open source: a zero-dependency capture SDK, plus a server and dashboard you self-host in one command. Wrap your client and every call gets attributed to the function that made it, with token counts, latency, and server-priced dollar cost. No account needed to try it.
# 1. capture: Python or TypeScript, zero deps
pip install metergraph
# npm install metergraph
# 2. wrap the client you already have
client = metergraph.wrap(OpenAI())
# 3. self-host the dashboard
docker compose up
# → http://localhost:8787The SDK captures usage metadata only, and the self-hosted server strips any content field at ingest through a column allowlist. Your prompts and completions never leave your process. That guarantee is structural, so you don't have to take a privacy policy on faith.
Python walks the stack automatically; TypeScript uses track(fn) for names that survive bundlers. route("ticket-classifier") groups calls by product surface. Every row lands as yourapp.billing:summarize_invoice.
Server-priced cost from an effective-dated, community-maintained catalog, so history reprices correctly when a model's price changes. Mount a newer file without rebuilding, or open a PR.
FastAPI + Postgres you run wherever you want. The dashboard is a React SPA served by the same server. No data leaves your network unless you point the SDK at the hosted tier.
How the hosted engine works
The same one-line SDK (TypeScript & Python) or native OpenTelemetry ingestion logs every call asynchronously: cost, tokens, latency, tool usage. It fingerprints traffic into routes, the recurring call sites that become the unit of everything downstream.
Week-one value: per-route anomaly detection. Spend spikes, retry loops, silent-failure signatures, cost-per-unit drift. Each alert arrives with a diagnosis attached instead of a red line on a chart.
Metergraph replays sampled production inputs against challenger models and configurations on your own provider accounts, inside your allow-lists, residency rules, and unit-cost ceilings. It never sits in your request path.
Per-route scorecards: quality parity with confidence intervals, per-characteristic non-inferiority testing, cost and latency deltas. Everything is denominated in your unit economics (cost per answer, per resolution, per report), not percentages.
One click from scorecard to pull request, to session-sticky canary, to promotion or rollback, with real-outcome comparison while the canary runs. Eval sets get committed to your repo as code. The dashboard is the receipt, not the product.
judge reliability on this route: 0.87 agreement with your labels · bias-probed · sampling stopped at CI convergence
What the hosted tier finds
Every lever below came out of discovery: work teams already know is worth doing and defer for months anyway. The same shadow machinery tests all of it.
Same route, cheaper model, proven parity. Every company we interviewed had done the manual version, and none of them enjoyed it.
Route the hard slices to a stronger model within budget. For teams who want 110% of the performance for 100% of the cost.
~30% on eligible routes. The “free money” nobody has two spare weeks for. Flagged per route, SDK support included.
Perfect the prompt on a frontier model, compile an example-rich version for a cheap one. 80 to 90% of the quality for a fraction of the cost.
Cheap model first, static validation, escalate on failure. Shadow evals measure exactly what share the small model handles at parity.
Reasoning budgets, retries, fallbacks. Half of one team's migration savings came from parameters, not the swap.
Cache-control tuning and KV-cache pricing arbitrage across providers. Invisible in dashboards, measurable from capture.
When web search flips to full-token pricing and becomes your biggest per-answer cost, you hear about it the same day.
Editions
Same SDK, same routes, same content-blind capture. The difference is whether Metergraph just shows you the cost, or continuously tests your traffic and ships the fix.
Free · Apache-2.0 · self-hosted
docker compose up)The autonomous cost/quality engineer
The trust posture
SDK, not gateway. Capture is asynchronous, so an outage on our side can't touch your traffic. Canary and promotion ride a dynamic-config flag with feature-flag semantics.
Shadow replay runs on your provider accounts and respects your allow-lists, data residency, and unit-cost ceilings. A challenger not on your allow-list is a contract violation, not a test.
We test every LLM judge per route against your own labels and implicit feedback, then run automated bias probes. Scorecards show the judge's measured reliability next to its verdicts. Where no judge qualifies, we say so.
Bradley-Terry aggregation, bootstrap confidence intervals, sequential testing, per-characteristic non-inferiority. A recommendation surfaces only when the interval clears your parity threshold, and it still has to survive a real canary before promotion.
Eval sets are code, committed to your repo. Routes, traces, and scorecards are agent-queryable via MCP. Export everything. Coexists with the tooling you already built.
The capture layer is Apache-2.0 and self-hostable forever. Point the same SDK at your own server or the hosted tier by changing one environment variable.
FAQ
The capture SDKs (Python & TypeScript, on PyPI and npm) and the server plus
dashboard, all Apache-2.0. Self-host with one docker compose up and you get
cost, token, and latency tracking attributed to the function and route that made each
call, priced from a community-maintained catalog. No account, no token limits, no
gated features on the tracking layer.
The evaluation-and-action layer: continuous shadow evals of your live traffic, judge-qualified scorecards, evidence-backed pull requests, canary rollouts, and anomaly diagnosis. It's live now. Self-hosted Metergraph shows you the cost; the hosted engine tests alternatives and ships the fix.
No, by design. The SDK captures asynchronously, and native OpenTelemetry GenAI ingestion works if you're already instrumented. Enforcement (canary, promotion, rollback) rides a dynamic-config flag your SDK polls. It behaves like a feature flag; nothing gets proxied.
Not unless you explicitly opt in. The SDK sends usage metadata only, and the self-hosted server strips any content field at ingest through a column allowlist, so content can't reach the database in the first place. Content capture is an explicit opt-in that only has any effect against the hosted service.
Yours. Replay runs on your own provider accounts, inside your contractual constraints. The eval tokens we spend on judging and scoring are our cost of goods: we spend eval tokens to find you a multiple in savings.
BYO keys by default, per-route replay opt-outs, zero-retention modes, and residency-aware sampling, so EU data only touches EU endpoints, including for evals. If you need a BAA or an in-VPC deployment, request that conversation from inside the dashboard.
A platform fee tiered on traced volume, with a shadow-eval budget included in each tier. Design partners get it free and white-glove while we calibrate, in exchange for feedback and a case study. The open-source layer stays free regardless.
Start self-serve
Sign up with a verified email and Metergraph creates your free, content-blind workspace. Create your ingest key in Setup—no approval or sales conversation required.
read-only SDK · can't break production · content-blind by default