Engineering 8 min read· 25 Aug 2026· By Engineering
Observability for LLM apps — what to log, what to alert on
The five metrics every production LLM feature should track.
An LLM feature without observability is a random cost generator. Instrument these five things from day one.
- Latency — first-token and total, p50/p95/p99
- Cost — per request, aggregated per endpoint
- Token counts — input and output
- Error rate — by class (auth, rate, upstream, parse)
- Eval score — sampled against a golden dataset
TokenBaazar's dashboard covers 1-4 out of the box. You have to build 5 yourself — start with 20 hand-graded examples.