Quick answer
LLM application observability needs three planes: errors (GlitchTip/Sentry-compatible), metrics and traces (SigNoz, Grafana stack), and product outcomes (PostHog funnels). Model logs alone miss queue backlog and OAuth bugs that customers feel first. Wire alerts to IM and runbooks in Outline. CorpIM Engineering loop demos GlitchTip + SigNoz + security CI as one incident narrative.
Key takeaways
- Track model timeout as application error, not only vendor status page.
- Worker queue depth predicts async AI job failures before user reports.
- Correlate deploy tags with error fingerprints.
- Status page (Upptime) for customer-visible API latency.
- Open-source stack lowers data residency friction for EU buyers.
Who this is for
- Engineering teams running async LLM workers, agents with tools, or RAG pipelines.
- Startups comparing self-host observability vs Datadog/New Relic lock-in.
- Teams preparing for enterprise security reviews needing trace retention policies.
Who should skip
- Batch offline eval jobs with no production SLA—batch logs in object storage suffice.
- Teams already on a managed APM with LLM plugins and happy margin—migrate only if cost or residency forces it.
- Readers needing incident runbooks only—see production RCA.
Stack map
| Layer | Open tool class | LLM-specific signals |
|---|---|---|
| Errors | GlitchTip | 500s, context overflow, tool exceptions |
| Metrics/traces | SigNoz | p95 latency, queue depth, GPU worker health |
| Logs | Loki (optional) | Prompt IDs, tenant IDs (redact content) |
| Product | PostHog | Success funnel, flag exposure |
| Uptime | Upptime | Public API SLO |
| Security CI | Trivy, Semgrep | Container CVEs on inference workers |
Compare with managed APM using self-host TCO. For agent traces specifically, see AI agent observability.
What to log (and what not to)
| Log | Why | Do not log |
|---|---|---|
| request_id, tenant_id, model_id | Correlate errors across services | Full customer prompts (unless policy allows) |
| token counts, latency ms | Cost and SLA debugging | PII from retrieved chunks |
| tool call name + status | Agent RCA | Raw API keys in error payloads |
| flag variant, prompt_version | Rollout regression | Complete model outputs in prod logs |
Retention policies matter for SOC 2 privacy controls—define before auditors ask.
Alerting rules (starter)
- Error rate > 2× baseline for 10 minutes → incident channel.
- p95 API > SLA threshold → status page draft.
- Queue depth critical → scale workers per Outline runbook.
- Failed payment webhook → RevOps todo (not engineering-only).
- Flag exposure + error spike → pause rollout.
Three-plane incident story
When OAuth 500s spike:
- Errors — GlitchTip fingerprint ERR-8821 groups route.
- Traces — SigNoz shows DB timeout vs LLM timeout.
- Product — PostHog shows login funnel drop on same cohort.
Without three planes, engineering fixes the wrong layer. CorpIM Engineering loop ties these IDs in one IM narrative for Startup Copilot prompts.
Incident → compliance
Postmortems stored as Probo evidence for change and availability controls—links SOC 2 checklist for AI SaaS.
CorpIM demo
Studio → Engineering: ERR-8821, OBS-441, SEC-TVY-12. Messages → Errors & security channel.
FAQ
Do we need a separate “LLM observability” product?
Often no. Standard errors + traces + product funnels cover most SaaS LLM failures. Dedicated LLM trace tools help when you debug prompt chains at scale—add when single-agent traces in SigNoz are insufficient.
GlitchTip vs Sentry?
GlitchTip is Sentry-protocol compatible with open-source hosting option. Pick based on residency, cost, and whether you need Sentry’s commercial integrations.
How does observability tie to adoption metrics?
Latency and silent errors kill D7 retention before GlitchTip shows a crash. Watch funnel + errors together.
Self-host SigNoz or use Grafana Cloud?
Self-host when EU data residency or long trace retention is non-negotiable. Cloud when team has no ops capacity—see TCO calculator.