Quick answer
AI feature adoption is not “users clicked the AI button.” Measure the prompt → successful outcome funnel, time-to-first-value, D7/D30 retention on AI cohorts, and adoption of new models behind feature flags. Vanity metrics (sessions, messages sent) hide failure modes: slow inference, bad defaults, and silent abandon at onboarding step 3. Wire PostHog-class analytics into your Startup OS product loop and review weekly under Guide → Monday product rhythm.
Key takeaways
- Define success events per AI feature (export completed, ticket resolved, code merged).
- Segment cohorts by plan, model version, and prompt template.
- Pair analytics with Fider/feedback votes to prioritize roadmap.
- Retention dips often trace to one onboarding step—not “users don’t like AI.”
- Feature flags (Unleash) let you measure adoption during gradual rollout.
Who this is for
- Product leads shipping copilot, agent, or generative features in B2B SaaS.
- Growth teams tired of reporting “AI messages per user” to the board.
- Engineers pairing analytics events with billing meters—same schema, two consumers.
Who should skip
- Pre-launch with zero users—define success events in spec, instrument after first cohort.
- Internal-only AI tools with <5 users—qualitative feedback may suffice short term.
- Readers needing rollout mechanics only—see gradual rollout guide first.
Metrics that matter
| Metric | What it tells you | Anti-pattern |
|---|---|---|
| Activation rate | First successful AI outcome within 24h | Counting “opened AI panel” |
| Prompt completion rate | UX friction before model runs | Ignoring drop at file upload step |
| Outcome success rate | Model + product quality | Blaming model when UX blocks submit |
| Latency p95 | Perceived “broken” feature | Only monitoring provider status page |
| D7 retention (AI users) | Habit vs novelty | Blending AI and non-AI cohorts |
| Flag exposure → conversion | Safe rollout of new models | 100% flip without cohort compare |
| Error rate per feature | Silent abandon before support ticket | Only GlitchTip without product funnel |
Vanity metrics to demote
- Total prompts without success definition.
- “AI messages per user” without error rate.
- Demo-only usage from internal accounts.
- Top-line traffic to /pricing without trial activation.
- Model benchmark scores unrelated to your task success event.
Implementation steps
- Instrument
ai_feature_started,ai_feature_succeeded,ai_feature_failedwith error codes andmodel_id. - Build funnel in PostHog from signup → first success; segment by plan and onboarding path.
- Alert on cohort retention deltas—CorpIM demo alert
PH-RET-091. - Link top feedback items (Fider) to Plane epics with ICE scores.
- Review in Guide weekly top-3 when retention drops.
- Join usage trends with billing meters so high usage without retention flags margin risk.
When adoption is low — decision tree
| Signal | Likely cause | Next action |
|---|---|---|
| High start, low completion | UX friction | Session replay; fix step 2–3 |
| High completion, low success | Model/prompt quality | Eval set; prompt iteration |
| High success, low D7 | No habit loop | Notifications, templates, workflow embed |
| High latency p95 | Infra/model size | See observability stack |
| Errors spike post-deploy | Bad rollout | Rollback flag; RCA playbook |
Run the Retention dip response playbook: isolate cohort → gather tickets/feedback → hypothesis → experiment → ship → watch 7-day cohort. Do not jump to “bigger model” without funnel evidence.
CorpIM demo
Studio → Product loop: north-star cards, roadmap, feedback votes, retention todo. Open CorpIM Studio
FAQ
What counts as a “successful outcome”?
One verifiable product event: file exported, PR opened, ticket marked resolved, report saved—not “model returned text.” Define per feature with product and engineering jointly.
Should we track token usage in PostHog?
Track tokens for cost debugging if needed, but billable units should mirror Lago meters. Avoid two definitions of “usage.”
How often should AI cohorts be reviewed?
Weekly on Monday for retention; daily automated alerts on >10% funnel drop. Align with weekly operating rhythm.
Do public leaderboards replace product analytics?
No. MMLU and SWE-bench scores do not tell you if your onboarding funnel converts. Use leaderboards for model selection; use funnels for product adoption.
Continue the semantic path
Gradual rollout for models and prompts · Weekly operating rhythm · AI-native Startup OS