AI Agent Operations

What people ask Nova,
and which answers actually helped

Ask Nova Ledger is the analytics page for the chat assistant. Top questions, top topics, model mix, cost, and feedback signals (thumbs up/down, follow-up clicks). Use it to find documentation gaps (questions Nova answers poorly), feature gaps (questions Nova cannot answer), and product opportunities (recurring questions that point at unmet needs).

Get Started Talk to Sales
app.novaaiops.com / ask-nova-ledger
● LIVE
Top
questions tracked
Helpful %
from feedback
Per-doc
gap detection
Weekly
topic refresh
Top Questions

What users are actually asking

The page reports the top 50 questions of the week with frequency, helpful percentage, and a sample of paraphrased instances. Recurring questions point at three things: documentation gaps (the answer exists but is hard to find), feature gaps (the answer is "we do not support that yet"), and product opportunities (the question is a feature request in disguise).

  • Top 50 weekly: ranked by frequency; deduplicated by paraphrasing similar wordings into one row
  • Helpful percent: each question shows what percent of askers gave a thumbs up to the answer
  • Three categories: docs gap (rewrite docs), feature gap (build feature), product op (consider new feature)
app.novaaiops.com / ask-nova-ledger · top
Cost & Model Mix

Where Ask Nova's budget goes

Ask Nova's cost is broken down by model (Haiku, Sonnet, Opus), by topic, and by tenant. The data feeds into Cost Circuit Breaker so noisy tenants can be capped. The model mix tells you where you can move tasks to cheaper models without quality loss.

  • By model: Haiku for short answers, Sonnet for medium, Opus for "explain this incident"
  • By topic: topic-by-topic costs surface the expensive areas of the conversation
  • By tenant: feeds Cost Circuit Breaker so a runaway tenant can be capped automatically
app.novaaiops.com / ask-nova-ledger · cost
Doc Gap Detection

When Nova answers poorly, the docs improve

Topics with low helpful percentage are flagged for documentation review. The page shows the original question, Nova's answer, the user's thumbs-down, and (if the user provided one) the follow-up clarification. That data feeds the docs team's backlog directly. After docs improve, the helpful percentage usually climbs within a week.

  • Helpful percent threshold: topics under 60% helpful flagged automatically for docs review
  • Captured artifacts: question + Nova answer + user feedback bundled for the docs team to triage
  • Improvement loop: after docs ship, the page tracks helpful percent for the same topic for two weeks
app.novaaiops.com / ask-nova-ledger · gap
Privacy

Question text scrubbed for PII before storage

Ask Nova's questions go through Prompt Egress Scanner before they reach the LLM, and the scrubbed version is what lands in the ledger. Ledger search and analytics work on the scrubbed corpus. PII never sits at rest. The privacy story is the same as the rest of the platform.

  • Scrubbed before storage: PII redacted at write time; ledger never holds raw user text
  • Searchable scrubbed corpus: analytics work on the scrubbed text; placeholders preserved
  • Tenant-scoped: one tenant's questions never appear in another tenant's analytics
app.novaaiops.com / ask-nova-ledger · privacy
Video walkthrough coming soon

Subscribe to Nova AI Ops on YouTube for demos, tutorials, and feature deep-dives.

Your users are telling you what they need

Ask Nova's logs are the cheapest user research you have. The ledger turns them into a list of fixable items.

Get Started Request a Demo