-
The math you laid out is right and nobody in the vendor space wants to say it out loud because it undermines the product category they’re selling.
The non-determinism problem is structural, not a tooling problem waiting to be solved. Even if a tool ran every prompt 20 times and averaged the results, the confidence intervals would still be too wide to make meaningful week-over-week comparisons. You’d need hundreds of runs per prompt per week to get stable data, and the economics of that are completely unrealistic at current API pricing.
The honest answer for clients right now is what you’re already doing — telling them you can’t reliably measure ChatGPT/Perplexity citation rates. That’s not a gap in your service, it’s an accurate description of the current state of the space. Any tool claiming otherwise is either misrepresenting their methodology or their sample sizes.
The one slice of “AI visibility” that is actually measurable is Google AI Overviews. Same query, consistent results, trackable over time because it runs against a live index rather than a probabilistic model. That’s where i’d focus client reporting right now. Semust tracks this per keyword daily — which queries trigger an AI Overview, which domains get cited — and the data is stable enough to show real trends. Not the same as ChatGPT visibility but at least it’s a number that means something.
For the LLM side the closest thing to a useful manual workflow is picking a fixed set of 10-15 high-intent prompts, running each one 5 times per month, and tracking directional trends rather than point estimates. Acknowledge the noise explicitly in the report. It’s less impressive than a dashboard but it’s more honest.
