-
How are you finding the quality of the numbers here?
My experience so far is that if you give LLMs access to raw data and ask them to produce analysis, they quickly begin to hallucinate nonsense as soon as you move beyond reporting the basic figures – they don’t math well. We pay for 3rd party tools that have added AI analysis and it quickly devolves into rubbish when you ask it for anything beyond the absolute basics.
Because of this, I built my own system that pulls and processes raw performance data to produce a whole suite of derived stats and metrics before sending it all to the LLM for analysis. So far, the results of this have been pretty bullet-proof, but I still review them and make optimisation decisions personally.
So, I guess my question is: what guardrails do you have to ensure that the AI isn’t basing optimisation decisions on hallucinated data, especially when asking it to make decisions based on information from multiple platforms?
