Is anyone tracking how chatgpt answers change over time?

[One-Job2733]

I’ve noticed this too. The same prompt can give slightly different answers depending on timing, wording, or even model updates, so it’s hard to “track” it the way we track rankings in SEO. It feels more like monitoring trends than positions.

I’ve seen some people log prompts monthly and compare themes rather than exact wording. Curious if anyone has built a consistent system for this that actually works long term?

[whiteflowergirl]

I wonder if chatgpt models are pulling from different sources at different times like one week it prioritizes recent content, next week older authoritative stuff??

[Ranketta]

“I’ve been monitoring” – how were you monitoring? The method matters, some are highly unreliable, some can be labeled as accurate enough.
Everyone who talks about tracking needs to acknowledge that it is not technically tracking, it is simulating real user conversations, so the output quality relies on the ability to accurately mimic the real conversations.

Ask away, I will do my best to help 🙂

[ClinicalPrompt]

I’ve recently been studying this question too, what helped me draw conclusions was charting the mentions over a time frame.

If two users ask ChatGPT the same question, the result will most likely not be the same, that’s just how LLMs work since they’re not deterministic. However if you were to plot the responses a large set of users get for the same prompt you’d be able to see which conclusions are more probable for the LLM to respond.

To minimize this probabilistic way of thinking from the LLM on your own tracking, you can leverage context. When you enter a prompt in ChatGPT, it’ll consider past conversations and what it “thinks” of you. To simulate your client’s clients, you want to give context in your prompts as if you’re them, e.g.:

Ignore any prior conversation context.
Do not personalize the answer based on assumptions about the user.
Answer only based on general knowledge and typical market perception.

Context:
The user is:
– Role: {persona_role}
– Industry: {industry}
– Company size: {size}
– Geography: {location}
– Intent: {intent_description}
– Buying stage: {awareness/consideration/decision}

The user asks:
“{core_prompt}”

Provide a neutral, comprehensive answer reflecting what ChatGPT would most likely say to this type of user.

If you’re using the API, you can also set temperature = 0 and top_p = 1 to reduce variance.

Again, the best way is to prompt the LLM multiple times over a period of time, and plotting this data would give you a great view of how your client is doing.

[sibly]

Most tracking tools will take a screenshot. But yeah unless it’s a question that’s in the knowledge base it writes a new answer every time

[Jazzlike_Bit_2261]

How are you monitoring? the answer is unique depends on who is asking.

[sriracha_saws]

I’ve noticed more disparity when model versions update because gpt-5.1 used to give us totally different answers,
now gpt-5.2 has changed things again. It seem like every time they update the model everything shuffles

[pffffftokay]

We have no idea how chatgpt decides what to surface. We’re just guessing based on output. Much like with SEO its largely a game of experimenting and seeing what works. Gotta be careful who you trust though because just like with SEO there are a lot of grifters out there.

[Sad-Speech-932]

Been running the same test prompts on Chatgpt since October and the variance is absolutely unhinged. One of our clients went from 12% visibility to 43% back down to 18% in like 6 weeks without any major changes on our end.
We use Peec ai to track this and the charts look like a drunk person trying to draw a straight line loll

[[deleted]]

[removed]

[LegendaryAngryWalrus]

Otterly has sources tracking on answers.

[Excellent-Lead-8027]

Yeah, tracking it manually same issue here. Try logging responses weekly and comparing. OpenAI updates models silently, so it’s a moving target.

[Victorie_ralph]

There is no use in tracking AI answers, like you describe. You shouldn’t track AI the same as keywords in Google. The answers are completely different from one person to the next. There has been a study (can’t seem to find the source atm) that discoverd that you have to ask the same question at least 1400 times before you get two exactly the same answers.

[lehan112]

There are a lot of tools out there, I use trackerly . ai. When you say you’re monitoring, are you talking about typing in the prompt manually in chatgpt yourself?

Track the citations! that should give you a big clue