Have you analyzed using search console and PageSpeed Insights? I find it’s much easier to narrow down my targets if I overlay multiple data sources, and typically the triple threat from GA, GSC, and PageSpeed Insights will reveal opportunities for improvement.

If you’re familiar with caching, your own visits to the site will not be the same as a new user. You have assets downloaded locally, allowing your repeat site visits to afford you a better and faster experience. I do not advise testing site speed on your own device or browser, even in incognito. You will get more objective / better data from tools, but if you have to audit manually, try using a proxy or something similar to force the site to behave as if you are a new user each time.