When that happens your site is likely throttled. Check your link profile and ensure that you are following the rules. They will ding you and never tell you, that’s part of their punishment. Yes, there are manual actions but those are the ones they want you to fix. The algorithm penalties go unnoticed sometimes for years.