-
Are 410s a waste of effort?
Crawlers keep trying to crawl URLs and images that I deleted 10 years ago. I've been giving them 410 errors for years. But they just keep trying. Should I delete the 410s and let them have 404 errors instead?
-
1 Reply
-
Ah, the ol’ persistent crawler problem. Fun times. Here’s the scoop:
410 vs 404:
* 410 = “Gone forever, don’t come back”
* 404 = “Not found, maybe check later?”In theory, 410 should work better. In practice? Crawlers can be dumb as rocks.
Options:
1. Keep 410s: You’re doing it right, crawlers are being jerks.
2. Switch to 404s: Might help, might not. Worth a shot if you’re tired of 410s.
3. Robots.txt: Block those specific URLs if possible.
4. .htaccess: Redirect old stuff to your homepage.Personal take: I’d keep the 410s. It’s the “right” way, even if crawlers are ignoring it. But if it’s driving you nuts, try 404s for a month and see what happens.
-
Are the urls still indexed?
Are they linked from somewhere on the site?
I usually only use 410’s until they are out of the index then delete the 410 -
I’d have to assume that there are links to them somewhere on your site or from backlinks, there’s a reason crawlers keep hitting those URLs.
-
Where are you getting notifications of the 400 code? GSC? Maybe it is using an old xml sitemap. That would have Google keep going down a path that doesn’t exist.
Log in to reply.