-
SEO Reminder!
Robots.txt is a crawl management file and it doesn’t control indexing.
-
6 Replies
-
Unfortunately many SEOs don’t get this.
-
Can you explain what you mean by this further?
-
It is sufficient to have the sitemap path on the robots.txt side.
-
robots.txt controls nothing.
It’s like leaving a sugar cookie and a chocolate chip cookie on a park bench with a sign that says “Please eat the sugar cookie”.
Some people will only eat the sugar cookie, some will eat the chocolate chip, some will eat both, some will eat none.
Same thing with search engines. You can express your desire, but they are free to ignore you and do what they feel like.
Also crawling is different than indexing. Just because something is crawled doesn’t mean it will make it into the search results.
-
.htaccess
-
guess how google indentifies pbns? Look at robots.txt files and look for noindex to backlink tools.
Log in to reply.