If you have ever made a mistake with your robots.txt file – either by letting Googlebot crawl something you didn’t want it to or because you accidentally blocked Googlebot (or someone hacked your site and did it) – it can be some nail biting times until you see Googlebot picking up the new robots.txt changes. […]
New Robots.txt Checker & Editor in Google Webmaster Tools
Even seasoned SEOs can sometimes stop and wonder “Did I do that robots.txt right?”, especially when dealing with more complicated robots.txt files that might disallow certain directories or certain bots. There are a few robots.txt checkers out there,but sometimes even the responses can be confusing for new webmasters. Fortunately Google has just released a new […]