X
    Categories: Google

New Robots.txt Checker & Editor in Google Webmaster Tools

Even seasoned SEOs can sometimes stop and wonder “Did I do that robots.txt right?”, especially when dealing with more complicated robots.txt files that might disallow certain directories or certain bots.  There are a few robots.txt checkers out there,but sometimes even the responses can be confusing for new webmasters.  Fortunately Google has just released a new tool to make that all much easier.

Google has released a brand new Robots.txt Tester right in Google Webmaster Tools.  The tester will check your syntax in your robots.txt file, showing your errors and warnings related to your robots.txt file.

The new checker can be found in the Crawl section of Google Webmaster Tools.

It will also allow you to edit the file on the fly so you can check it as you work on it. There is also a link that allows you to look at the current robots.txt file that is live for that site.

And best of all, they have added a new quick form to the bottom that allows you to easily disallow the most common Googlebots from either your entire site or on a directory or page level.  You can disallow Googlebot, Googlebot-News, Googlebot-Image, Googlebot-Video, Googlebot-Mobile, Mediapartners-Google and Adsbot-Google.  You cannot use it to add other non-Google bots directly with the form however.

This is a great addition to Webmaster Tools, especially for those webmasters who know just enough about robots.txt to get themselves (and the index-ability of their sites!) in trouble.

The following two tabs change content below.

Jennifer Slegg

Founder & Editor at The SEM Post
Jennifer Slegg is a longtime speaker and expert in search engine marketing, working in the industry for almost 20 years. When she isn't sitting at her desk writing and working, she can be found grabbing a latte at her local Starbucks or planning her next trip to Disneyland. She regularly speaks at Pubcon, SMX, State of Search, Brighton SEO and more, and has been presenting at conferences for over a decade.
Jennifer Slegg :Jennifer Slegg is a longtime speaker and expert in search engine marketing, working in the industry for almost 20 years. When she isn't sitting at her desk writing and working, she can be found grabbing a latte at her local Starbucks or planning her next trip to Disneyland. She regularly speaks at Pubcon, SMX, State of Search, Brighton SEO and more, and has been presenting at conferences for over a decade.