Google has added an addition to their robots.txt tool. Now, you can do the usual check of your robots.txt file, but then you can also verify the live version and then submit it to notify Google that your robots.txt file has changed and request that Googlebot crawl it again.
You can access the new tool within Google Webmaster Tools by selecting “Crawl” in the left sidebar, and then choosing robots.txt Tester.
Jennifer Slegg
Latest posts by Jennifer Slegg (see all)
- 2022 Update for Google Quality Rater Guidelines – Big YMYL Updates - August 1, 2022
- Google Quality Rater Guidelines: The Low Quality 2021 Update - October 19, 2021
- Rethinking Affiliate Sites With Google’s Product Review Update - April 23, 2021
- New Google Quality Rater Guidelines, Update Adds Emphasis on Needs Met - October 16, 2020
- Google Updates Experiment Statistics for Quality Raters - October 6, 2020