X
    Categories: GoogleSEO

How to Unblock All CSS & Javascript for Googlebot Using Robots.txt

If you have received one of the many warnings Google sent out about “Googlebot cannot access CSS and JS files” in your Google Search Console, there has been much confusion about how to ensure you are allowing Googlebot to crawl these blocked resources without potentially opening yourself up to security issues, as the commonly blocked parts of WordPress – /wp-admin/ and /wp-content/plugins – are triggering the warnings, depending on how WordPress and various plugins are set up.

Fortunately, there is an easy way to allow Googlebot to crawl all CSS and JavaScript files while still allowing other areas to remain blocked.

Simply add the following to your robots.txt and it will allow Googlebot to crawl all the CSS and JavaScript files on your site.

User-Agent: Googlebot
Allow: .js
Allow: .css

Also remember that Google does not index .css and .js in their search results, so you do not need to worry about these files causing any unnecessary SEO issues.  In fact, the opposite will happen since blocking .css and .js files “can result in suboptimal rankings.”

Once you add this, you can confirm using the Google Search Console “Fetch as Google” to ensure the resources were successfully unblocked with the new robots.txt file.

While some commenters pointed out a similar robots.txt in comments here, Gary Illyes from Google confirmed the use of this in robots.txt at stackoverflow.

the simplest form of allow rule to allow crawling javascript and css resources:

User-Agent: Googlebot
Allow: .js
Allow: .css

This will allow anything like https://example.com/deep/style.css?something=1 or https://example.com/deep/javascript.js, and leaves no much space for interpretation for other search engines.

Once you have this, you can test your setup in Search Console – Blocked resources feature.

We have more details on the “Googlebot cannot access CSS and JS files” warning here, and here are instructions so you can find exactly which resources Google thinks you are blocking, even if you are certain it is a false positive (it probably isn’t).

The following two tabs change content below.

Jennifer Slegg

Founder & Editor at The SEM Post
Jennifer Slegg is a longtime speaker and expert in search engine marketing, working in the industry for almost 20 years. When she isn't sitting at her desk writing and working, she can be found grabbing a latte at her local Starbucks or planning her next trip to Disneyland. She regularly speaks at Pubcon, SMX, State of Search, Brighton SEO and more, and has been presenting at conferences for over a decade.
Jennifer Slegg :Jennifer Slegg is a longtime speaker and expert in search engine marketing, working in the industry for almost 20 years. When she isn't sitting at her desk writing and working, she can be found grabbing a latte at her local Starbucks or planning her next trip to Disneyland. She regularly speaks at Pubcon, SMX, State of Search, Brighton SEO and more, and has been presenting at conferences for over a decade.