The SEM Post

Latest News About SEO, SEM, PPC & Search Engines

You are here: Home / Google / How to Unblock All CSS & Javascript for Googlebot Using Robots.txt

How to Unblock All CSS & Javascript for Googlebot Using Robots.txt

at PST By

how to unblock css jsIf you have received one of the many warnings Google sent out about “Googlebot cannot access CSS and JS files” in your Google Search Console, there has been much confusion about how to ensure you are allowing Googlebot to crawl these blocked resources without potentially opening yourself up to security issues, as the commonly blocked parts of WordPress – /wp-admin/ and /wp-content/plugins – are triggering the warnings, depending on how WordPress and various plugins are set up.

Fortunately, there is an easy way to allow Googlebot to crawl all CSS and JavaScript files while still allowing other areas to remain blocked.

Simply add the following to your robots.txt and it will allow Googlebot to crawl all the CSS and JavaScript files on your site.

User-Agent: Googlebot
Allow: .js
Allow: .css

Also remember that Google does not index .css and .js in their search results, so you do not need to worry about these files causing any unnecessary SEO issues.  In fact, the opposite will happen since blocking .css and .js files “can result in suboptimal rankings.”

Once you add this, you can confirm using the Google Search Console “Fetch as Google” to ensure the resources were successfully unblocked with the new robots.txt file.

While some commenters pointed out a similar robots.txt in comments here, Gary Illyes from Google confirmed the use of this in robots.txt at stackoverflow.

…the simplest form of allow rule to allow crawling javascript and css resources:

User-Agent: Googlebot
Allow: .js
Allow: .css

This will allow anything like https://example.com/deep/style.css?something=1 or https://example.com/deep/javascript.js, and leaves no much space for interpretation for other search engines.

Once you have this, you can test your setup in Search Console – Blocked resources feature.

We have more details on the “Googlebot cannot access CSS and JS files” warning here, and here are instructions so you can find exactly which resources Google thinks you are blocking, even if you are certain it is a false positive (it probably isn’t).

The following two tabs change content below.
My Twitter profileMy Facebook profileMy Google+ profileMy LinkedIn profile

Jennifer Slegg

Founder & Editor at The SEM Post
Jennifer Slegg is a longtime speaker and expert in search engine marketing, working in the industry for almost 20 years. When she isn't sitting at her desk writing and working, she can be found grabbing a latte at her local Starbucks or planning her next trip to Disneyland. She regularly speaks at Pubcon, SMX, State of Search, Brighton SEO and more, and has been presenting at conferences for over a decade.

Sign up for our newsletter


Comments

  1. Yes, this is mandate now and it seems google has revamped its algorithm of ranking with the JS and CSS usability as prime factor.

    I have received for all most all the clients having the wordpress where i blocked the access to wp-include through robots file.

  3. I think the solution is incomplete. Directives mentioned in this article are not complete. Only “Allow: .js” will allow only those “.js” file that are at root level.

    To allow all the “.js” and “.css” files, the wild card operator (*) is required. See the code below;

    User-Agent: Googlebot
    Allow: *.js
    Allow: *.css

    • Yes, the solution referenced in this article is INCORRECT. You must have the “wildcard” command in the lines for this to work.

      This is correct:

      User-Agent: Googlebot
      Allow: *.js
      Allow: *.css

      This is NOT correct:

      User-Agent: Googlebot
      Allow: .js
      Allow: .css

      Make sure to include this at the top of the robots file since commands are read from top to bottom.

      Yoast has published a great new resource stating that you should also be removing (unblocking) access to /wp-admin/ as well.

      That’s a must-read and you can find that resource here (read the comments as well).

      https://yoast.com/dont-block-your-css-and-js-files/

  4. The suggestion is definitely wrong cause it doesn’t cover every case.

    Apart from that suggesting a predefined robots.txt is never a good idea.

    Never disallow anything except if you know what you are doing and after looking your server log files.

  5. If you have wordpress website then should need to edit like first go your wordpress dashboard
    and choose seo section and click on edit files and

    default code like

    User-agent: *
    Disallow: /wp-admin/

    insert new text below this code like

    User-Agent: Googlebot
    Allow: *.js
    Allow: *.css

Trackbacks