X
    Categories: AppleGoogle

Apple’s Applebot Follows Googlebot’s Instructions in Robots.txt Files

John Mueller from Google pointed out an interesting fact about Apple’s Applebot – if a robots.txt file makes no mention of Applebot, but does mention Googlebot, the Applebot will follow the same instructions as for Googlebot.

Apple’s Applebot is used for both Siri and Spotlight Suggestions for iOS users.

What if you do not want the Applebot to follow Googlebot’s instructions?  You will need to add specific instructions for Applebot into your robots.txt, and it will then follow those instructions and not the ones for Googlebot.

From their Applebot support documents:

User-agent strings will contain “Applebot” together with additional agent information. For example:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)

If robots instructions don’t mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions.

John Mueller also mentions that he is unaware of any other crawler that follows Googlebot instructions.  It is an interesting decision, but since few people are aware of the Applebot, it does make sense that they don’t want to waste a webmaster’s resources by crawling pages that the webmaster has said they don’t want Googlebot crawling.  It isn’t as well known as Bing’s Bingbot.

The following two tabs change content below.

Jennifer Slegg

Founder & Editor at The SEM Post
Jennifer Slegg is a longtime speaker and expert in search engine marketing, working in the industry for almost 20 years. When she isn't sitting at her desk writing and working, she can be found grabbing a latte at her local Starbucks or planning her next trip to Disneyland. She regularly speaks at Pubcon, SMX, State of Search, Brighton SEO and more, and has been presenting at conferences for over a decade.
Jennifer Slegg :Jennifer Slegg is a longtime speaker and expert in search engine marketing, working in the industry for almost 20 years. When she isn't sitting at her desk writing and working, she can be found grabbing a latte at her local Starbucks or planning her next trip to Disneyland. She regularly speaks at Pubcon, SMX, State of Search, Brighton SEO and more, and has been presenting at conferences for over a decade.