Many SEOs rely pretty heavily on link tools such as Ahrefs and Majestic in order to audit links or simply watch for newly linking sites.  But some websites are either tired of receiving link removal requests or are selling backlinks and don’t want them to show up as a low quality link under analysis, and are blocking sites such as Ahrefs and Majestic (along with other link tools such as Moz) from crawling them.

Tiring of Link Removal Request Spam

Many webmasters are tiring of link removal requests, especially on large quality sites that could have tens of thousands of outgoing links.  And because many of the emails are automated will repeatedly resend, there is little webmasters can do besides – ironically – reporting repeated link removal requests as spam.

Hiding Sketchy Links

But there is the other side of the coin where those who sell poor quality backlinks don’t necessarily want them showing up in link tool sites such as Majestic, especially if they appear to come from sketchy sites or have a huge number from a single site.

In a YouMoz submission from Robert Fisher, he reveals what lengths some SEOs are going to in order to prevent their sites from being accessed by the third party link tool bots.

Blocking Third Party Link Tool Bots

The way sites are blocking link crawling bots is not through a regular robots.txt file or meta tag –  so not something that is easily checkable from.  Instead, they are blocking them by giving them a 406 Not Acceptable error response code, which was confirmed by using a user agent switcher to test.  This means that those checking for blocked bots in the robots.txt file are going to assume that those link related bots are able to crawl the sites properly.

In this case, it does seem to be a .htaccess issue, which makes a user agent switcher necessary to see whether a site is blocking link bots or not.

But Why?

Now, blocking link tools isn’t anything new, many have been doing it for years in an attempt to hide their personal link networks from prying eyes – and Google.  Given enough data and link information, many link tools can make some pretty good estimates at what sites are linked together, something spammers or those selling high quality links don’t want to have.   There are many .htaccess scripts floating around that will block any number of the third party linking tools.

And with SEOs able to report spam to Google, many don’t want to give their competitors any ammunition to report them for spamming, especially if they are crossing over that white line.  While blocking these bots will make it more difficult to find backlink data on sites you don’t own, it won’t be impossible either, just the skillset of SEOs who had to deal with linking prior to having all the fancy link tools available.

Then of course there is the negative SEO aspect of it – if someone decides to point 50,000 crappy links at their competitor, they want to make it has hard as possible for their competitor to find those links, and by blocking link bots, it means they will have to discover them on their own or wait for them to show up in Google Webmaster Tools.

What Can be Done?

What can people do?  SEOs have always known that what appears in Google Webmaster Tools is the “be all, end all” when it comes to the links they see.  If you have been penalized by Google – either algorithmically or manually – you should be checking Google Webmaster Tools daily until the penalty has been lifted as one part of your penalty killing strategy.

On the other hand, tools like Majestic and Ahrefs tend to show links much, much sooner, gives much more depth into each link and gives SEOs much more data to analyze than we will likely ever see in Google Webmaster Tools.  This is why third party tools are so crucial to SEOs and they should play a big role in your plan as well – but with that grain of salt incase you do have backlinks from sites blocking link tools.

I suspect as webmasters get more and more tired of endless link removal requests (which is why most tend to get ignored) that they will begin blocking services such as these – or at least for the link removal tools themselves.  And some of the more competitive market areas will likely find the prevalence of this much higher than those in low competitive markets.

I have reached out to both Majestic and Ahrefs for comment and will update when I hear back.