How The Times Misplaced 270,000 Links in 6 Lines of Code

This is a post I’ve been meaning to write for a long time but I recently came across a search which demonstrates the idea perfectly so I decided it was time to put my thoughts down on paper (so to speak).

Before I explain - check out this search for [the times]:

You can see that 5 of the top 6 results are owned by the times (clicking on www.the-times.co.uk, www.thetimes.co.uk & www.sunday-times.co.uk all redirect you to the main times website which ranks number 1).

So what’s happening here?

Well what’s actually happening is slightly complicated. If you go to www.the-times.co.uk you are 301 redirected to the main times website. This 301 redirect clearly isn’t being spidered by Google however since these domains have their own listing in the search results. So I checked their robots.txt (the same for all 3 sites):

#Version: 0.1
#Last updated: 17/06/2005
#Timesonline Newspaper
#Site contents Copyright Times Newspapers Ltd
#Please note our terms and conditions http://www.timesonline.co.uk/section/0,,497,00.html
#Spidering is not allowed by our terms and conditions
#Authorised spidering is subject to permission
#For authorisation please contact us - see http://www.nisyndication.com/about_us.html
User-agent: *
Disallow: /

Ah-ha! While they have 301 redirected the root page they have forgotten to 301 redirect their robots.txt file which still instructs the search engines not to crawl the website. Since the search engines look at the robots.txt first before they crawl the rest of the site they’re not getting as far as the 301. Since Google doesn’t know the domains redirect they are still able to rank in their own right (without a snippet of course).

A similar effect can be achieved with 302 redirects. For example - check out this search result from a yahoo search for [tom critchlow]:

http://www.tom-critchlow.co.uk is a 302 redirect to my page on the Distilled site and ranks number one in yahoo for a search for my name. I haven’t dropped many links to it yet so Google hasn’t picked it up yet but I’m fairly sure it will in due course.

So what we’re seeing here is a way of owning some search result listings without having to actually create any new websites, just use some clever redirects. This is a very powerful tool for reputation management but there’s a few things worth bearing in mind:

##Drawbacks of this method

The first and most obvious drawback to this method is that the links which point to your spare domain are not passing any link juice onto your main website. Between those domains there are 267,700 links which are not passing any value to the main times website. Since these links are links which your main site could easily have gained it seems a waste not to get any value out of them for your main site.

The second issue is that these pages are actually quite difficult to rank. I haven’t done extensive testing for 302 redirects but in the case of a site blocked by robots.txt the search engines can’t crawl the site to get any indication of what it’s about therefore it’s harder to rank and will require more links than if it was a genuine site with real content. (Note that link building for 301 or 302 redirected domains is difficult since people won’t naturally link to these domains since there isn’t any content on them!)

The third issue is more hypothetical but in my eyes you run the risk of Google changing the way it handles 302 redirects or pages behind a robots.txt. In my eyes, the above search results don’t add that much value to the user since the top results all point to the same website so I think you run the risk of these search results disappearing at some stage. If you’re relying on these pages for reputation management then they could disappear overnight further down the line.

##Benefits of this method

The major benefit of this method which springs to mind is that you don’t have to fragment your traffic. All those sites point to your main content and so you don’t lose any traffic to mini-sites or profile pages.

The second benefit is that these domains are really quick, easy and cheap to set up. Setting up a mini-site or creating some kind of secondary online presence to rank with can be an expensive project and difficult to get past a company’s marketing department. By using this method you neatly sidestep all of those concerns.

##Tips & Tricks

Ensure the keyphrase is in the URL - since the search engines can’t get any indication of relevance from the content on the site you need to ensure that they can get relevance from the domain name. You’ll also need a fair number of anchor text rich links, the number will depend on how competitive your search term is.

Unlike pages behind a robots.txt the search engines actually crawl 302 redirects and look at the landing page to determine what snippet should be displayed therefore it’s worth paying attention to which page you’re redirecting to ensure that your snippet and title are sensible and relevant.

I’m sure there will be more tips and tricks on their way as I experiment with linking to tom critchlow from a few more places but if anyone else wants to share some tips on how to rank 302’d domains or pages behind robots.txt please shout out in the comments!

Get blog posts via email

1 Comment

  1. In the Netherlands we have some travel related brand name queries, that actually show affiliate links, instead of a subpage for that brand name. Talk about bad SEO :P

    reply >

Leave a Reply

Your email address will not be published. Required fields are marked *

*
*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>