Will Critchlow's Posts



Google’s Robots.txt Parser is Misbehaving

The newly-released open source robots.txt parser is not, as Google claims, the same as the production Googlebot parsing code. In addition, we have found cases where each of the official resources disagrees with the others. As a result, there is currently no way of knowing how the real Googlebot treats robots.txt instructions. Read on for example robots.txt files that are treated differently by Googlebot and by the open source parser.

Googlers: if you’re reading this, please help us clarify for the industry how Googlebot really interprets robots.txt.

Google recently released an open source robots.txt parser that they claimed is “production code”. This was very much needed because, as they said in the announcement blog post, “for 25 years, the Robots Exclusion Protocol (REP) was only a de-facto standard”.

Continue reading >>

SearchLove Community Speakers 2019: Applications Now Open

It’s crazy to think that a year has passed since we launched the SearchLove community speaker project. Since then we’ve worked alongside 9 relatively new speakers from Birmingham (UK), Leeds, Belfast, Boston and San Diego, and then had the pleasure of seeing them deliver amazing presentations to audiences at SearchLove London, San Diego and Boston. 

Here’s the best bit: we’re going to do it all over again. Applications for SearchLove London community speakers are now open and we want you to apply!

What is a SearchLove community speaker?

Our community speaker sessions are 20 minutes long and delivered by speakers who have been supported and coached by the Distilled team to take their presenting skills to the next level.  

Continue reading >>

Interviewing Google’s John Mueller at SearchLove: domain authority metrics, sub-domains vs. sub-folders and more

I was fortunate enough to be able to interview Google’s John Mueller at SearchLove and quiz him about domain authority metrics, sub-domains vs. sub-folders and how bad is ranking tracking really.

I have previously written and spoken about how to interpret Google’s official statements, technical documentation, engineers’ writing, patent applications, acquisitions, and more (see: From the Horse’s Mouth and the associated video as well as posts like “what do dolphins eat?”). When I got the chance to interview John Mueller from Google at our SearchLove London 2018 conference, I knew that there would be many things that he couldn’t divulge, but there were a couple of key areas in which I thought we had seen unnecessary confusion, and where I thought that I might be able to get John to shed some light.

Continue reading >>

Help us find the next search industry rising star for SearchLove San Diego 2019

After the success of running community speaker sessions at SearchLove London we are delighted to be bringing them to our American conferences, starting with SearchLove San Diego in March 2019. Our community speaker sessions are 20 minutes long, presented by relatively new speakers, who we support and coach and then give the chance to stand on stage in front of 200+ digital marketers from around the world.

If this sounds like something you would love to experience, then we are on the lookout for speakers who:

  • Are San Diego locals (no further than 2-2.5 hours drive from the venue). We want to support the community where our conference runs and help our speakers raise their local profile.
  • Have some speaking experience, preferably a couple of speaking gigs, and are looking to break onto an even bigger stage.

Continue reading >>

Marginal losses: the hidden reason your SEO performance is lagging

Without a structured testing program, our experience shows that it’s very likely that most SEO efforts are at best taking two steps forward and one step back by routinely deploying changes that make things worse.

Sign up if you’d like to receive SEO split testing case studies and more information about ODN.


This is true even when the thinking behind a change is solid, is based on correct data, and is part of a well-thought-out strategy.

Continue reading >>

< older posts