Will Critchlow's Posts



Free web-based robots.txt parser based on Google’s open source C++ parser

The punchline: I’ve been playing around with a toy project recently and have deployed it as a free web-based tool for checking how Google will parse your robots.txt files, given that their own online tool does not replicate actual Googlebot behaviour. Check it out at realrobotstxt.com.

While preparing for my recent presentation at SearchLove London, I got mildly obsessed by the way that the deeper I dug into how robots.txt files work, the more surprising things I found, and the more places I found where there was conflicting information from different sources.

Continue reading >>

SearchLove San Diego 2020: Community Speaker Applications Now Open

Are you looking to secure your first speaking gig? Maybe you’ve done a couple of local meetups and are looking to share your knowledge in front of a bigger audience. Well good news, applications for community speaker sessions for SearchLove San Diego 2020 are now open! Not sure what our community speaker sessions are about? Read on.

If you would love the opportunity to speak in front of 250 people in sunny San Diego then we’d love to hear from you. We are on the lookout for speakers who:

  • Are San Diego locals (no further than 2-2.5 hours drive from the venue). We want to support the community where our conference runs and help speakers raise their local profile.

Continue reading >>

Google’s Robots.txt Parser is Misbehaving

The newly-released open source robots.txt parser is not, as Google claims, the same as the production Googlebot parsing code. In addition, we have found cases where each of the official resources disagrees with the others. As a result, there is currently no way of knowing how the real Googlebot treats robots.txt instructions. Read on for example robots.txt files that are treated differently by Googlebot and by the open source parser.

Googlers: if you’re reading this, please help us clarify for the industry how Googlebot really interprets robots.txt.

Google recently released an open source robots.txt parser that they claimed is “production code”. This was very much needed because, as they said in the announcement blog post, “for 25 years, the Robots Exclusion Protocol (REP) was only a de-facto standard”.

Continue reading >>

SearchLove Community Speakers 2019: Applications Now Open

It’s crazy to think that a year has passed since we launched the SearchLove community speaker project. Since then we’ve worked alongside 9 relatively new speakers from Birmingham (UK), Leeds, Belfast, Boston and San Diego, and then had the pleasure of seeing them deliver amazing presentations to audiences at SearchLove London, San Diego and Boston. 

Here’s the best bit: we’re going to do it all over again. Applications for SearchLove London community speakers are now open and we want you to apply!

What is a SearchLove community speaker?

Our community speaker sessions are 20 minutes long and delivered by speakers who have been supported and coached by the Distilled team to take their presenting skills to the next level.  

Continue reading >>

Interviewing Google’s John Mueller at SearchLove: domain authority metrics, sub-domains vs. sub-folders and more

I was fortunate enough to be able to interview Google’s John Mueller at SearchLove and quiz him about domain authority metrics, sub-domains vs. sub-folders and how bad is ranking tracking really.

I have previously written and spoken about how to interpret Google’s official statements, technical documentation, engineers’ writing, patent applications, acquisitions, and more (see: From the Horse’s Mouth and the associated video as well as posts like “what do dolphins eat?”). When I got the chance to interview John Mueller from Google at our SearchLove London 2018 conference, I knew that there would be many things that he couldn’t divulge, but there were a couple of key areas in which I thought we had seen unnecessary confusion, and where I thought that I might be able to get John to shed some light.

Continue reading >>

< older posts