The newly-released open source robots.txt parser is not, as Google claims, the same as the production Googlebot parsing code. In addition, we have found cases where each of the official resources disagrees with the others. As a result, there is currently no way of knowing how the real Googlebot treats robots.txt instructions. Read on for example robots.txt files that are treated differently by Googlebot and by the open source parser.
Googlers: if you’re reading this, please help us clarify for the industry how Googlebot really interprets robots.txt.
Google recently released an open source robots.txt parser that they claimed is “production code”. This was very much needed because, as they said in the announcement blog post, “for 25 years, the Robots Exclusion Protocol (REP) was only a de-facto standard”.