Towards the end of last year, I joined Tom Anthony in the R&D division at Distilled (you can read more about that here). We have been quietly working on a number of internal tools to support our consultants and help our clients. Today I’m announcing our first external tool which is the output of a hackday.
To help test some of the tools we are building, we needed a way of easily generating pages with various search engine directives enabled or disabled. One hackday later (and the inevitable few hours of overrun to finish the documentation) and we are ready to launch crawlbin.com.
The people who will benefit most from crawlbin are those writing or testing crawlers or tools that need to be able to handle potentially misleading signals. If you have a crawler as part of your inhouse tools then please do consider using crawlbin as a target for your tests.