Crawl Budget Optimisation Through Log File Analysis
Log file analysis is one of those tasks you might not do often - due to data availability & time constraints - but that can provide insights you wouldn’t be able to discover otherwise, particularly for large sites. If you’ve never done a log analysis or are unsure what exactly to look for and where to start, I’ve built a guideline to help you:
- Get started with some log file analysis tools
- Understand what log files are useful for
- Digging into the data and think how to better redistribute crawling resources
Log files are essentially a diary of all requests made to your site for a specific time period. The data is very specific and more in depth than you could gather from a crawl, Google Analytics and Google Search Console combined.