Here in the R&D department at Distilled, we work on a number of different products to improve the way we manage our clients’ sites, the latest of which is a Python library (that we have open-sourced) to help use Google’s disavow tool within your Search Console. We’ve called the library Moneypenny (she disavows secret agents, we are disavowing links … geddit?) and if Python is not your bag, don’t worry, there is a web application at disavow.it to help you along the way.
In short, Moneypenny allows you to create or modify an existing disavow file (consistently formatting URLs, removing duplicates and creating ‘domain:’ entries), to make sure you comprehensively and accurately disavow only the backlinks you do not want. There is also the ability to test your disavow file against a list of URLs to make sure it behaves just as you’d expect, and to help with reporting.
What is Google’s disavow tool?
It is a common scenario nowadays to start working with a new client, to find that a previous agency had built a variety of lower quality links that are doing more harm than good. Whilst you can email the webmaster of any site which you do not want linking to you, asking them to remove their link, it all too often goes unanswered.
Released in 2012, Google’s disavow tool allows you to specify a list of URLs which link to your site which you wish to ‘disavow’, declaring that you wish them not to be counted in Google’s algorithm. We aren’t going to go into a lot detail about how to use the disavow tool here - there are plenty of great resources you can use to learn more about the tool, take a look in the ‘Resources’ section of this post below.
In a nutshell - Google expects a .txt file with each new entry that you wish to disavow within your Search Console (https://www.google.com/webmasters/tools/disavow-links-main).
Domain entries vs URL entries
While conducting an audit of all the sites that are linking to your site, you may find that you have a large number of links coming from one problematic domain. For example, ‘www.example.com/a/spam’, ‘www.example.com/#spam’, ‘www.example.com/a/c/?spam’, and rather than individually disavowing each one (which is not only time consuming but will not prevent links from new pages from that domain), Google allow you to block sites at the domain level. In this example, preventing anything that comes from ‘www.example.com’ can be blocked.
Moneypenny is still early in its development and we welcome contributions from the SEO community to help build it out with new features. There is some Python incoming in the section below, so feel free to skip straight ahead to the next section if you just want a handy front end tool.
Moneypenny is installable via pip:
pip install moneypenny
And while there are a few helper functions, to aid with importing, cleaning, and parsing URLs, all of these are all contained within disavow_file_to_dict(), which takes in a .txt file and an optional domain limit (covered in the next section); and outputs a dictionary, containing some useful counts for checking purposes; and importantly, ‘url_entries’ and ‘domain_entries’’ which contain the entries for your disavow file.
This output can be merged back into your original file (to maintain your original order and comments) using combine_with_original_disavow(). The first parameter is the result of extract_file_contents() on your original file, and the second parameter is the dictionary output of disavow_file_to_dict().
Disavow.it is our simple to use web app, powered by Moneypenny. Just paste in all the URLs you want to disavow or upload a text file (one per line), and hit ‘Create/Clean Disavow File’.
You’ll get a summary of work disavow.it has done behind the scenes and a link to download your .txt file ready to upload to your Search Console.
A sample output from Disavow.it
When entering URLs, you can include comments (by having a # at the start of the line) and they will remain in place in the outputted document. There is also the option to download a version of your disavow file without comments.
Disavow.it will use Moneypenny to normalize all the URLs you fed it, which helps detect duplicates, and ensure that URLs are error free. The dupe detection can be very handy when you’ve gathered links from multiple sources to use in your disavow file.
Also, you can set a domain limit, which will look for any domains that exceed the limit, and replace them with a single ‘domain:’ entry. So in the example from before, if you had set the domain limit to 3, instead of individually disavowing those three links, there would instead be a single ‘domain:www.example.com’ entry.
Once you have your disavow file, you can check it against a list of URLs under the ‘Test Disavow File’ tab. Simply upload your disavow file and the file containing the URLs you wish to test and hit ‘Apply Disavow File to URLs’. Again you’ll get some summary statistics and the chance to download a list of those links disavowed and those not disavowed.
To do list
In the future, we hope to support uploading output from sites such as Majestic or Link Research Tools in order to make auditing your backlinks much simpler.
We are also working on a desktop application for much larger disavow tasks, which are not doable using a web application.
In the meantime, what do you think of the tool? I'd love to hear your thoughts and feedback.
Should I write constructive comments to help the team at Google?
The comments are not read by any humans and are for your benefit only, perhaps to record your rationale for disavowing the particular link, or any interaction you might have had with the webmaster of the site linking to you.
By uploading a disavow file am I not drawing unwanted attention to myself?
Uploading a disavow file is in no way an admission of guilt to trying to game Google’s PageRank algorithm and Google definitely do not see it as such. Nor should you panic if the owner of a site that you link to asks for you to take down the link lest they add you to their disavow file. Google does not use the contents of any disavow file as the basis for any algorithm or decision making process.
I’ve made a mistake, is there anyway to reavow sites I’ve blocked?
Fortunately, unlike secret agents, sites that you have mistakenly blocked can be brought back into the fold, by uploading a revised version of your disavow file to your Search Console.