The internet is notorious for tracking users, from behaviour to the finer details of who we are. Sometimes, we just don't want to disclose our identity and protect our privacy. While proxy servers are no secret to most of the technically inclined, there are still many users who lack the knowledge to access them.
With the help of Paul Rosenthal, CEO of Trusted Proxies (a damn good proxy expert, and all around great guy), I hope to give you enough knowledge to become inquisitive and hopefully give these services a try. All of the answers were kindly written by Paul, and if you'd like to find out more please do so in the comments.
Please use the quick jump to links below to skip to the information you need
- What exactly is a proxy server?
- What are proxy servers mainly used for in the SEO industry?
- What does a proxy server address look like?
- How exactly do they work?
- What is required to use a proxy server?
- What are the common types of proxy servers? HTTP, Socks, SSL
- In regards to proxy anonymity, are there different types / levels?
- Are there any risks with using Proxy servers?
- Any additional expert tips?
VPN (Virtual Private Networks)
A Proxy Server is an intermediate computer server, that sits between a computer accessing the internet (the client), and the destination website the client is trying to surf.
SEO’s use Proxy Servers to hide or change their true IP address. There are several reasons you might want to do this:
- If you access competitor websites a lot for research, their webservers log your IP address and may be able to trace your activities back to you. By routing your requests via a Proxy Server, your true identity is hidden
- If your agency manages lots of Social Media accounts on behalf of clients, eg Facebook and Twitter, your agency will be making lots of requests for many different accounts but all from the same IP address. Facebook and Twitter may accidentally detect this legitimate activity as if you are spamming, and so block you. Routing each of your clients’ Social Media activity via a different Proxy Server makes it look like normal activity with each account’s activity coming from a different IP address.
- SEO agencies tend to send lots of queries to Google, Bing and other Search Engines. If you’re all on the same office network, all that activity from the same IP address, especially if you use keyword ranking software, looks like a potential Denial of Service attack to the Search Engines. They might then block your IP address, preventing you from further searching. Sending your queries via different Proxy Servers mitigates that, and can thus also be used to speed up your keyword ranking reports.
- To surf as if you are in another country. Many websites, especially Search Engines, serve up different content depending on your country. They determine it by detecting your IP address and looking it up in a commercial Geo-IP database (Google being Google of course are believed to use their own proprietary algorithms for working it out). For example, if your SEO Agency was based in the UK but you had a client in the USA, the results you get from Google.com are going to be different than your client's results. This is because Google is very helpfully trying to “personalise” the results. Several SEO packages like Advanced Web Ranking, WebCEO and Rank Tracker can mimic this by tweaking the parameters it sends to Google to “trick” it into thinking you’re in another location. I’ve seen SEO’s report different levels of success with this technique, with AWR getting the best results in my experience. But as Google particularly seeks to “personalise” more and more, they are suspected to increasingly be taking your IP address into account when they personalise the SERPs. And this technique doesn’t work with all search engines or software. Therefore we see a lot of SEO Agencies who have clients overseas use Proxy Servers based in their clients’ countries, to ensure they get the same personalised results as their clients.
A Proxy Server’s IP address looks just the same as any IP address on the internet. It is composed of 4 numbers between 0 and 255, each separated by a “.” Each IP address can also be broken down into classes - as this is a difficult subject please carry on reading about classes here and here. If you need to use lots of Proxy Servers, you might want to avoid sequential IP addresses as lots of requests from sequential IP addresses could look nearly as suspicious as coming all from the same one! In which case being able to use Proxy Servers on at least multiple C Classes, and ideally multiple B and A Classes, makes them look even more random and hence less detectable.
The :8080 in this example refers to the port number that you access the Proxy Server on. By default, almost all web traffic flows through port 80 and as it is thus assumed, and does not need to be specified as part of the address which is why you never see it. Proxy Servers on the other hand are often accessed on different ports, and in this example, the Proxy Servers is on port 8080. Top answer here is a good explanation between 80 and 8080.
When your web browser or software is configured to use a Proxy Server, the requests it sends to the internet do not go directly to the target website. Rather they are sent first to the Proxy Server, and the Proxy Server forwards the request on to the target website. Every internet request has its originating IP address in its “header”. The Proxy Server may replace your IP address for its IP address in the header. When the target websites returns its results eg the web page content, it returns it back to the Proxy Server, which in turn returns it to you.
For a standard “HTTP” Proxy Server, you don’t need anything extra. Every web browser supports Proxy Servers, and most software does also. On a Windows PC if the software doesn't support Proxy Servers directly, you can configure Internet Explorer to use a Proxy Server, and then the software is usually “forced” to use the Proxy Server also. All you need to know is how to configure your browser or software to use a Proxy Server.
- Configure Firefox to Use a Proxy Server
- Configure Google Chrome to Use a Proxy Server
- Configure Internet Explorer to Use a Proxy Server
- Configure Safari to Use a Proxy Server
There’s two main types of Proxy Server these days, “HTTP” or “SOCKS”. The names refer to which part of the network protocol they use to work. HTTP is the usual standard and any software that can use a Proxy Server can use HTTP. SOCKS is slightly less popular and not supported by as much software. Technically SOCKS is more powerful as it can send more types of traffic via a Proxy Server. However, because most “legitimate” uses of a Proxy Server can achieve their desired result from HTTP Proxy Servers, there is not as much need for SOCKS, and so SOCKS use is sometimes considered to be a bit suspicious.
Hence if you can achieve what you need to with HTTP (which for SEO’s is true 99% of the time), there is no need to look for SOCKS Proxy Servers. One detail to look out for is to ensure your Proxy Server can support SSL. SSL is what is used to encrypt a secure internet connection, and not all Proxy Servers support it by default. If they don’t then the secure parts of your internet requests will not be sent via the Proxy Server but instead will go direct, tipping off your target website as to your true identity!
Proxy Servers have many technical uses, some nothing of interest to an SEO and you should avoid. Here’s the main types:
- Transparent: In the headers, it identifies itself to target websites as being a Proxy Server, and gives your originating address. This might be used by an ISP for example to speed up certain internet access, and they want target websites to know what they’re doing.
- Anonymous: Identifies itself to target websites as a Proxy Server, but hides your IP address. This protects your identity but also tells the target website it probably isn’t really you. People would use this type of Proxy Server for partial anonymity to hide their real IP address, but don’t want or need to hide that they’re doing it. A bit like being “ex-directory” in the phone book. Certain websites may then choose to block you anyway (or a devious website owner may choose to serve you up some other special content to trick you eg fake prices!).
- Distorting: These are a variation on Anonymous Proxy Servers in that they still alert target websites that you’re using a Proxy Server, but instead of hiding your IP address completely, will switch it for a fake one.
- High Anonymity or “Elite” Proxy Servers: They do not identify your traffic as having been forwarded via a Proxy Servers, the requests look just like regular internet requests direct from your computer. But your IP address is replaced with that of the Proxy Server. So target websites have no idea that you’re coming via a Proxy Server, nor any idea of your real IP address and identity. This is the type most suitable for SEO’s.
The most common type of Proxy Server that people search for and try using are free Proxy Servers. They’re also known as “open” or “public” Proxy Servers. “Proxy Server Lists” is a popular search term for finding them. People are used to so many things on the Internet being free, eg Gmail, Facebook, photo sharing etc, that they assume free Proxy Servers are no problem either. But using free/open/public Proxy Servers is a disaster for SEO Agencies and should be avoided at all costs. This is because nobody in their right minds installs a Proxy Server on their computer and lets any stranger from anywhere get access to itand use all their bandwidth. 99.99% of free Proxy Servers are in fact either:
- Private Proxy Servers that a Network Administrator has incorrectly configured and accidentally made it public. As they’re “public”, if you’ve found them, so has everyone else, and within days if not hours, they will be overwhelmed and performance will become painfully slow until the Proxy Server crashes altogether
- Or, far more likely, is a computer that has been hacked with a virus or Trojan and has been turned into an open, public Proxy Server, without the owner’s knowledge or consent, and may have become part of a hacker’s “botnet”. Eventually the owner will realise something is up and their anti-virus or anti-malware software will remove the Proxy Server.
In either case, the Proxy Server is very unreliable and will eventually stop working and need to be replaced – you find yourself on a constant “cat and mouse” chasing after new working Proxy Servers – along with hundreds of thousands of other people.
But there are worse problems. Your computer, network and data is at a huge security risk. Free open/public Proxy Servers were created by all kinds of people many with bad intentions. They are being used by hundreds or thousands of other random strangers, many of whom are using them for blackhat activity which you could be accidentally associated with. Your data is at high risk of interception, and the connection you just opened to a highly compromised computer could in itself inadvertently open a security hole into your organisation. Worse still, hackers are known to set up “honey pots”. Knowing that large numbers of people are searching for them, they set up “public” Proxy Servers, and actively monitor them, to see what kinds of interesting information they can pick up and exploit.
Therefore, you should only use Private, paid for Proxy Servers from a reputable and trustworthy Proxy Service provider. This is NOT like email services. Everyone knows you can pay for email, but why bother when the free services like Hotmail and Gmail are almost as good if not better. But 99.99% of free Proxy Servers could be very bad for your Corporate health and should be avoided at all costs!
- Example of a free proxy service - VISIT AT YOUR OWN RISK
- Example of a paid proxy service from Paul's company.
- If you need to use Proxy Servers for your SEO Agency and you come to rely on them for your business, you should seek out a reputable, reliable, Enterprise Class Proxy Server supplier. Like any other service provider for your business, you need to know they’re reliable, have a fast infrastructure, have knowledgeable Sales and Support people you can speak to anytime, and are entirely safe to use.
- Price absolutely should not be your determining factor. As we have seen, free is a disaster. And very cheap providers are often just harvesting dangerous free Proxy Servers from the Internet, packaging them up and selling them on. A private Proxy Server provider that advertises that they will frequently replace your Proxy Servers should you get blocked is another warning sign. That’s because real, fast, reliable Proxy Servers cost real money to run, so you have to question how a low cost provider could be replacing so many of them so quickly. Ironically what looks like a higher level of service may be an indicator of potential trouble! You should only use a provider where you can verify they use their own private network of servers in high quality Data Centers.
- For best results, you may also prefer a provider who has servers in many different locations and large numbers of unique A, B and C Classes for their IP addresses, to avoid detection by not just your own traffic on sequential or closely related IP addresses, but insulation against what others may be doing in the same IP address “neighbourhood”. The cheaper a provider is, and the less fussy they are about who they have as clients, the more you need to worry about what others may be doing that could affect you, even if the Proxy Servers you buy are private. Eg Google spammers use cheaper providers, and if Google gets ticked off enough, they blacklist entire blocks of IP addresses, and your Proxy Server could be one of those “tarred with the same brush”.
- SEOs with overseas clients may want to use a Proxy Server provider that has Proxy Servers in many worldwide locations.
VPN stands for “Virtual Private Network”. Their main use is typically to connect remote workers to their office’s central server, over a secure encrypted link so that all communications are kept secure. As this will have the effect of changing your IP address, companies have sprung up offering private VPN services. You connect to their VPN server, your IP address is changed to theirs, and then forwards on your web traffic, so the result is similar to that of a Proxy Server.
The obvious difference is that all your traffic between you and the VPN server is encrypted. But this is not as useful as it sounds, because if you’re surfing to a regular, non-secure website, the traffic between the VPN server and the target website is not encrypted so you didn't gain that much. The one big difference and benefit you do get is that all traffic from your computer is automatically sent via the VPN server, not just that sent by your browser or special software. This can be useful for certain websites, typically flash or video heavy sites, that don’t work properly over HTTP only Proxy Servers.
At first sight VPN’s sound better as they disguise your originating IP address for all your internet traffic. But there are downsides:
- Configuring your computer to use a VPN can be a bit technical or require downloading and installing special software, with all the potential Support headaches that entails
- A VPN will be a bit slower than a comparable Proxy Server due to the extra processing required to initially connect and then encrypt and de-encrypt all the traffic
- You may not want to send all of your traffic via the VPN
- You can only use one VPN connection at a time. This may be fine for many applications, but if you need to use multiple connections, eg the examples above for using multiple Social Media accounts or speeding up keyword ranking reports, it won’t work so smoothly.
Hence my recommendation is if a regular Proxy Server will do the job you need doing, stick with that and don’t use a VPN.
Other great resources for SEO proxy server education:
- What is a Reverse Proxy and How Can it Help My SEO?
- Proxy caching for SEO
- Why SEO experts use reserve proxy servers?
- How to block proxy servers via htaccess
Thanks for reading, whoever you are ;)