Proxy server essential information for SEO’s

The internet is notorious for tracking users, from behaviour to the finer details of who we are. Sometimes, we just don’t want to disclose our identity and protect our privacy. While proxy servers are no secret to most of the technically inclined, there are still many users who lack the knowledge to access them.

With the help of Paul Rosenthal, CEO of Trusted Proxies (a damn good proxy expert, and all around great guy), I hope to give you enough knowledge to become inquisitive and hopefully give these services a try. All of the answers were kindly written by Paul, and if you’d like to find out more please do so in the comments.

Please use the quick jump to links below to skip to the information you need

VPN (Virtual Private Networks)

In one sentence, what exactly is a proxy server?

A Proxy Server is an intermediate computer server, that sits between a computer accessing the internet (the client), and the destination website the client is trying to surf.

Back to top

What are proxy servers mainly used for in the SEO industry?

SEO’s use Proxy Servers to hide or change their true IP address. There are several reasons you might want to do this:

  • If you access competitor websites a lot for research, their webservers log your IP address and may be able to trace your activities back to you. By routing your requests via a Proxy Server, your true identity is hidden
  • If your agency manages lots of Social Media accounts on behalf of clients, eg Facebook and Twitter, your agency will be making lots of requests for many different accounts but all from the same IP address. Facebook and Twitter may accidentally detect this legitimate activity as if you are spamming, and so block you. Routing each of your clients’ Social Media activity via a different Proxy Server makes it look like normal activity with each account’s activity coming from a different IP address.
  • SEO agencies tend to send lots of queries to Google, Bing and other Search Engines. If you’re all on the same office network, all that activity from the same IP address, especially if you use keyword ranking software, looks like a potential Denial of Service attack to the Search Engines. They might then block your IP address, preventing you from further searching. Sending your queries via different Proxy Servers mitigates that, and can thus also be used to speed up your keyword ranking reports.
  • To surf as if you are in another country. Many websites, especially Search Engines, serve up different content depending on your country. They determine it by detecting your IP address and looking it up in a commercial Geo-IP database (Google being Google of course are believed to use their own proprietary algorithms for working it out). For example, if your SEO Agency was based in the UK but you had a client in the USA, the results you get from Google.com are going to be different than your client’s results. This is because Google is very helpfully trying to “personalise” the results. Several SEO packages like Advanced Web Ranking, WebCEO and Rank Tracker can mimic this by tweaking the parameters it sends to Google to “trick” it into thinking you’re in another location. I’ve seen SEO’s report different levels of success with this technique, with AWR getting the best results in my experience. But as Google particularly seeks to “personalise” more and more, they are suspected to increasingly be taking your IP address into account when they personalise the SERPs. And this technique doesn’t work with all search engines or software. Therefore we see a lot of SEO Agencies who have clients overseas use Proxy Servers based in their clients’ countries, to ensure they get the same personalised results as their clients.

What does a proxy server address look like?

proxy-address-example1_2

A Proxy Server’s IP address looks just the same as any IP address on the internet. It is composed of 4 numbers between 0 and 255, each separated by a “.” Each IP address can also be broken down into classes - as this is a difficult subject please carry on reading about classes here and here. If you need to use lots of Proxy Servers, you might want to avoid sequential IP addresses as lots of requests from sequential IP addresses could look nearly as suspicious as coming all from the same one! In which case being able to use Proxy Servers on at least multiple C Classes, and ideally multiple B and A Classes, makes them look even more random and hence less detectable.

The :8080 in this example refers to the port number that you access the Proxy Server on. By default, almost all web traffic flows through port 80 and as it is thus assumed, and does not need to be specified as part of the address which is why you never see it. Proxy Servers on the other hand are often accessed on different ports, and in this example, the Proxy Servers is on port 8080. Top answer here is a good explanation between 80 and 8080.

Back to top

How exactly do they work?

When your web browser or software is configured to use a Proxy Server, the requests it sends to the internet do not go directly to the target website. Rather they are sent first to the Proxy Server, and the Proxy Server forwards the request on to the target website. Every internet request has its originating IP address in its “header”. The Proxy Server may replace your IP address for its IP address in the header. When the target websites returns its results eg the web page content, it returns it back to the Proxy Server, which in turn returns it to you.

proxy-server-ip-replacement

What is required to use a proxy server?

For a standard “HTTP” Proxy Server, you don’t need anything extra. Every web browser supports Proxy Servers, and most software does also. On a Windows PC if the software doesn’t support Proxy Servers directly, you can configure Internet Explorer to use a Proxy Server, and then the software is usually “forced” to use the Proxy Server also. All you need to know is how to configure your browser or software to use a Proxy Server.

How to’s

Browser tools

Back to top

What are the common types of proxy servers? HTTP, Socks, SSL

There’s two main types of Proxy Server these days, “HTTP” or “SOCKS”. The names refer to which part of the network protocol they use to work. HTTP is the usual standard and any software that can use a Proxy Server can use HTTP. SOCKS is slightly less popular and not supported by as much software. Technically SOCKS is more powerful as it can send more types of traffic via a Proxy Server. However, because most “legitimate” uses of a Proxy Server can achieve their desired result from HTTP Proxy Servers, there is not as much need for SOCKS, and so SOCKS use is sometimes considered to be a bit suspicious.

Hence if you can achieve what you need to with HTTP (which for SEO’s is true 99% of the time), there is no need to look for SOCKS Proxy Servers. One detail to look out for is to ensure your Proxy Server can support SSL. SSL is what is used to encrypt a secure internet connection, and not all Proxy Servers support it by default. If they don’t then the secure parts of your internet requests will not be sent via the Proxy Server but instead will go direct, tipping off your target website as to your true identity!

Back to top

In regards to proxy anonymity, are there different types / levels?

Proxy Servers have many technical uses, some nothing of interest to an SEO and you should avoid. Here’s the main types:

  1. Transparent: In the headers, it identifies itself to target websites as being a Proxy Server, and gives your originating address. This might be used by an ISP for example to speed up certain internet access, and they want target websites to know what they’re doing.
  2. Anonymous: Identifies itself to target websites as a Proxy Server, but hides your IP address. This protects your identity but also tells the target website it probably isn’t really you. People would use this type of Proxy Server for partial anonymity to hide their real IP address, but don’t want or need to hide that they’re doing it. A bit like being “ex-directory” in the phone book. Certain websites may then choose to block you anyway (or a devious website owner may choose to serve you up some other special content to trick you eg fake prices!).
  3. Distorting: These are a variation on Anonymous Proxy Servers in that they still alert target websites that you’re using a Proxy Server, but instead of hiding your IP address completely, will switch it for a fake one.
  4. High Anonymity or “Elite” Proxy Servers: They do not identify your traffic as having been forwarded via a Proxy Servers, the requests look just like regular internet requests direct from your computer. But your IP address is replaced with that of the Proxy Server. So target websites have no idea that you’re coming via a Proxy Server, nor any idea of your real IP address and identity. This is the type most suitable for SEO’s.

Back to top

Are there any risks with using Proxy servers?

The most common type of Proxy Server that people search for and try using are free Proxy Servers. They’re also known as “open” or “public” Proxy Servers. “Proxy Server Lists” is a popular search term for finding them. People are used to so many things on the Internet being free, eg Gmail, Facebook, photo sharing etc, that they assume free Proxy Servers are no problem either. But using free/open/public Proxy Servers is a disaster for SEO Agencies and should be avoided at all costs. This is because nobody in their right minds installs a Proxy Server on their computer and lets any stranger from anywhere get access to itand use all their bandwidth. 99.99% of free Proxy Servers are in fact either:

  • Private Proxy Servers that a Network Administrator has incorrectly configured and accidentally made it public. As they’re “public”, if you’ve found them, so has everyone else, and within days if not hours, they will be overwhelmed and performance will become painfully slow until the Proxy Server crashes altogether
  • Or, far more likely, is a computer that has been hacked with a virus or Trojan and has been turned into an open, public Proxy Server, without the owner’s knowledge or consent, and may have become part of a hacker’s “botnet”. Eventually the owner will realise something is up and their anti-virus or anti-malware software will remove the Proxy Server. 

In either case, the Proxy Server is very unreliable and will eventually stop working and need to be replaced – you find yourself on a constant “cat and mouse” chasing after new working Proxy Servers – along with hundreds of thousands of other people.

But there are worse problems. Your computer, network and data is at a huge security risk. Free open/public Proxy Servers were created by all kinds of people many with bad intentions. They are being used by hundreds or thousands of other random strangers, many of whom are using them for blackhat activity which you could be accidentally associated with. Your data is at high risk of interception, and the connection you just opened to a highly compromised computer could in itself inadvertently open a security hole into your organisation. Worse still, hackers are known to set up “honey pots”. Knowing that large numbers of people are searching for them, they set up “public” Proxy Servers, and actively monitor them, to see what kinds of interesting information they can pick up and exploit.

Therefore, you should only use Private, paid for Proxy Servers from a reputable and trustworthy Proxy Service provider. This is NOT like email services. Everyone knows you can pay for email, but why bother when the free services like Hotmail and Gmail are almost as good if not better. But 99.99% of free Proxy Servers could be very bad for your Corporate health and should be avoided at all costs!

 Any additional expert tips?

  • If you need to use Proxy Servers for your SEO Agency and you come to rely on them for your business, you should seek out a reputable, reliable, Enterprise Class Proxy Server supplier. Like any other service provider for your business, you need to know they’re reliable, have a fast infrastructure, have knowledgeable Sales and Support people you can speak to anytime, and are entirely safe to use.
  • Price absolutely should not be your determining factor. As we have seen, free is a disaster. And very cheap providers are often just harvesting dangerous free Proxy Servers from the Internet, packaging them up and selling them on. A private Proxy Server provider that advertises that they will frequently replace your Proxy Servers should you get blocked is another warning sign. That’s because real, fast, reliable Proxy Servers cost real money to run, so you have to question how a low cost provider could be replacing so many of them so quickly. Ironically what looks like a higher level of service may be an indicator of potential trouble! You should only use a provider where you can verify they use their own private network of servers in high quality Data Centers.
  • For best results, you may also prefer a provider who has servers in many different locations and large numbers of unique A, B and C Classes for their IP addresses, to avoid detection by not just your own traffic on sequential or closely related IP addresses, but insulation against what others may be doing in the same IP address “neighbourhood”. The cheaper a provider is, and the less fussy they are about who they have as clients, the more you need to worry about what others may be doing that could affect you, even if the Proxy Servers you buy are private. Eg Google spammers use cheaper providers, and if Google gets ticked off enough, they blacklist entire blocks of IP addresses, and your Proxy Server could be one of those “tarred with the same brush”.
  • SEOs with overseas clients may want to use a Proxy Server provider that has Proxy Servers in many worldwide locations.

What is a VPN?

VPN stands for “Virtual Private Network”. Their main use is typically to connect remote workers to their office’s central server, over a secure encrypted link so that all communications are kept secure. As this will have the effect of changing your IP address, companies have sprung up offering private VPN services. You connect to their VPN server, your IP address is changed to theirs, and then forwards on your web traffic, so the result is similar to that of a Proxy Server.

Back to top

How does a VPN differ from a Proxy server?

The obvious difference is that all your traffic between you and the VPN server is encrypted. But this is not as useful as it sounds, because if you’re surfing to a regular, non-secure website, the traffic between the VPN server and the target website is not encrypted so you didn’t gain that much. The one big difference and benefit you do get is that all traffic from your computer is automatically sent via the VPN server, not just that sent by your browser or special software. This can be useful for certain websites, typically flash or video heavy sites, that don’t work properly over HTTP only Proxy Servers.

Back to top

Which is better, VPN or Proxy?

At first sight VPN’s sound better as they disguise your originating IP address for all your internet traffic. But there are downsides:

  • Configuring your computer to use a VPN can be a bit technical or require downloading and installing special software, with all the potential Support headaches that entails
  • A VPN will be a bit slower than a comparable Proxy Server due to the extra processing required to initially connect and then encrypt and de-encrypt all the traffic
  • You may not want to send all of your traffic via the VPN
  • You can only use one VPN connection at a time. This may be fine for many applications, but if you need to use multiple connections, eg the examples above for using multiple Social Media accounts or speeding up keyword ranking reports, it won’t work so smoothly.

Hence my recommendation is if a regular Proxy Server will do the job you need doing, stick with that and don’t use a VPN.


Other great resources for SEO proxy server education:

Thanks for reading,  whoever you are ;)

Dave Sottimano

Dave Sottimano

David Sottimano comes from a varied background in Corporate Marketing and Professional Sales. His love affair between the internet and marketing has finally found the perfect balance at Distilled, and continues to flourish each day. He graduated...   read more

Get blog posts via email

23 Comments

  1. Nice insights David. How about proxy use free private web proxy. Whether the IP address is trackable by log server?

    reply >
    • David Sottimano

      Could you clarify the question a bit for me? Are you asking if the free proxy server will log your IP address when you connect to it? If so, then yes it usually will.

  2. Thanks, David! The article is extremely useful and worth saving to refer to it from time to time.

    reply >
  3. Thanks David, really enjoyed that - something I have only vaguely known about and now feel much more knowledgeable!

    I also think it would be great if Distilled U had more content like this, as it still remains fairly basic.

    Cheers,
    Jamie

    reply >
  4. galileo

    This article is a nice start in educating people about proxies.
    There are a few things that your readers should be aware of.

    There is no such thing as an elite proxy that can protect your privacy completely. Web sites can still find out your real ip using javascript, flash, or java.

    VPN solution is much better. In theory it is slower but the difference is in the range of 3-5% and that's due to overhead in the packet headers that the tunnel protocol uses not the encryption.

    Configuring the VPN is actually easy in windows vista/7 etc. as the support is in the OS. Multiple VPN providers and tunnels can be configured on a single computer and enabled/disabled with a mouse click, thus gaining the different ip/country location for each connection. Traffic between the vpn and the target site is not encrypted so you gain nothing? Traffic between your regular connection and the target site is also not encrypted so you lose nothing.

    “honey pots” are usually set up by security researches who are targeting hackers, not other way around.

    I do realize that this article is an advertisement for the service offered but if you really want to educate your users you should get someone who is an expert in computer security or at least a sys/net admin.

    reply >
    • Thanks galileo for your comments. The purpose of this article, as implied by the title, is “Essential info for SEO’s”.

      The balance that David was looking for was to distill a lot of technical information down into easy to understand actionable points that a non-technical SEO could use.

      David could indeed have spoken to a Sys Admin and could have gone into many more layers of technical info on how Proxy Servers and VPN’s work. But that would go over the heads of many readers of this blog.

      With one exception I don’t disagree with anything you have said. But what David was looking for when he spoke to me (and this isn’t an advert BTW) was a simplified understanding of the main points and how they are useful to SEO’s. I have a lot of experience with David’s audience. And I can tell you from that experience, for example that a lot of non-technical people do not find it as easy to set up a VPN themselves, compared to configuring their software to use a Proxy Server, and that in some SEO applications, a VPN just does not work as well.

      The one thing I disagree with you on and want to flag to readers is about “honey pots”. I don’t think you can discount the possibility that hackers do set them up. And while it is known that researchers also set them up to catch hackers, the question remains whether a Corporation, especially one that handles confidential data on behalf of their clients, wants to risk their data falling into the hands and being examined by any unknown or untrusted third party? Hence my warning to readers to steer clear of Public Proxy Servers.

  5. Thank you for posting this wonderful article. This was indeed educative for non-technical people like us.

    I know this may sound like a stupid question but I want to confirm that when we access search results via the proxy server Google will use that particular server's IP address to give us results.
    Our experience shows that even by setting your location to another city (in a web browser) the search results (non-personalized) are actually different from what actually is seen in at another location in the country.
    Does this mean that we need to take proxy servers all over the country?

    reply >
    • David Sottimano

      Well, thank you for reading it. Google is using IP location to "localize" results for you, and there's nothing we can do about it.

      It's very difficult to see a common ranking between, you, the client and the client's visitors. As you've noticed, results can change based on city, region and country. You can thank Google Venice for that.

      Don't get me wrong, if I had the cash I would prefer to have every ranking for every keyword in every city - but it's just not feasible, or smart. Does it mean rank tracking is dead? Hell no...

      Rank tracking from any location will show you SERP fluctuation, so if you're running a link building campaign, how else do you measure SEO efforts? Your core SEO KPIs should be a combination of rank position for target keywords and increase of organic traffic to said pages.

      Rank tracking from any location will always be a good measure of your SEO efforts, just don't put too much stock into exact positions - because they are volatile. If you built links to a page that was ranking 10th and it moved up - then you confirm that whatever you did worked.

      Does that help?

  6. I had no idea about proxies before i read about it in here

    reply >
  7. Its does David, Thanks for the response.

    reply >
  8. Proxy will be helpful when there are multiple clients doing same business and hired a same company for link building, then it should be better to work using proxies rather than working on same ip for each clients' website.

    reply >
  9. Shyama

    Our company is going to place our cooprate proxy in China ( our service providers data centre) . All companies in other countries must go through this proxy server. Is this a good idea?. What currently this proxy server is located in US. What implication could this bring to our users.

    Thanks for any feedback

    reply >
  10. Hey David,

    A lot of people that don't understand that SEO companies need to use proxies to help their clients rank higher on Google. I was wondering what you take on building links per IP would look natural? This is one thing I am researching as I going to be building lots of links for highly competitive keywords.

    Garen

    reply >
  11. Thank you for this post Dave. I am currently testing out a VPN and proxies and your detailed post indeed has help clarify some of my nagging questions.

    Cheers
    tony

    reply >
  12. Do you have any video of that? I'd care to find out some additional information.

    reply >
  13. Hi Dave, Thanks for the detailed guide. This has helped me a lot to learn the inns and outs of proxy servers. Do you have any idea if I buy from Trusted Proxies, can I get all private proxy servers from USA? or they will be based on several countries. Thanks for your reply.

    reply >
    • Dave Sottimano

      Hi Taswir, I'm pretty sure they can accommodate you :) Drop them an email and they'll guide you through whatever setup you need.

  14. I really like the AWR software and they recommend those guys at Trusted Proxies so I will give them a try since it really doesn't work great without using proxy. I keep getting 403 even using the settings to be nice to search engines, and take breaks, etc. Thanks for the article.

    reply >
  15. Need to setup private proxy servers in Mexican datacenters or my datacenters, any recomendation of someone that can sell us rack severs with the setup in them?

    thanks.

    reply >
    • Dave Sottimano

      That's out of my league, I'm sure Paul and the Trusted proxies team might be able to advise better than I can. Try emailing them :)

  16. Thank you Dave for this detailed post. From now I will not use free proxies.

    reply >

Leave a Reply

Your email address will not be published. Required fields are marked *

*
*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>