ProxyMesh Blog

Reliable Open Proxies

03 March 2013

ProxyMesh offers an easy way to make use of open proxies. The typical way of using open proxies involves compiling a long list of IPs that must be updated at least once a day, and using software that can automatically switch between these IPs. These open proxy IPs must also be regularly checked to ensure they're still in operation, and are not inserting malicious code into your requests. ProxyMesh eliminates these requirements. All you have to do is configure your client to access the ProxyMesh open proxy server, and ProxyMesh takes care of the rest. To learn more & signup, go to the ProxyMesh Pricing Page. And for details on how this open proxy server works, read on...

Open Proxy IPs

The ProxyMesh open proxy server maintains a list of known open proxy IPs, which are used by the proxy server to forward your requests. This works just like the US & UK proxies, except instead of going thru low-latency elite proxies, your requests are forwarded thru less reliable & higher latency open proxies. While a large percentage of these proxies are in the US, over half are located in many different countries around the world. These proxies typically do not stay online very long, and are not operated on reliable infrastructure. The tradeoff for this lack of reliability is a huge increase in quantity & variability of IP addresses. The ProxyMesh open proxies list typically countains at least 1000 IPs at a time, with approximately 100 IPs changing every hour.

Realtime Reliability

Because open proxies are not reliable, real-time error checking is required to provide a consistent service. If any request thru an open proxy fails due to a proxy error, that error is recorded, and the request is re-tried up to 3 more times, using a different proxy each time. Any proxy that gets 3 or more errors will immediately be removed from the list.

Periodic Updates

To keep a fresh open proxy list, the proxy list is re-checked every 15 minutes, and any proxies that fail these checks are removed. These checks test the following:

  1. a valid request can be sent thru the open proxy
  2. a valid response is received within 3 seconds
  3. that response has not been corrupted and does not contain malicious code
  4. the open proxy does not have a known abusive IP address

Over 95% of open proxies fail these tests. Therefore, the ProxyMesh open proxy server only keeps the 5% of open proxies that are actually usable.

Retry Strategy

Even with periodic proxy checks & realtime removal of unreliable proxies, you must have a retry strategy when using this proxy server. Open proxies are, by their nature, unreliable & error prone. They could go down at any time, be configured incorrectly, or they may not honor your specific request. For these reasons, you should retry all 40x & 50x response codes at least 3 times, and optionally make use of the custom ProxyMesh headers. The way this could work is:

  1. make a request thru the open proxy server
  2. get a 40x or 50x response error
  3. from the response, extract the IP from the X-ProxyMesh-IP header
  4. retry the request with a X-ProxyMesh-Not-IP header containing the IP

The X-ProxyMesh-Not-IP header can take a comma list of IP addresses, so you can accumulate bad IPs to skip for future requests. If you do this, it is recommended to cache the IPs for a maximum of 1 day, as they will likely be out-of-date or offline after 24hrs.

Elite vs Open Proxies

The ProxyMesh open proxies server provides a very different set of tradeoffs compared to the US & UK proxy servers. With the open proxy server, you get quantity & variability at the expense of speed & reliability. And the US & UK proxies are what's known as elite proxies, in that they remove all identifying information from requests, whereas the open proxies provide no guarantee of anonymity. So if you need speed, reliability, and anonymity, then the US & UK proxies are the best choice. But if you want a lot of IPs, the open proxy server might be an acceptable tradeoff.