urllib3 Proxy Configuration

urllib3 is a powerful, low-level HTTP client library for Python that provides connection pooling, thread safety, and retry logic. It's the foundation that powers the popular Requests library.

urllib3 has built-in proxy support through ProxyManager, giving you fine-grained control over HTTP connections through ProxyMesh rotating proxies.

Why Use urllib3 Directly?

Lower overhead - Direct access without Requests' session abstraction
Connection pooling control - Fine-tune pool sizes and reuse
Thread safety - Built-in support for concurrent access
Retry configuration - Granular control over retry behavior
Foundation library - Understand what powers Requests

Installation

Install urllib3 using pip:

pip install urllib3

urllib3 has built-in proxy support—no additional packages needed to get started.

Basic Proxy Configuration

urllib3 provides ProxyManager for routing requests through a proxy with connection pooling:

import urllib3

# Create proxy manager with authentication
proxy = urllib3.ProxyManager(
    "http://username:password@PROXYHOST:PORT"
)

response = proxy.request("GET", "https://api.ipify.org?format=json")
print(response.data.decode())  # {"ip": "..."}

The ProxyManager handles connection pooling automatically for efficient multiple requests.

Connection Pool Configuration

import urllib3

proxy = urllib3.ProxyManager(
    "http://user:pass@PROXYHOST:PORT",
    num_pools=10,    # Number of connection pools
    maxsize=10,      # Max connections per pool
)

# Efficiently make many requests
for i in range(100):
    response = proxy.request("GET", f"https://api.example.com/item/{i}")
    print(f"Item {i}: {response.status}")

Retry Configuration

import urllib3

retry = urllib3.Retry(
    total=3,
    backoff_factor=0.5,
    status_forcelist=[429, 500, 502, 503]
)

proxy = urllib3.ProxyManager(
    "http://user:pass@PROXYHOST:PORT",
    retries=retry
)

response = proxy.request("GET", "https://api.example.com/data")

IP Authentication

import urllib3

# No credentials needed with IP whitelisting
proxy = urllib3.ProxyManager("http://PROXYHOST:PORT")
response = proxy.request("GET", "https://api.ipify.org")

Custom Proxy Headers

urllib3 supports sending headers to the proxy with proxy_headers:

import urllib3

proxy = urllib3.ProxyManager(
    "http://user:pass@PROXYHOST:PORT",
    proxy_headers={"X-ProxyMesh-Country": "US"}
)

response = proxy.request("GET", "https://api.ipify.org?format=json")
print(response.data.decode())

To also receive proxy response headers (like X-ProxyMesh-IP), use the python-proxy-headers extension:

pip install python-proxy-headers

from python_proxy_headers import urllib3_proxy_manager

proxy = urllib3_proxy_manager.ProxyHeaderManager(
    "http://user:pass@PROXYHOST:PORT",
    proxy_headers={"X-ProxyMesh-Country": "US"}
)

response = proxy.request("GET", "https://api.ipify.org?format=json")
print(f"Routed through: {response.headers.get('X-ProxyMesh-IP')}")

See the python-proxy-headers documentation for more details.

Common Use Cases

Thread-Safe Concurrent Requests

import urllib3
import concurrent.futures

proxy = urllib3.ProxyManager(
    "http://user:pass@PROXYHOST:PORT",
    num_pools=10,
    maxsize=20
)

def fetch_url(url):
    response = proxy.request("GET", url, timeout=30)
    return {"url": url, "status": response.status}

urls = [f"https://httpbin.org/anything/{i}" for i in range(50)]

with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    results = list(executor.map(fetch_url, urls))

print(f"Fetched {len(results)} URLs")

POST Requests with JSON

import urllib3
import json

proxy = urllib3.ProxyManager("http://user:pass@PROXYHOST:PORT")

response = proxy.request(
    "POST",
    "https://httpbin.org/post",
    body=json.dumps({"key": "value"}),
    headers={"Content-Type": "application/json"}
)

print(json.loads(response.data.decode()))

Timeout Configuration

import urllib3

timeout = urllib3.Timeout(connect=5.0, read=30.0)

proxy = urllib3.ProxyManager(
    "http://user:pass@PROXYHOST:PORT",
    timeout=timeout
)

response = proxy.request("GET", "https://api.ipify.org")

Multiple Proxy Locations

import urllib3
import random

PROXIES = [
    urllib3.ProxyManager("http://user:pass@PROXYHOST:PORT"),
    urllib3.ProxyManager("http://user:pass@PROXYHOST:PORT"),
    urllib3.ProxyManager("http://user:pass@PROXYHOST:PORT"),
]

def fetch_rotating(url):
    proxy = random.choice(PROXIES)
    return proxy.request("GET", url)

response = fetch_rotating("https://api.ipify.org")
print(response.data.decode())

SSL/TLS Configuration

import urllib3
import certifi

proxy = urllib3.ProxyManager(
    "http://user:pass@PROXYHOST:PORT",
    cert_reqs="CERT_REQUIRED",
    ca_certs=certifi.where()
)

response = proxy.request("GET", "https://api.ipify.org")

Download Files

import urllib3

proxy = urllib3.ProxyManager("http://user:pass@PROXYHOST:PORT")

response = proxy.request(
    "GET",
    "https://example.com/file.pdf",
    preload_content=False
)

with open("download.pdf", "wb") as f:
    for chunk in response.stream(8192):
        f.write(chunk)

response.release_conn()

ProxyMesh Headers Reference

Send these headers to control proxy behavior:

X-ProxyMesh-Country - Route through a specific country (e.g., "US"). Only works with world proxy or open proxy
X-ProxyMesh-IP - Request a specific outgoing IP address
X-ProxyMesh-Not-IP - Exclude specific IPs from rotation

The proxy returns X-ProxyMesh-IP in the response with the IP address used.

Resources

Related Python Proxy Guides

Explore proxy configuration for other Python HTTP libraries:

Requests - Higher-level library built on urllib3
httpx - Modern HTTP client with async support
PycURL - Python bindings for libcurl

Start Free Trial