Requests Proxy Configuration

← Back to Python Libraries

The Requests library is the most popular Python HTTP client, known for its simple and elegant API. With over 50 million downloads per month, Requests is the go-to choice for making HTTP requests in Python.

Requests has built-in proxy support, making it easy to route traffic through ProxyMesh rotating proxy servers. This guide shows you how to configure proxies for web scraping, data collection, and anonymous requests.

Why Use Proxies with Requests?

When building web scrapers or data collection scripts, you'll often encounter:

  • IP blocking - Websites block your IP after too many requests
  • Rate limiting - APIs restrict requests per IP address
  • Geo-restrictions - Content varies by geographic location
  • Privacy requirements - Need to mask your origin IP

ProxyMesh rotating proxies solve these problems by automatically cycling through different IP addresses.

Installation

Install Requests using pip:

pip install requests

Requests has built-in proxy support—no additional packages needed to get started.

Basic Proxy Configuration

Requests supports proxies through the proxies parameter. Specify proxy URLs for HTTP and HTTPS traffic:

import requests

# ProxyMesh proxy with username/password authentication
proxy_url = "http://username:password@PROXYHOST:PORT"

response = requests.get(
    "https://api.ipify.org?format=json",
    proxies={"http": proxy_url, "https": proxy_url}
)

print(response.json())  # {"ip": "..."}

Each request through ProxyMesh may use a different IP address from the rotation pool.

Session-Level Proxy Configuration

For multiple requests, configure proxies at the session level for better performance with connection pooling:

import requests

session = requests.Session()
session.proxies = {
    "http": "http://username:password@PROXYHOST:PORT",
    "https": "http://username:password@PROXYHOST:PORT"
}

# All requests share connections and use the proxy
response1 = session.get("https://example.com/page1")
response2 = session.get("https://example.com/page2")

Environment Variable Configuration

Requests automatically reads proxy settings from environment variables:

export HTTP_PROXY="http://username:password@PROXYHOST:PORT"
export HTTPS_PROXY="http://username:password@PROXYHOST:PORT"

import requests

# Automatically uses proxies from environment
response = requests.get("https://api.ipify.org?format=json")

This keeps proxy credentials out of your code—useful for production deployments.

IP Authentication

If you've whitelisted your server's IP in the ProxyMesh dashboard, no credentials are needed:

import requests

# No username/password required with IP authentication
response = requests.get(
    "https://api.ipify.org?format=json",
    proxies={"https": "http://PROXYHOST:PORT"}
)

Custom Proxy Headers

For basic proxy usage, no custom headers are needed—ProxyMesh automatically rotates IPs. For advanced control, you can send headers like X-ProxyMesh-Country to select a specific country.

Sending custom headers to the proxy over HTTPS requires the python-proxy-headers extension:

pip install python-proxy-headers
from python_proxy_headers import requests_adapter

response = requests_adapter.get(
    "https://api.ipify.org?format=json",
    proxies={"https": "http://user:pass@PROXYHOST:PORT"},
    proxy_headers={"X-ProxyMesh-Country": "US"}
)

# Access the IP address assigned by ProxyMesh
proxy_ip = response.headers.get("X-ProxyMesh-IP")
print(f"Routed through: {proxy_ip}")

Available proxy headers:

  • X-ProxyMesh-Country - Route through a specific country (use with world proxy)
  • X-ProxyMesh-IP - Request a specific IP for session consistency
  • X-ProxyMesh-Not-IP - Exclude specific IPs from rotation

See the python-proxy-headers documentation for more details.

Common Use Cases

Web Scraping with Rotating IPs

import requests
from bs4 import BeautifulSoup

session = requests.Session()
session.proxies = {"https": "http://user:pass@PROXYHOST:PORT"}

urls = [
    "https://example.com/page/1",
    "https://example.com/page/2",
    "https://example.com/page/3",
]

for url in urls:
    response = session.get(url)
    soup = BeautifulSoup(response.text, "html.parser")
    title = soup.find("title").text
    print(f"{url}: {title}")

POST Requests Through Proxy

import requests

proxy = {"https": "http://user:pass@PROXYHOST:PORT"}

# POST with JSON data
response = requests.post(
    "https://httpbin.org/post",
    json={"key": "value"},
    proxies=proxy
)

print(response.json())

Handling Timeouts and Retries

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
session.proxies = {"https": "http://user:pass@PROXYHOST:PORT"}

# Configure retries
retry = Retry(total=3, backoff_factor=1, status_forcelist=[429, 500, 502, 503])
adapter = HTTPAdapter(max_retries=retry)
session.mount("https://", adapter)

try:
    response = session.get("https://example.com", timeout=30)
    response.raise_for_status()
    print(response.text[:200])
except requests.exceptions.RequestException as e:
    print(f"Request failed: {e}")

Different Proxy Locations

Connect to different ProxyMesh locations for geographic targeting:

import requests

PROXY_LOCATIONS = {
    "us": "PROXYHOST:PORT",
    "uk": "PROXYHOST:PORT", 
    "de": "PROXYHOST:PORT",
    "jp": "PROXYHOST:PORT",
}

def fetch_from_location(url, location):
    proxy_host = PROXY_LOCATIONS[location]
    proxy = {"https": f"http://user:pass@{proxy_host}"}
    return requests.get(url, proxies=proxy)

# Fetch from different regions
us_response = fetch_from_location("https://api.ipify.org", "us")
uk_response = fetch_from_location("https://api.ipify.org", "uk")

print(f"US IP: {us_response.text}")
print(f"UK IP: {uk_response.text}")

Download Files Through Proxy

import requests

proxy = {"https": "http://user:pass@PROXYHOST:PORT"}

response = requests.get(
    "https://example.com/file.pdf",
    proxies=proxy,
    stream=True
)

with open("downloaded.pdf", "wb") as f:
    for chunk in response.iter_content(chunk_size=8192):
        f.write(chunk)

ProxyMesh Headers Reference

Send these headers to control proxy behavior:

  • X-ProxyMesh-Country - Route through a specific country (e.g., "US"). Only works with world proxy or open proxy
  • X-ProxyMesh-IP - Request a specific outgoing IP address
  • X-ProxyMesh-Not-IP - Exclude specific IPs from rotation

The proxy returns X-ProxyMesh-IP in the response with the IP address used.

Resources

Related Python Proxy Guides

Explore proxy configuration for other Python HTTP libraries:

  • httpx - Modern HTTP client with async support and HTTP/2
  • aiohttp - High-performance async HTTP for concurrent scraping
  • Scrapy - Full web crawling framework

Start Free Trial