CloudScraper Proxy Configuration

← Back to Python Libraries

CloudScraper is a Python library that automatically bypasses Cloudflare's anti-bot protection, including JavaScript challenges, CAPTCHAs, and browser verification. Built on top of Requests, CloudScraper solves Cloudflare challenges automatically.

Combined with ProxyMesh rotating proxies, CloudScraper lets you access Cloudflare-protected sites through different IP addresses, reducing blocks and maintaining high success rates for data collection.

Why CloudScraper + Rotating Proxies?

Cloudflare-protected sites implement multiple layers of protection:

  • JavaScript challenges - CloudScraper solves these automatically
  • IP reputation - Rotating proxies provide fresh IPs
  • Rate limiting - IP rotation distributes requests
  • Geographic restrictions - Access content from specific countries

Installation

Install CloudScraper using pip:

pip install cloudscraper

CloudScraper extends Requests, so it inherits full proxy support—no additional packages needed.

Basic Proxy Configuration

CloudScraper is a drop-in replacement for Requests with automatic Cloudflare bypass:

import cloudscraper

scraper = cloudscraper.create_scraper()

response = scraper.get(
    "https://cloudflare-protected-site.com",
    proxies={
        "http": "http://username:password@PROXYHOST:PORT",
        "https": "http://username:password@PROXYHOST:PORT"
    }
)

print(f"Status: {response.status_code}")
print(response.text[:500])

CloudScraper automatically handles JavaScript challenges while routing through your rotating proxy.

Session-Level Proxy

import cloudscraper

scraper = cloudscraper.create_scraper()
scraper.proxies = {
    "http": "http://user:pass@PROXYHOST:PORT",
    "https": "http://user:pass@PROXYHOST:PORT"
}

# All requests use the proxy
response1 = scraper.get("https://protected-site.com/page1")
response2 = scraper.get("https://protected-site.com/page2")

Browser Emulation

import cloudscraper

scraper = cloudscraper.create_scraper(
    browser={
        "browser": "chrome",
        "platform": "windows",
        "desktop": True
    }
)

scraper.proxies = {"https": "http://user:pass@PROXYHOST:PORT"}

response = scraper.get("https://protected-site.com")

IP Authentication

import cloudscraper

scraper = cloudscraper.create_scraper()
scraper.proxies = {"https": "http://PROXYHOST:PORT"}

response = scraper.get("https://protected-site.com")

Custom Proxy Headers

CloudScraper is built on Requests, so you can use the python-proxy-headers extension for custom proxy headers:

pip install python-proxy-headers
import cloudscraper
from python_proxy_headers.requests_adapter import HTTPProxyHeaderAdapter

scraper = cloudscraper.create_scraper()

adapter = HTTPProxyHeaderAdapter(proxy_headers={"X-ProxyMesh-Country": "US"})
scraper.mount("http://", adapter)
scraper.mount("https://", adapter)

scraper.proxies = {"https": "http://user:pass@PROXYHOST:PORT"}

response = scraper.get("https://protected-site.com")
print(f"Routed through: {response.headers.get('X-ProxyMesh-IP')}")

See the python-proxy-headers documentation for more details.

Common Use Cases

Scraping Protected E-commerce Sites

import cloudscraper
from bs4 import BeautifulSoup

scraper = cloudscraper.create_scraper()
scraper.proxies = {"https": "http://user:pass@PROXYHOST:PORT"}

response = scraper.get("https://cloudflare-protected-shop.com/products")
soup = BeautifulSoup(response.text, "html.parser")

for product in soup.select("div.product"):
    name = product.select_one("h2").text
    price = product.select_one(".price").text
    print(f"{name}: {price}")

Rotating Proxy Locations

import cloudscraper
import random

PROXIES = [
    "http://user:pass@PROXYHOST:PORT",
    "http://user:pass@PROXYHOST:PORT",
    "http://user:pass@PROXYHOST:PORT",
]

def get_scraper():
    scraper = cloudscraper.create_scraper()
    scraper.proxies = {"https": random.choice(PROXIES)}
    return scraper

# Scrape multiple pages with rotation
urls = [f"https://protected-site.com/page/{i}" for i in range(20)]

for url in urls:
    scraper = get_scraper()
    response = scraper.get(url)
    print(f"{url}: {response.status_code}")

Handling Challenge Types

import cloudscraper

scraper = cloudscraper.create_scraper(
    browser={
        "browser": "chrome",
        "platform": "linux",
        "mobile": False
    },
    delay=10  # Wait time for JS challenges
)

scraper.proxies = {"https": "http://user:pass@PROXYHOST:PORT"}

try:
    response = scraper.get("https://heavily-protected-site.com")
    print(f"Success: {response.status_code}")
except cloudscraper.exceptions.CloudflareChallengeError as e:
    print(f"Challenge failed: {e}")
except cloudscraper.exceptions.CloudflareCaptchaError as e:
    print(f"CAPTCHA required: {e}")

Retry with Different IP on Block

import cloudscraper
import random

PROXIES = [
    "http://user:pass@PROXYHOST:PORT",
    "http://user:pass@PROXYHOST:PORT",
    "http://user:pass@PROXYHOST:PORT",
]

def scrape_with_retry(url, max_retries=3):
    for attempt in range(max_retries):
        scraper = cloudscraper.create_scraper()
        scraper.proxies = {"https": random.choice(PROXIES)}
        
        try:
            response = scraper.get(url, timeout=30)
            
            if response.status_code == 200:
                return response
            elif response.status_code == 403:
                print(f"Blocked, retrying with different IP...")
                continue
                
        except Exception as e:
            print(f"Attempt {attempt + 1} failed: {e}")
    
    return None

response = scrape_with_retry("https://protected-site.com")
if response:
    print(f"Success: {response.status_code}")

POST Requests Through Protected Sites

import cloudscraper

scraper = cloudscraper.create_scraper()
scraper.proxies = {"https": "http://user:pass@PROXYHOST:PORT"}

# First get the page to solve any challenges
scraper.get("https://protected-site.com/login")

# Then POST with session cookies
response = scraper.post(
    "https://protected-site.com/login",
    data={"username": "user", "password": "pass"}
)

Geographic Targeting

import cloudscraper

LOCATIONS = {
    "us": "PROXYHOST:PORT",
    "uk": "PROXYHOST:PORT",
    "de": "PROXYHOST:PORT",
}

for location, proxy_host in LOCATIONS.items():
    scraper = cloudscraper.create_scraper()
    scraper.proxies = {"https": f"http://user:pass@{proxy_host}"}
    
    response = scraper.get("https://geo-restricted-site.com")
    print(f"{location}: {len(response.text)} bytes")

ProxyMesh Headers Reference

Send these headers to control proxy behavior:

  • X-ProxyMesh-Country - Route through a specific country (e.g., "US"). Only works with world proxy or open proxy
  • X-ProxyMesh-IP - Request a specific outgoing IP address
  • X-ProxyMesh-Not-IP - Exclude specific IPs from rotation

The proxy returns X-ProxyMesh-IP in the response with the IP address used.

Resources

Related Python Proxy Guides

Explore proxy configuration for other Python HTTP libraries:

  • Requests - The library CloudScraper is built on
  • Scrapy - Full web crawling framework
  • AutoScraper - Automatic ML-based scraping

Start Free Trial