Blog/Tutorial

How to Rotate Proxies in Python

Jun 18, 2026·10 min read

Proxy rotation is the practice of distributing requests across multiple IP addresses to avoid rate limits, IP bans, and bot detection. This guide covers every rotation pattern you'll need — from basic list rotation to residential gateway rotation with retry logic — with working Python code you can drop into your scraper today.

Prerequisites

Install the required packages:

bash

pip install requests httpx[http2] scrapy

This guide covers three Python HTTP clients: requests (synchronous), httpx (async + sync), and Scrapy (framework).

Method 1: Rotating residential proxy gateway

The simplest and most reliable approach is using a residential proxy provider that handles rotation at the gateway level. You send all requests through a single endpoint, and the provider assigns a fresh residential IP to each request automatically.

With Proxies.VISION, your credentials look like this:

python

import requests

# Rotating session — new IP on every request
PROXY_HOST = "connect.proxies.vision"
PROXY_PORT = 8080
PROXY_USER = "your_username"
PROXY_PASS = "your_password"

proxy_url = f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
proxies = {"http": proxy_url, "https": proxy_url}

response = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=30)
print(response.json())  # {"origin": "203.0.113.45"}  ← different every time

Gateway rotation is recommended over list rotation for most use cases — you get access to a pool of 42M+ residential IPs without managing a list, and the rotation is handled server-side.

Method 2: Sticky sessions (same IP per session)

For checkout flows, account operations, or any task requiring session continuity, use sticky session mode. Proxies.VISION holds the same IP for up to 30 minutes.

python

import requests

# Sticky session — same IP for up to 30 minutes
PROXY_HOST = "connect.proxies.vision"
PROXY_PORT = 8080
PROXY_USER = "your_username-session-1234-ttl-30"
PROXY_PASS = "your_password"

proxy_url = f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
proxies = {"http": proxy_url, "https": proxy_url}

session = requests.Session()
session.proxies = proxies

# All requests in this session use the same residential IP
r1 = session.get("https://example.com/login")
r2 = session.post("https://example.com/cart", json={"item_id": "123"})
r3 = session.post("https://example.com/checkout")

Method 3: List rotation with requests

If you have a static proxy list, here's a production-grade rotation class with retry logic, failure tracking, and backoff:

python

import requests
import random
import time
from typing import Optional

class ProxyRotator:
    def __init__(self, proxy_list: list[str], max_retries: int = 3):
        self.proxies = proxy_list
        self.failed: set[str] = set()
        self.max_retries = max_retries

    def _get_proxy(self) -> Optional[str]:
        available = [p for p in self.proxies if p not in self.failed]
        if not available:
            self.failed.clear()  # Reset on full exhaustion
            available = self.proxies
        return random.choice(available) if available else None

    def get(self, url: str, **kwargs) -> requests.Response:
        for attempt in range(self.max_retries):
            proxy = self._get_proxy()
            if not proxy:
                raise RuntimeError("No proxies available")

            proxy_dict = {"http": proxy, "https": proxy}
            try:
                response = requests.get(url, proxies=proxy_dict, timeout=30, **kwargs)
                if response.status_code == 429:
                    self.failed.add(proxy)
                    time.sleep(2 ** attempt)
                    continue
                return response
            except (requests.exceptions.ProxyError,
                    requests.exceptions.ConnectTimeout):
                self.failed.add(proxy)
                continue

        raise RuntimeError(f"All {self.max_retries} attempts failed for {url}")


# Usage
proxies = [
    "http://user:[email protected]:7777",
    "http://user:[email protected]:7777",
    "http://user:[email protected]:7777",
]

rotator = ProxyRotator(proxies)
response = rotator.get("https://httpbin.org/ip")

Method 4: Async rotation with httpx

For high-throughput scrapers, async execution with httpx dramatically improves speed. Here's a concurrent scraper with rotating residential proxies:

python

import asyncio
import httpx

PROXY_URL = "http://USERNAME:[email protected]:8080"

async def fetch(client: httpx.AsyncClient, url: str) -> dict:
    try:
        response = await client.get(url, timeout=30)
        return {"url": url, "status": response.status_code, "data": response.text[:200]}
    except Exception as e:
        return {"url": url, "error": str(e)}

async def scrape_batch(urls: list[str], concurrency: int = 10):
    async with httpx.AsyncClient(
        proxy=PROXY_URL,
        follow_redirects=True,
        headers={
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
            "Accept-Language": "en-US,en;q=0.9",
        }
    ) as client:
        semaphore = asyncio.Semaphore(concurrency)

        async def fetch_with_limit(url):
            async with semaphore:
                return await fetch(client, url)

        results = await asyncio.gather(*[fetch_with_limit(u) for u in urls])
    return results

# Run
urls = [f"https://httpbin.org/ip?req={i}" for i in range(50)]
results = asyncio.run(scrape_batch(urls, concurrency=10))
print(f"Completed {len(results)} requests")

Method 5: Scrapy middleware

For Scrapy spiders, the cleanest approach is a custom downloader middleware that injects proxy settings per request:

python

# middlewares.py
import random

class RotatingProxyMiddleware:
    """Rotate through a list of proxies on each request."""

    PROXIES = [
        "http://user:[email protected]:8080",
        # Add more if using multiple accounts / endpoints
    ]

    def process_request(self, request, spider):
        proxy = random.choice(self.PROXIES)
        request.meta["proxy"] = proxy

    def process_exception(self, request, exception, spider):
        # On proxy error, try a different proxy
        proxy = random.choice(self.PROXIES)
        request.meta["proxy"] = proxy
        return request  # Retry with new proxy


# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RotatingProxyMiddleware": 350,
    "scrapy.downloadermiddlewares.retry.RetryMiddleware": 550,
}

RETRY_HTTP_CODES = [429, 403, 500, 502, 503, 504]
RETRY_TIMES = 3
DOWNLOAD_TIMEOUT = 30

Rotation strategy guidelines

Per-request rotation for large-scale scraping

Use rotating sessions when you're collecting thousands of pages with no session dependency. Gateway-level rotation (one endpoint, automatic IP cycling) is more reliable than managing a static list.

Per-session rotation for account operations

Group all requests in a logical session (login → browse → action) under a single sticky IP. Change IP between sessions, not within them.

Respect rate limits even with rotation

Rotating IPs doesn't make you invisible. Add 1–3 second delays between requests, randomize user agents, and don't send more than ~100 requests/minute per endpoint.

Handle 429s explicitly

A 429 response means you've hit a rate limit on that IP. Mark it as temporarily unavailable, wait with exponential backoff, and try a different IP.

Ready to run this code?

Get your proxy credentials in 2 minutes.

Get credentials →View pricing

← All posts