How to rotate proxies in Python, and why yours may not rotate
A rotating gateway changes the IP per connection, not per request. Most Python clients keep connections open, so they quietly reuse one IP. Here’s what rotates, measured.
In short
requests.get() rotates because it opens a new connection each time. requests.Session, httpx.Client and Scrapy keep connections open and send every request through the same exit. The fix that works with any client: a fresh session label in the proxy username for each request, sent to the sticky port.
A rotating residential gateway chooses an exit IP when a connection to it opens. For an HTTPS site, that connection becomes a tunnel: the client asks the proxy to CONNECT to the website once, then sends request after request through it. The gateway never sees those requests, so it has no chance to switch the IP.
That is fine when a client opens a new connection per request. But connection reuse is exactly what good HTTP clients are built for: it saves a TCP and TLS handshake every time. So the clients you’d pick for serious scraping are the ones that don’t rotate. We sent three requests through our gateway with each:
| Client | Exit IPs | Rotates? |
|---|---|---|
requests.get() | 51.146.216.161 · 8.47.98.240 · 82.40.78.80 | Yes |
requests.Session() | 209.94.95.37 · 209.94.95.37 · 209.94.95.37 | No |
httpx.Client() | 76.135.82.93 · 76.135.82.93 · 76.135.82.93 | No |
httpx.Client(keep-alive 0) | 142.173.227.6 · 38.97.214.33 · 100.37.193.236 | Yes |
| Scrapy | 37.133.72.35 · 37.133.72.35 · 37.133.72.35 | No |
A fresh session label per request (works everywhere)
On the sticky port (10000) the gateway pins an exit to a session label. Give each request its own label and it gets a new exit for each, whatever your client does with its connections. This is the one we recommend.
import os
import uuid
import requests
USER = os.environ["PROXY_USER"]
PASS = os.environ["PROXY_PASS"]
GATEWAY = "via.proxies.vision:10000" # the sticky port: it reads a session label
def proxies(targeting: str = "") -> dict:
"""A fresh session label per call = a fresh exit IP, even on a kept connection."""
user = f"{USER}{targeting}-session-{uuid.uuid4().hex[:8]}-lifetime-3"
url = f"http://{user}:{PASS}@{GATEWAY}"
return {"http": url, "https": url}
def get(session: requests.Session, url: str, attempts: int = 3, **kw) -> requests.Response:
"""Retry network errors: residential exits occasionally drop a connection."""
targeting = kw.pop("targeting", "")
for i in range(attempts):
try:
return session.get(url, proxies=proxies(targeting), timeout=30, **kw)
except requests.exceptions.RequestException:
if i == attempts - 1:
raise
with requests.Session() as s:
for _ in range(3):
print(get(s, "https://api.ipify.org").text) # a different IP each timeOr: a new connection per request
requests.get() without a session already does this. With httpx, turn keep-alive off: httpx.Client(proxy=url, limits=httpx.Limits(max_keepalive_connections=0)). Every request then pays for a new handshake, which is slower than the label approach.
Retry network errors
Residential exits are home connections and occasionally drop mid-request. In our runs that showed up as a ConnectTimeout, an SSLError or a RemoteProtocolError now and then. Retrying with a new label gets a healthy exit; the get() helper above does that.
Keeping one IP on purpose
For a login, a cart or anything that must look like one visitor: the same label on every request.
import os
import requests
USER = os.environ["PROXY_USER"]
PASS = os.environ["PROXY_PASS"]
# Same label = same IP for up to 10 minutes, across connections and processes.
user = f"{USER}-country-de-session-checkout1-lifetime-10"
proxy = f"http://{user}:{PASS}@via.proxies.vision:10000"
with requests.Session() as s:
s.proxies = {"http": proxy, "https": proxy}
for _ in range(3):
print(s.get("https://api.ipify.org", timeout=30).text) # the same German IPlifetime-10 keeps the IP for up to ten minutes (up to 1440 is allowed). The label is what matters, not the connection: requests with the same label got the same German IP whether they shared a Session or not.
Many requests at once
A shared AsyncClient opens one connection per parallel request and then reuses them, so you get as many IPs as connections, not one per request.
import asyncio
import os
import uuid
import httpx
USER = os.environ["PROXY_USER"]
PASS = os.environ["PROXY_PASS"]
async def fetch(url: str) -> str:
# One client per request with its own session label: a new exit for each.
proxy = f"http://{USER}-session-{uuid.uuid4().hex[:8]}-lifetime-3:{PASS}@via.proxies.vision:10000"
for attempt in range(3):
try:
async with httpx.AsyncClient(proxy=proxy, timeout=30) as client:
return (await client.get(url)).text
except httpx.TransportError:
if attempt == 2:
return "failed"
return "failed"
async def main() -> None:
sem = asyncio.Semaphore(10) # at most 10 requests at a time
async def limited(url: str) -> str:
async with sem:
return await fetch(url)
ips = await asyncio.gather(*[limited("https://api.ipify.org") for _ in range(20)])
print(f"{len(set(ips) - {'failed'})} distinct IPs for {len(ips)} requests")
asyncio.run(main())In our run this printed 20 distinct IPs for 20 requests. The semaphore caps how many run at once; the label gives each its own exit.
Scrapy
Scrapy also keeps connections open. A downloader middleware that sets a new label per request fixes it.
import os
import uuid
class RotatingSessionMiddleware:
"""A fresh session label per request, so every request leaves through a new IP.
Scrapy keeps connections open; without the label it would reuse one exit."""
def __init__(self):
self.user = os.environ["PROXY_USER"]
self.password = os.environ["PROXY_PASS"]
def process_request(self, request, spider=None):
if "proxy" not in request.meta:
label = uuid.uuid4().hex[:8]
request.meta["proxy"] = f"http://{self.user}-session-{label}-lifetime-3:{self.password}@via.proxies.vision:10000"DOWNLOADER_MIDDLEWARES = {
"myproject.middlewares.RotatingSessionMiddleware": 350,
}With the middleware, three requests from one spider left through three different IPs. Without it, all three used the same one.
Why does requests.Session use the same proxy IP for every request?
Does httpx rotate the proxy IP on every request?
How do I keep the same IP for a login in Python?
Should I rotate through a list of proxies myself?
Keep reading
Residential Proxies With ASN Targeting: Pick the ISP
Pick the ISP behind a residential proxy with -asn- in the username: Comcast, Deutsche Telekom, Swisscom. Why it needs a country, what 465 means, tested code.
5 min read · #geo-targeting#residential-proxies
Go net/http Proxy Authentication: HTTP and SOCKS5
Authenticated HTTP and SOCKS5 proxies in Go without extra packages, why rotation needs DisableKeepAlives, and a code generator.
6 min read · #go#authentication
Scrapy: Could not open CONNECT tunnel with proxy (407, 403, SOCKS5)
What the status in Scrapy's TunnelError means, why socks5:// only times out, and a proxy middleware that works. Every case reproduced.
7 min read · #scrapy#python
Residential proxies, pay per GB
HTTP and SOCKS5. Country, city and sticky sessions in the username. No subscription.