How to Scrape Cloudflare-Protected Sites Without Getting Blocked

You write a scraper, it works on your machine, and the moment you point it at a Cloudflare site you get a wall: a challenge page, a 403, or an endless "checking your browser." Cloudflare sits in front of a huge share of the web now, so if you scrape at all, you'll meet it. Here's how to collect public data from Cloudflare-protected sites without tripping every alarm, and how to stay on the right side of the line while you do it.

First, the honest framing. This is about legitimate work: public data, price monitoring, research, testing your own properties. Respect the target's terms of service and its robots rules, keep your request rate reasonable, and don't hammer infrastructure you don't own. The techniques below reduce false-positive blocks on legitimate crawling; they aren't a license to abuse a service.

Why Cloudflare blocks you

Cloudflare stacks several defenses, and you can hit any of them. IP reputation comes first: a datacenter IP that thousands of scrapers already burned is suspect before you send a single header. Then there's the environment check, where JavaScript challenges like Turnstile and the 5-second interstitial look at whether a real browser is present. On top of that, rate and pattern analysis flags traffic that behaves like a script: too fast, too regular, no human rhythm.

Get blocked and it's usually one of those three, not some unbeatable magic. Fix them in order.

The playbook

1. Use good proxies, and keep the session sticky

This is the biggest lever. Residential and mobile IPs carry far better reputation than datacenter ranges, because they look like real users. Rotate them so you're not slamming a site from one address, but keep a session sticky when you need continuity: if you've earned a clearance cookie, the follow-up requests have to come from the same IP or Cloudflare throws the cookie out. Cheap datacenter proxies are a false economy here; they're where most "why am I blocked" stories start.

2. Send believable headers

A default Python or Node user-agent is a giveaway. Send a real browser user-agent, and make the rest of your headers consistent with it (accept-language, accept-encoding, the usual set). The point isn't to lie; it's to not stick out as an obviously scripted client when you're doing ordinary crawling.

3. Clear the challenges, then reuse what you get

When you hit an actual challenge, you have two things to handle. The Turnstile widget returns a token you submit with the form. The 5-second interstitial returns a cf_clearance cookie you reuse on later requests. Solve once, then reuse: don't re-challenge yourself on every request. With a solving API this is a single call that hands back the token or the cookie plus the matching user-agent.

import requests

# clear the 5s challenge once, keep the clearance for the session
sol = requests.post(
    "https://api.peak.fo/solve",
    headers={"X-API-Key": "pk_your_api_key"},
    json={"task_type": "cloudflare5stask", "url": "https://target.com/",
          "proxy": "http://user:pass@ip:port"},
    timeout=30,
).json()["data"]

s = requests.Session()
s.headers["User-Agent"] = sol["headers"]["user-agent"]
for k, v in sol["cookies"].items():
    s.cookies.set(k, v)
# now crawl with s, same proxy, at a human pace

4. Slow down and vary

Machine-perfect timing is a tell. Space requests out, add jitter, and don't crawl a thousand pages a minute from one session. Respect the rate the site can reasonably serve. Slower and steady beats fast and blocked, every time.

Browser or API for the challenge?

If your crawl needs to click through pages and read content that only renders after interaction, run a real browser and let it handle the challenge inline. If you just need the token or the cookie and then you're making plain HTTP requests, an API is lighter and scales without a browser per worker. Most large crawls end up using the API for the challenge and their own HTTP client for the actual fetching. We break the trade-off down in the Turnstile solving guide.

FAQ

How do I scrape a Cloudflare-protected site?

Use residential or mobile proxies with sticky sessions, send real browser headers, solve any Turnstile or 5-second challenge and reuse the token or cf_clearance cookie, and crawl at a human pace. Most blocks come down to a bad IP, a scripted-looking client, or too-fast requests.

Is scraping Cloudflare sites legal?

Scraping public data is generally fine, but it depends on the site's terms of service, its robots rules, and what you do with the data. Don't access private or account-gated content you're not authorized to, and respect rate limits. The tooling is neutral; the use is on you.

Why do I keep getting the 5-second challenge?

Usually because you're not reusing the cf_clearance cookie, or you're sending it from a different IP or user-agent than the one that earned it. Keep the session sticky and send the matching user-agent. See the cf_clearance guide.

Handling Cloudflare challenges at scale? Start free at peak.fo.