The thread that started this A couple months ago I saw a post on r/webscraping that summed up the current state of things better than I ever could: "We have scrapers, ordinary ones, browser automation… we use proxies for location based blocking, residential proxies for data centre blockers, we rotate the user agent, we have some third party unblockers too. But often, we still get captchas, and CloudFlare can get in the way too." Four layers of evasion. Still getting challenge pages. 167 upvotes, 52 comments, archived without a real solution. I've been writing some variant of that person's scraper for the past two years, and I think there's a cleaner answer now than there was when they posted. This is a writeup of what I think the actual problem is, what I tested, and the CLI-based setup I've ended up with. The four layers that don't get you past Cloudflare These are the evasion techniques the OP was already using: 1. User agent rotation Changes the User-Agent header per request.…