Bypass Cloudflare with these web scraping tools
Pierluigi Vinciguerra
Co Founder and CTO at Databoutique.com | Writing on The Web Scraping Club
In this article of The Web Scraping Club we see the Python tools we can use to bypass Cloudflare protected websites when trying to scrape them.
What is Cloudflare
Cloudflare is one of the most well-known anti-bot solutions and bypassing it could be a difficult challenge for any web scraping project.
In my past articles, I’ve written several times about bypassing Cloudflare using different approaches and I will summarize them in this post.
Undetected Chromedriver
We have seen in the?Anti-Detect Anti-Bot matrix post?that a good solution against Cloudflare could be the?Undetected Chromedriver python package.
Basically, it consists of a Chromedriver version modified for specific usage in web scraping projects.
Combined with Selenium, you can automate most of the Chromium browsers. Not only Chrome then, but also Brave and GoLogin.
The setup for this solution comes in two easy step
领英推荐
Playwright
In this blog, we often write about Playwright, because of its ductility and its ease to use. As an example, in?one of the latest “The Lab” posts we have seen how to use it together with GoLogin, but it’s not the only setup we can use. It supports natively all modern rendering engines including Chromium, WebKit, and Firefox so we can have a lot of fun testing around which solution fits the best for our needs.
During the last 4-5 months I’ve noticed that the best solution to bypass most of the Cloudflare-protected websites consists in using Firefox together with Playwright, as stated also?in our Anti-Detect Anti-Bot matrix.
Even in this case, the setup is quite easy.
After importing the package, we set up a Playwright session (sync or async, depending on your needs for multi-threading and parallelism) and then we load the page in the browser we prefer.
The different options and setups we can try are countless but I’m bringing to attention some of them I’ve found useful.
browser = p.chromium.launch_persistent_context(user_dir, headless=False)
Conclusions
Yes, Cloudflare can be a pain to bypass, especially on some websites where the rules for detecting bots are strict, but usually these configurations damage also real users.
We have the means for bypassing the anti-bot solution, but it all depends from case to case and there's no silver bullet for it. And if there was, it would work only until the next release. It’s always a cat-and-mouse game and techniques need to be always updated. The only permanent condition is to treat target websites respectfully and ethically, to not cause harm or malfunction.
If you liked this post and want to receive all every article from the Club in your email, consider subscribing for free at this link https://substack.thewebscraping.club/
Специалист Интернет-технологии
8 个月Only working solution nowdays https://rapidapi.com/ruspacenet-6x8jdbfLxk8/api/cloudflaresolver/
Founder & CEO | Salad Technologies
1 年Pierluigi Vinciguerra I recently signed up to the web scraping club and appreciate your work! At Salad we've been looking into this use case for our network: we can distribute 10 to 10,000 containerised workloads, compute at the edge with unique residential IPs. We're seeing some customers run headless browsers, and I'd be very curious to get your thoughts on use cases for our infrastructure ??
Cheers, keep up the good work! ??
Head of Marketing at GoLogin
1 年Thanks for mentioning GoLogin!
Software Engineer
1 年It’s common that protected websites set up Cloudflare without changing the origin’s IP address, which is very likely still visible on older DNS records. That should not be a problem, although sometimes they don’t deny connections outside CF IP ranges, resulting on leaving a door open for a very efficient way to bypass Cloudflare protections and caching.