Web Scraping

Scrapy65.2k★

Scrapy is the Python framework for when a scraping script turns into a scraping project. You write "spiders" (classes saying which pages to visit and what to extract) and Scrapy runs them concurrently with throttling, cookies, robots.txt handling and exports to JSON, CSV or XML. Free under BSD-3-Clause, maintained by Zyte, and still shipping: 2.19 landed in September. Setup is pip install scrapy on Python 3.11 or newer. It runs on Twisted, the usual learning-curve complaint; a mode without Twisted arrived this year but is labeled experimental. JavaScript-heavy pages need a browser, and scrapy-playwright is the maintained route. Blocks, CAPTCHAs and proxy rotation are not Scrapy's job. Zyte sells what sits around it. Scrapy Cloud hosts spiders free with 1-hour jobs and 7-day retention, then $9/month per unit for unlimited run time. Zyte API handles unblocking per 1,000 successful responses. Solo and small teams: free. Larger teams: still free; the bill is proxies. The catch: Scrapy parses whatever hostile sites send back. In 2026 a malicious Referrer-Policy header could make it import and call Python objects (fixed in 2.14.2), and S3 requests went out signed over plain HTTP (fixed in 2.17). Upgrade. For one-page jobs, requests plus BeautifulSoup is simpler.

Lightpanda41.4k★

Lightpanda is a headless browser written from scratch in Zig, not another Chromium wrapper. Your Puppeteer or Playwright scripts connect to it over CDP just as they'd connect to headless Chrome. On the project's own 100-page benchmark it used about 16x less memory and finished about 9x faster. It's AGPL-3.0 and free to self-host. Running it is a single binary or one docker run exposing a CDP server on port 9222. Version 1.0 shipped in October; it now speaks WebDriver BiDi, runs as an MCP server for AI agents, and has an agent mode that saves a session as a replayable script. Linux and macOS only; Windows goes through WSL. Lightpanda Cloud is the paid side: Explorer is free with 10 browser hours a month, Builder is $19/month for 300 hours and 30 concurrent sessions, then $0.08 per extra hour. Bring your own proxy on both. Solo: self-host or Explorer. Small teams: Builder, or Browserless and Browserbase if you need real Chrome. At scale, self-hosting is where the memory savings pay off. The catch: it isn't Chrome. PNG and PDF dumps are text-only renderings, so visual testing is out, and JavaScript-heavy sites can still break. Telemetry is on by default.

Crawlee26.4k★

Crawlee is a Node.js web scraping and browser automation library. Not a point-and-click scraper. A full framework for building crawlers that handle the hard stuff: request queuing, proxy rotation, browser fingerprinting, error recovery, and storage. Supports three modes: plain HTTP requests (fast, for simple pages), Cheerio (HTML parsing without a browser), and full browser automation via Playwright or Puppeteer (for JavaScript-heavy sites). You pick the right tool for each job. Built by Apify, who run a web scraping platform. They open-sourced their crawler framework and it's legitimately good. Completely free and Apache-2.0 licensed. No feature gates. You can deploy crawlers anywhere: your server, AWS, or Apify's cloud platform (which is paid but optional). The catch: Crawlee is Node.js only. Python developers should look at Scrapy. Also, web scraping is inherently fragile; sites change, CAPTCHAs evolve, rate limits tighten. Crawlee gives you the tools to handle this but you still need to maintain your scrapers. Apify Cloud is the easy button for deployment ($49/mo+) but self-hosting works fine.

camoufox13.1k★

Camoufox is a Firefox build that makes a scraper or AI agent look like an ordinary person's browser. Each launch gets a consistent fake device (screen, fonts, GPU, timezone, locale) drawn from real-world data, reported from the browser's C++ code instead of JavaScript overrides a site can catch. It's free: the browser is MPL 2.0 and the Python and TypeScript launchers are MIT. Setup is pip install -U "camoufox[geoip]" then camoufox fetch (or the npm package), and you swap your Playwright launch call. Playwright's code runs in an isolated world the page can't see. Nothing is paywalled, so solo devs, small teams and larger teams all get the same build. The real bill is residential proxies: your IP is the one thing it can't hide. Hosted Browserbase starts at $20/mo. The catch: the README still warns it may not be suitable for stable production use, and every release is versioned as a beta. The original author is on medical leave, so a core maintainer and contributors now do most of the work. It only ever presents as Firefox, which some bot defenses test for directly. And the README is mostly proxy-seller ads, so read those recommendations as paid placements.

crawlee-python9.7k★

Crawlee gives you a Python framework that handles the ugly parts: browser automation, request queuing, proxy rotation, and anti-bot countermeasures. It's what you build on top of instead of writing raw Selenium or Playwright scripts from scratch. Apache 2.0. Built by Apify, who also sell a cloud scraping platform, but the library itself is fully independent. Supports both HTTP crawling (fast, lightweight) and browser-based crawling (Playwright under the hood for JavaScript-heavy sites). Automatic retries, request deduplication, and session management come built in. Fully free to use. No gated features, no paid tier for the library itself. Apify's cloud platform is a separate product. You never need it. Solo developers get a production-grade scraping framework for $0. Teams of any size benefit from the built-in queue management and error handling that you'd otherwise build yourself. The catch: this is the Python port of Crawlee, which started as a Node.js library. The Python version is younger and the ecosystem of plugins and examples is smaller. If you're in the Node world, the original Crawlee (also by Apify) is more mature. And Apify's cloud integration is deeply embedded; you'll see references to it everywhere in the docs even if you never use it.

httrack4.8k★

HTTrack copies an entire website to your hard drive so you can browse it offline. Point it at a URL and it recursively pulls down the HTML, images, and assets, rebuilding the link structure so the local mirror works exactly like the live site. It's been doing this one job well since the early 2000s, on Windows as WinHTTrack and on Linux as WebHTTrack. There's nothing to set up beyond installing it. It resumes interrupted downloads, updates existing mirrors, and gives you filters to control how deep it crawls and what it grabs. For a tool this old, it's remarkably dependable, which is why it's still the default answer for site archiving. Anyone who needs a local copy of a site, for offline reading, archiving before a shutdown, or grabbing documentation you can't count on staying up, gets exactly that. It's not a modern scraper for structured data extraction, reach for Scrapy or Playwright if you need to parse and transform what you pull. HTTrack mirrors; it doesn't analyze. The catch: the web moved on. Heavy JavaScript sites that render content client-side don't mirror cleanly, because HTTrack grabs what the server sends, not what a browser builds. It shines on traditional, server-rendered sites and struggles with modern single-page apps. Know which kind you're pointing it at.

kage3.5k★

kage saves websites that actually still work offline. Hit "save page" in a browser and half the time you get a broken mess later, because the JavaScript phones home to servers that are gone. kage renders the page in headless Chrome, grabs the finished result, strips every script, and pulls the CSS, images, and fonts down to disk. What's left is a static copy that doesn't rot. Go, MIT licensed, free. It's a command-line tool that does more than dump a folder. It can output a browsable mirror, a compressed ZIM archive, or a self-contained executable you can hand to someone, and it'll serve the clone locally so you can preview it. It respects robots.txt and sitemaps while crawling, and the output is deterministic, so the same site produces a byte-identical archive you can checksum. That's a thoughtful detail most scrapers skip. Useful for archivists, researchers, or anyone who wants a permanent, readable copy of something before it disappears or changes. It's a single-purpose CLI, free for everyone, with nothing to buy. Run it when you need a snapshot you can trust to still open in five years. The catch is what it removes. Stripping JavaScript is the whole point, but it also means anything that genuinely needs scripts to function, a web app, an interactive tool, won't survive the trip. kage preserves pages you read, not apps you use. Know which one you're archiving before you run it.

insane-search2.7k★

When Claude Code hits a 403 and gives up, insane-search keeps going. It installs from a plugin marketplace URL with no API keys and no configuration, then takes over automatically whenever an ordinary fetch gets blocked. MIT licensed, free, and it pulls in its own dependencies like curl_cffi and yt-dlp the first time it needs them. The mechanism is an escalation ladder. It starts with public endpoints and feeds, drops to mobile and .json and /rss variants of the URL, then impersonates a real browser's TLS fingerprint, and finally spins up an actual headless browser for JavaScript-rendered pages. Every response gets scanned for OGP and JSON-LD, so even a partial page yields a title and a summary. At a login wall or paywall it stops and reports authentication required instead of pretending. Solo research and one-off lookups: it works, and it works on sites that routinely stonewall agents. Teams building something durable: do not put this in a pipeline you depend on. The catch: everything past the syndication step is precisely what anti-bot systems exist to detect, and the project hands you the terms-of-service question directly. Reddit, X, and LinkedIn all prohibit this in writing. The usual outcome is not a lawsuit, it is your IP quietly getting nowhere.

browserless1.8k★

browserless wraps Puppeteer, Google's headless-Chrome library, into a friendlier toolkit for the boring production jobs: screenshot a page, render it to PDF, or pull clean HTML and text out of a site. It handles the parts that usually bite you, retries, device emulation, and a built-in ad blocker so junk requests do not slow the render. MIT-licensed and free to run yourself. (This is Microlink's browserless, not the separate browserless.io service.) As a library it drops into a Node project with npm, using either full Puppeteer or the lighter puppeteer-core. A CLI ships too for quick one-offs. The real ops cost is the same one every headless-Chrome setup pays: Chromium is memory-hungry and crashes under load, so running it at scale means babysitting browser instances, and that is the work you are choosing to own. For occasional screenshots or PDFs inside an app you already run, the self-hosted library is free and enough. When you do not want to manage Chromium yourself, the same team offers Microlink, a pay-as-you-go hosted API that starts free. Solo and small teams: self-host and save the cash. Teams generating renders at volume, or tired of Chromium falling over: the hosted API is the honest trade, you pay to make the ops problem someone else's. The catch is that headless Chrome is never truly set-and-forget. Self-hosting saves money right up until a traffic spike takes your browser pool down at 2 a.m. Price your own time into the decision.