Web Scraping

crawlee-python9.2k★

Crawlee gives you a Python framework that handles the ugly parts: browser automation, request queuing, proxy rotation, and anti-bot countermeasures. It's what you build on top of instead of writing raw Selenium or Playwright scripts from scratch. Apache 2.0. Built by Apify, who also sell a cloud scraping platform, but the library itself is fully independent. Supports both HTTP crawling (fast, lightweight) and browser-based crawling (Playwright under the hood for JavaScript-heavy sites). Automatic retries, request deduplication, and session management come built in. Fully free to use. No gated features, no paid tier for the library itself. Apify's cloud platform is a separate product. You never need it. Solo developers get a production-grade scraping framework for $0. Teams of any size benefit from the built-in queue management and error handling that you'd otherwise build yourself. The catch: this is the Python port of Crawlee, which started as a Node.js library. The Python version is younger and the ecosystem of plugins and examples is smaller. If you're in the Node world, the original Crawlee (also by Apify) is more mature. And Apify's cloud integration is deeply embedded; you'll see references to it everywhere in the docs even if you never use it.

httrack4.5k★

HTTrack copies an entire website to your hard drive so you can browse it offline. Point it at a URL and it recursively pulls down the HTML, images, and assets, rebuilding the link structure so the local mirror works exactly like the live site. It's been doing this one job well since the early 2000s, on Windows as WinHTTrack and on Linux as WebHTTrack. There's nothing to set up beyond installing it. It resumes interrupted downloads, updates existing mirrors, and gives you filters to control how deep it crawls and what it grabs. For a tool this old, it's remarkably dependable, which is why it's still the default answer for site archiving. Anyone who needs a local copy of a site, for offline reading, archiving before a shutdown, or grabbing documentation you can't count on staying up, gets exactly that. It's not a modern scraper for structured data extraction, reach for Scrapy or Playwright if you need to parse and transform what you pull. HTTrack mirrors; it doesn't analyze. The catch: the web moved on. Heavy JavaScript sites that render content client-side don't mirror cleanly, because HTTrack grabs what the server sends, not what a browser builds. It shines on traditional, server-rendered sites and struggles with modern single-page apps. Know which kind you're pointing it at.

kage2.4k★

kage saves websites that actually still work offline. Hit "save page" in a browser and half the time you get a broken mess later, because the JavaScript phones home to servers that are gone. kage renders the page in headless Chrome, grabs the finished result, strips every script, and pulls the CSS, images, and fonts down to disk. What's left is a static copy that doesn't rot. Go, MIT licensed, free. It's a command-line tool that does more than dump a folder. It can output a browsable mirror, a compressed ZIM archive, or a self-contained executable you can hand to someone, and it'll serve the clone locally so you can preview it. It respects robots.txt and sitemaps while crawling, and the output is deterministic, so the same site produces a byte-identical archive you can checksum. That's a thoughtful detail most scrapers skip. Useful for archivists, researchers, or anyone who wants a permanent, readable copy of something before it disappears or changes. It's a single-purpose CLI, free for everyone, with nothing to buy. Run it when you need a snapshot you can trust to still open in five years. The catch is what it removes. Stripping JavaScript is the whole point, but it also means anything that genuinely needs scripts to function, a web app, an interactive tool, won't survive the trip. kage preserves pages you read, not apps you use. Know which one you're archiving before you run it.