Crawlee
Web scraping and browser automation for Node.js
The Lens
By Erik Loyd, SaaS CEO and former COO/CFO of an AWS Premier Partner.
Updated Aug 2026
Crawlee is a Node.js web scraping and browser automation library. Not a point-and-click scraper. A full framework for building crawlers that handle the hard stuff: request queuing, proxy rotation, browser fingerprinting, error recovery, and storage.
Supports three modes: plain HTTP requests (fast, for simple pages), Cheerio (HTML parsing without a browser), and full browser automation via Playwright or Puppeteer (for JavaScript-heavy sites). You pick the right tool for each job. Built by Apify, who run a web scraping platform. They open-sourced their crawler framework and it's legitimately good.
Completely free and Apache-2.0 licensed. No feature gates. You can deploy crawlers anywhere: your server, AWS, or Apify's cloud platform (which is paid but optional).
The catch: Crawlee is Node.js only. Python developers should look at Scrapy. Also, web scraping is inherently fragile; sites change, CAPTCHAs evolve, rate limits tighten. Crawlee gives you the tools to handle this but you still need to maintain your scrapers. Apify Cloud is the easy button for deployment ($49/mo+) but self-hosting works fine.
Free vs Self-Hosted vs Paid
fully freeWhat's Free
The entire Crawlee framework: HTTP crawling, browser automation (Playwright/Puppeteer), request queuing, proxy rotation, error handling, data storage, autoscaling. Apache-2.0 licensed. Deploy anywhere.
Apify Cloud (Optional, Paid)
Crawlee was built by Apify and integrates with their cloud platform:
- Free tier: $5/mo credit, shared compute. Good for testing.
- Starter ($49/mo): 100 compute units, proxy access.
- Scale ($499/mo): 1,000 compute units, priority support.
- Enterprise: Custom pricing.
You never have to use Apify Cloud. Crawlee works standalone.
Self-Hosting Cost
Depends on scale. A $10/mo VPS runs simple crawlers. Browser-based crawling (Playwright) needs more RAM. Budget $20-40/mo for moderate workloads. At scale, proxy costs dominate ($50-500+/mo depending on volume).
Verdict
The framework is free. Your costs are infrastructure and proxies, which apply regardless of which scraping tool you use.
Framework is fully free. Infrastructure and proxy costs are your only expenses.
What to do by team size
- Solo
- Excellent for side projects and data collection. Self-host on a cheap VPS.
- Small team
- Self-host or use Apify Cloud starter for team convenience.
- Medium team
- Apify Cloud Scale or self-host with proper infrastructure. Proxies become the big expense.
- Large team
- Apify Enterprise or build internal scraping infrastructure. Crawlee is the right foundation.
Get tools like this every Wednesday
One featured tool, three on the radar. No fluff.
Similar Tools
A low score is not a verdict on quality. Young and niche tools start low by design. How we calculate scores
Trust Signals
About
- Stars
- 25,866
- Forks
- 1,679
Explore Further
More tools in the directory
openclaw
Your own personal AI assistant. Any OS. Any Platform. The lobster way. 🦞
389.9k ★everything-claude-code
The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.
264.9k ★hermes-agent
The agent that grows with you
247.9k ★