Background Jobs

Celery28.9k★

Celery handles background work in Python applications: sending emails, processing uploads, running scheduled reports, anything that shouldn't block the user. It's a distributed task queue: you define tasks as Python functions, Celery runs them on separate worker processes, and you get results back when they're done. Fully free under a BSD-derived license. No paid tier, no hosted version, no enterprise upsell. You need a message broker (Redis or RabbitMQ, both free to self-host) and that's it. It handles retries, scheduling, rate limiting, and task chaining out of the box. The catch: Celery is powerful but operationally heavy. Debugging failed tasks across distributed workers is painful. The configuration surface is enormous. There are hundreds of settings and the docs assume you already know what you want. If you just need simple background jobs, something like BullMQ (Node.js) or Dramatiq (Python) is far less config. Celery earns its complexity when you actually need distributed processing at scale.

Sidekiq13.6k★

Sidekiq handles background jobs in Ruby: sending emails, processing uploads, generating reports, anything that shouldn't block a web request. It uses Redis to queue jobs and processes them in separate worker threads. Sidekiq's killer feature is throughput. It uses threads instead of processes (unlike Resque, the older alternative), so a single Sidekiq process handles 25+ concurrent jobs by default. A $20/mo VPS running one Sidekiq process can chew through millions of jobs per day. The open source version covers most needs: job queuing, retries with exponential backoff, scheduled jobs, a real-time web dashboard, and dead job management. The web UI is excellent: you can see queue depths, retry counts, and job history at a glance. Sidekiq Pro ($99/mo per process) adds reliable fetch (jobs won't be lost if a process crashes), batches (group jobs and run a callback when all complete), and unique jobs. Sidekiq Enterprise ($179/mo per process) adds rate limiting, periodic jobs (cron replacement), and rolling restarts. The catch: Ruby only. If you're not in the Rails ecosystem, this doesn't apply to you. The paid tiers are per-process, which adds up fast: 4 processes on Enterprise is $716/mo. And you need Redis running alongside it, adding another piece of infrastructure to manage.

agenda9.7k★

Agenda is a job scheduler for Node.js. Define jobs as functions, schedule them with cron syntax or human-readable intervals like "in 20 minutes", and Agenda persists them so they survive a restart. MIT licensed and free, with no paid tier. Version 6 is a full TypeScript rewrite, and it kills the old objection. Storage is pluggable now: MongoDB by default, with Postgres and Redis backends available, so this is no longer a scheduler that drags Mongo into a stack that does not want it. Real-time notifications ride on Redis, Postgres LISTEN/NOTIFY, or MongoDB change streams instead of pure polling. Priorities, concurrency limits, retries with backoff, repeatable jobs, and fork mode for sandboxed execution all ship in the box, alongside optional companion packages for a web dashboard and a REST API. Solo developers building Node apps that need background work: solid, and free. Small teams: works well through moderate job volume, and the dashboard makes it easy to see what is stuck. Larger throughput deserves honest measurement first, because raw jobs per second is not the ground Agenda competes on. The catch is the upgrade. Version 6 is ESM-only and needs Node 18 or newer, so coming from v5 is a real migration with a guide to read, not a version bump. And if you are pushing thousands of jobs a second, taskforcesh/bullmq on Redis is still the faster answer. Agenda's pitch is scheduling with good ergonomics and a persistence layer you get to pick, not maximum throughput.

BullMQ9.4k★

BullMQ is the go-to job queue for Node.js applications, backed by Redis, with priorities, scheduling, retries, concurrency limits, and rate limiting built in. You push jobs onto a queue, workers pick them up and process them. Retries, delays, rate limiting, priorities, repeatable jobs, and job dependencies are all built in. It's the Node.js equivalent of Celery (Python) or Sidekiq (Ruby). You define a queue, add jobs to it, and BullMQ handles the rest, including the messy parts like failed job retries, stalled job recovery, and concurrency control. MIT. Requires Redis (or any Redis-compatible store like Valkey or KeyDB). The catch: you need Redis running. That's either a managed Redis service ($15-50/mo) or self-hosted Redis (free but you maintain it). BullMQ also has a paid companion: Bull Dashboard Pro for $99/yr gives you a nicer UI for monitoring jobs. The free Bull Board works fine for most needs. The bigger catch: if your stack isn't Node.js, BullMQ doesn't help you. It's Node-only.

bree3.3k★

Node.js has no built-in way to run scheduled jobs without tying up the event loop. Bree handles that with worker threads so your main process doesn't block. Basically, it's cron for your Node.js app, but each job runs in its own thread and can't crash your server. MIT license, JavaScript. Each job runs in a Node.js worker thread with its own memory space. If a job crashes, your app keeps running. Supports cron expressions, intervals, one-time runs, and human-readable schedules ("every 5 minutes"). Built-in concurrency control. Prevent the same job from running twice. Fully free. It's a library. No paid tier, no hosted version. Setup is simple: install the package, define your jobs as separate files, configure the schedule. The worker thread approach means jobs are truly isolated. A memory leak in one job doesn't affect your app. Solo projects and small apps: Bree is a clean solution when you need background jobs without adding Redis or a separate queue system. The catch: Bree runs in-process, which means if your server restarts, scheduled jobs reset. There's no persistence layer. Missed jobs don't retry automatically. For durable job processing that survives restarts, you need BullMQ (backed by Redis) or Temporal. Bree is best for periodic tasks where missing one run isn't catastrophic.