Skip to content

TaskRunner

Distributed & Scalable
Add instances to scale throughput horizontally — consumer groups, leader election, and automatic worker tuning do the rest.
Reliable by Design
Visibility timeouts with heartbeats prevent double-processing; pending reclaims and retries recover crashed workers automatically.
Delayed Scheduling
Schedule jobs for future execution via a Redis ZSET polled by the leader, paged into the stream as they come due.
Unique Jobs
Enforce de-duplication of the same job (optionally scoped by a custom key) across a configurable time window using distributed locks.
Observability
Per-task timing metrics, predicted wait-time estimates, TPS, and queue statistics — surfaced via hooks and the stats API.
Automatic Tuning
The leader recomputes worker-pool capacity to hold predicted wait time within a target band, clamped to min/max bounds.

Why TaskRunner?

Most "task queue" libraries in Go either target a single process or force you to run a separate broker process (as with Celery or Sidekiq). TaskRunner is a library — there is no daemon to deploy. You embed it in your application, point it at a Redis instance you already have, and you get a production-grade distributed task processor:

  • At-most-once dispatch, at-least-once processing. Heartbeats extend a message's visibility timeout while a worker is busy, so a slow task is never reclaimed and double-processed by another consumer. If the worker genuinely crashes, the pending-entry reclaimer picks it up.
  • Cooperative leader election (atomic SET / PEXPIRE / DEL via Lua) ensures exactly one instance runs the scheduler, computes timing stats, and publishes worker-pool capacity. Followers apply the leader's capacity.
  • Pluggable queue backend. Everything speaks the small contracts.MessageQueue interface. Redis Streams ships in the box; you can implement your own.
flowchart LR
    subgraph Producers
        P1[App / API]
    end
    subgraph Redis
        RS[(Redis Streams)]
        ZS[(Delayed ZSET)]
        LK[(Leader Lock)]
        HM[(Timing Hashes)]
    end
    subgraph "TaskRunner instances"
        T1[Instance A<br/>leader]
        T2[Instance B<br/>follower]
        T3[Instance C<br/>follower]
    end
    P1 -->|Dispatch / DispatchDelayed| RS
    P1 -->|DispatchDelayed| ZS
    RS --> T1
    RS --> T2
    RS --> T3
    ZS -->|leader polls| T1
    T1 -.publishes capacity.-> T2
    T1 -.publishes capacity.-> T3
    LK -.owns.-> T1
    T1 --> HM
    T2 --> HM
    T3 --> HM

What's next?

  • Getting Started — install, requirements, and a working quickstart in 5 minutes.
  • Architecture — understand the fetcher/worker split, election, and data flow.
  • Configuration Reference — every field of TaskRunnerConfig explained.
  • API Reference — the public types, methods, and sentinel errors.
  • Examples — runnable programs for queues, scheduling, unique jobs, and benchmarking.
  • Benchmarks — measured costs of the hot paths.

Project facts
Language Go 1.24.9+
Module github.com/soroosh-tanzadeh/taskrunner
Transport Redis 6+ (Redis 6.2+ recommended for pending-idle filtering)
Dependencies go-redis/v9, panjf2000/ants/v2, go-redsync/v4, logrus, google/uuid, Masterminds/semver/v3
License GPL-3.0
Tests go test ./... — uses miniredis, so no real Redis required; CI adds a real redis:7.4 container

Get started View on GitHub