Deployment

TaskRunner is a library you embed, so "deployment" mostly means: how you run your application across multiple instances and how you size Redis. This page covers the production concerns.

Topology

flowchart LR
    LB[Load balancer / API] --> A[App instance A<br/>+ TaskRunner]
    LB --> B[App instance B<br/>+ TaskRunner]
    LB --> C[App instance C<br/>+ TaskRunner]
    A --> R[(Redis<br/>Streams + state)]
    B --> R
    C --> R
    R -.- S[(Redis Sentinel / Cluster<br/>optional HA)]

• Run N >= 2 instances. One wins leadership; the rest process messages as followers. Leadership failover is automatic within ~TTL + Wait (~11s default).
• All instances use the same ConsumerGroup. That's how Redis Streams distributes work across them. Different groups each receive a full copy.
• All instances call StartDelayedSchedule (if you use delayed tasks), so the new leader after a failover is already running the scheduler loop.

Redis sizing

Resource	Guidance
Version	6.2+ (for XPENDING IDLE filtering). 7.x recommended.
Memory	Size for peak in-flight + retained stream. With deleteOnAck=true (default), the stream stays small -- size mainly for the PEL and delayed ZSET.
Connections	Each instance holds PoolSize connections (default per go-redis). With M instances and pool size P, plan for M * P connections + headroom.
Persistence	For durability of in-flight messages, use AOF (appendfsync everysec) or RDB snapshots. Streams + PEL live in the dataset, so any persistence strategy that protects your data protects your queue.
Eviction policy	noeviction (the default). TaskRunner keys must not be evicted -- a lost lock or PEL entry means double-processing or lost work.

Single Redis vs. Cluster vs. Sentinel

TaskRunner works against:

• Single Redis -- simplest, fine for most workloads. Single point of failure; mitigated by Sentinel or persistence.
• Redis Sentinel -- HA without sharding. The go-redis client supports failover via redis.NewFailoverClient. TaskRunner is agnostic; just pass it the client.
• Redis Cluster -- sharded. redis.NewClusterClient satisfies redis.Scripter for the election Lua scripts. Make sure all TaskRunner keys for one consumer group hash to the same slot, or use hash tags ({group}:tasks). In practice, since TaskRunner's keys are already grouped by <consumerGroup>, wrapping the group in {} ensures colocation.

Election correctness under failover

The leader lease relies on Redis's single-threaded atomicity. During a Sentinel/Cluster failover, there's a brief window where two nodes may believe they're master. TaskRunner's TTL (5s) bounds the risk, but expect a possible double-leader tick during failover. Because all leader chores are idempotent, this is safe -- you may just see a duplicate scheduler tick or metrics reset.

Graceful shutdown

Start(ctx) blocks until the context is canceled. On cancellation:

1. Fetcher and election loops exit.
2. The worker pool drains with a 30-second timeout (workerPool.ReleaseTimeout(30s)).
3. The timing bulk writer flushes remaining samples and closes.
4. The error channel is closed (your drainer's range exits).
5. status becomes stateStopped; shutdown() runs.

ctx, cancel := context.WithCancel(context.Background())
go func() { _ = tr.Start(ctx) }()

// listen for SIGTERM
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
<-sigCh
log.Info("shutting down...")
cancel() // triggers the drain above

// Give in-flight tasks up to 35s to finish (30s pool drain + slack).
select {
case <-time.After(35 * time.Second):
    log.Warn("shutdown timed out; some tasks may be requeued")
case <-drained: // optional: signal from a wg tracking your own long tasks
}

In Kubernetes, set terminationGracePeriodSeconds to comfortably above your drain timeout (e.g. 40s if you use the 30s pool drain).

Memory footprint

Per instance, TaskRunner's own memory is modest: the task registry, the timing buffer (flushed each second), the ants pool's goroutine stacks. The dominant factor is your task payload and any state your Action holds. From the shipped benchmark harness (cmd/worker), processing 100k 500ms tasks on a single instance shows a working-set in the low tens of MB for the engine itself.

To measure your own:

var m runtime.MemStats
runtime.ReadMemStats(&m)
log.Printf("alloc=%d KB", m.Alloc/1024)

Health checks

Expose a readiness endpoint that confirms the runner is in the Started state:

http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
    // status is an atomic.Uint64; stateStarted is the running state.
    // There's no public accessor, so track it yourself alongside Start.
    if started.Load() {
        w.WriteHeader(http.StatusOK)
        return
    }
    w.WriteHeader(http.StatusServiceUnavailable)
})

/healthz (liveness) can be trivial -- the process is up. /readyz (readiness) should gate on Start having completed so you don't receive traffic (or, for worker-only instances, so you don't get killed before draining).

CI

The repo ships a GitHub Actions workflow (.github/workflows/go.yml) that runs go build + go test against a real redis:7.4 container. For your downstream service CI:

1. go test ./... -- uses miniredis, no Redis required.
2. go test -race ./... -- the runner is heavily concurrent; race detector is valuable. (Note: there is a known race in one master test; see AGENTS.md "Known: race in one test" if you fork.)
3. go vet ./... and gofmt -l . -- keep these clean.

For integration tests against a real Redis, use a service container:

services:
  redis:
    image: redis:7.4-bookworm
    ports: ["6379:6379"]
    options: >-
      --health-cmd "redis-cli ping"
      --health-interval 10s
      --health-timeout 5s
      --health-retries 5

Security

• TLS to Redis: go-redis supports redis.Options.TLSConfig. Pass it and TaskRunner uses it transparently.
• AUTH: set via redis.Options.Username / Password.
• Network isolation: TaskRunner keys are unencrypted; restrict Redis to your app network. Avoid exposing Redis to the public internet.
• Multi-tenant Redis: use distinct WithPrefix per environment (prod, staging) to namespace streams. Timing hashes and leader keys are already namespaced by ConsumerGroup.