Leader Election

TaskRunner uses a cooperative leader election built on Redis. Exactly one instance in a consumer group holds the leadership lease at any time, and that instance is the only one allowed to perform "global" operations:

• Run the delayed scheduler (moving due ZSET entries into the stream).
• Compute and reset timing metrics.
• Decide worker-pool capacity and publish it for followers to apply.

Followers are not idle -- they still consume and process messages from the shared stream. They simply defer the global chores to whoever won the lease.

The lease

Leadership is a single Redis key, derived from the consumer group name:

taskrunner:leader:<base32(sha1(ConsumerGroup))>

The value held in that key is a random per-instance ID. Three atomic Lua scripts operate on it (election/scripts.go):

Operation	Script	Semantics
Acquire	SET key id PX ttl NX	Wins only if the key is absent. Returns 1 on success.
Renew	GET key; if == id then PEXPIRE key ttl	Only the current holder can extend its own lease.
Release	GET key; if == id then DEL key	Only the current holder can step down.

Because each operation is a single Lua script, it executes atomically on the Redis server -- there is no window in which two instances could both believe they renewed, or in which an instance renews a lease it no longer owns.

The election loop

Each instance runs two timers (election/elector.go):

stateDiagram-v2
    [*] --> Follower
    Follower --> runElection: every Wait (6s)
    runElection --> Leader: atomicSet returned 1
    runElection --> Follower: atomicSet returned 0 (someone else holds it)
    Leader --> renew: after TTL/2 + jitter
    renew --> Leader: atomicRenew returned 1 (still ours)
    renew --> Follower: atomicRenew returned 0 (we lost it)
    Follower --> runElection: next tick
    Leader --> Follower: on demote

Default timings (set in runner/election.go):

Param	Value	Meaning
TTL	5s	How long the lease is valid without renewal.
Wait	6s	How often a follower retries atomicSet.
JitterMS	5ms	Random jitter added to the renewal interval to avoid synchronized stampedes.

The holder renews after TTL/2 + jitter, i.e. roughly every 2.5s -- well within the 5s TTL, so a few failed PEXPIREs are tolerated before expiry.

Promotion and demotion signals

The Elector exposes three channels:

• onPromote <-chan time.Time -- fired once when this instance wins the lease.
• onDemote <-chan time.Time -- fired once when this instance loses it.
• onError <-chan error -- fires on Redis errors during set/renew.

TaskRunner.StartElection wires these to t.isLeader (an atomic.Bool):

case <-OnPromote:
    t.isLeader.Store(true)
case <-onDemote:
    t.isLeader.Store(false)
case err := <-OnError:
    t.captureError(err)

Throughout the engine, t.IsLeader() is checked before doing global work. For example, the delayed scheduler:

if !t.IsLeader() {
    continue   // only the leader enqueues due jobs
}

Failure modes

Redis error during renewal

If atomicRenew returns an error (network blip, Redis MOVED, etc.), the elector assumes it lost the lease, emits a demotion, and schedules the next runElection for Wait later. This is the conservative choice: it's better to skip a tuning cycle than to risk two leaders.

Redis error during acquire

If atomicSet errors, the error is pushed onto onError and the next attempt is scheduled. The instance stays a follower.

Instance crash

No more PEXPIREs are sent. After TTL (5s) the key expires and another instance's next runElection wins. Worst-case leadership gap: TTL + Wait (~11s).

Split brain (network partition)

Redis is the source of truth. A partitioned instance that cannot reach Redis cannot renew, so its lease expires. There is no fencing token, so an instance that believes it is leader but is partitioned from Redis will keep acting until its next failed renewal. The 5s TTL bounds this window.

Why cooperative, not strongly consistent?

TaskRunner's leader chores (scheduler ticks, metric resets, capacity publication) are all idempotent and cheap. A brief overlap during a partition causes at worst a duplicate scheduler tick or a redundant metrics reset -- never data corruption. The simplicity of a single-key lease beats the complexity of a quorum-based protocol like Raft for this workload.

Using the elector standalone

The election package is usable on its own, independent of the runner:

import "github.com/soroosh-tanzadeh/taskrunner/election"

elector, onPromote, onDemote, onError := election.NewElector(election.Opts{
    Redis:    rdb,
    Key:      "my-app-coordinator",
    TTL:      5 * time.Second,
    Wait:     6 * time.Second,
    JitterMS: 5,
})
go func() {
    for {
        select {
        case <-onPromote:
            log.Println("I am now the leader")
        case <-onDemote:
            log.Println("I am no longer the leader")
        case err := <-onError:
            log.Println("election error:", err)
        }
    }
}()
elector.Start()
defer elector.Stop()

Opts.Redis is a redis.Scripter (satisfied by *redis.Client and *redis.ClusterClient), so the elector works against a Redis Cluster too.