Redis Key Layout

This is the complete map of every Redis key TaskRunner creates, read or writes. Use it to namespace multi-tenant Redis, debug with redis-cli, size memory, or reason about what gets cleaned up on a flush.

All keys are namespaced by ConsumerGroup unless noted. Substitute <group> with your configured TaskRunnerConfig.ConsumerGroup, and <prefix>:<queue> with your redisstream prefix/queue.

Streams & PEL

Key	Type	Purpose
<prefix>:<queue>	Stream	The main message queue. XADD on dispatch, XREADGROUP by fetchers, XACK/XDEL on completion.
stream:<...> PEL	(implicit)	The Pending Entries List attached to the consumer group. Not a separate key -- it's part of the stream's group state. XINFO STREAM <key> shows its size.

The consumer group itself is created lazily (XGROUP CREATE ... MKSTREAM) on the first XREADGROUP that hits NOGROUP. Consumers within the group are named <group>_<host>_<fetcherID> and cleaned up after 10 min of idle.

Delayed scheduling

Key	Type	Purpose
taskrunner:<group>:delayed_tasks	ZSET	Members are DelayedTask JSON; scores are run_at unix seconds. Leader polls every 5s with ZRANGEBYSCORE 0..now, dispatches each, and ZREMs.

Leader election

Key	Type	Purpose
taskrunner:leader:<base32(sha1(<group>))>	String	The leadership lease. Value = random instance ID. TTL = 5s (renewed every ~2.5s). Operated atomically via Lua (SET NX PX, value-guarded PEXPIRE, value-guarded DEL).

The key is a hash of the group name so it's stable across instances and free of special characters.

Worker tuning

Key	Type	Purpose
taskrunner:<group>:worker_tuning_capacity	String	Published by the leader with the chosen pool capacity. TTL = 20s (or LongQueueThreshold, whichever is greater). Followers read it via applyRemoteWorkerTuning.

Timing metrics

Key	Type	Fields
taskrunner:<group>:metrics	Hash	<taskName>_sum (cumulative ms), <taskName>_count (samples), delayed_tasks_schedule_sum, delayed_tasks_schedule_count. Reset to zero on MetricsResetInterval by the leader.

Replication detection

Key	Type	Purpose
taskrunner:replication:<host>	String	Per-instance heartbeat. Value = instance UUID. TTL = 15s, renewed every 10s. GetNumberOfReplications counts these keys (cached 10s) to derive the auto-detected replication factor.

Unique jobs

Key	Type	Purpose
taskrunner:unique:<taskName>:<uniqueKey>	String	Distributed mutex (redsync). Held from Dispatch until the worker's afterProcess. TTL = Task.UniqueFor seconds. Value = random lock token, verified on release.

When UniqueKey is nil, the key collapses to taskrunner:unique:<taskName>.

Queue metrics ring (backend)

Key	Type	Purpose
ring:<prefix>:metrics:<queue>	List	Circular buffer (capacity 1000) of per-message claim delays (nanoseconds), backing queue.GetMetrics. RPOPLPUSH rotates it.

Full map

graph LR
    subgraph "Per consumer group"
        S["{prefix}:{queue}<br/>Stream + PEL"]
        D["taskrunner:{group}:delayed_tasks<br/>ZSET"]
        L["taskrunner:leader:{hash}<br/>String (lease)"]
        W["taskrunner:{group}:worker_tuning_capacity<br/>String"]
        M["taskrunner:{group}:metrics<br/>Hash"]
    end
    subgraph "Per instance"
        R["taskrunner:replication:{host}<br/>String"]
    end
    subgraph "Per unique job"
        U["taskrunner:unique:{task}:{key}<br/>String (lock)"]
    end
    subgraph "Per queue (backend)"
        RG["ring:{prefix}:metrics:{queue}<br/>List"]
    end
    Producers --> S
    Producers --> D
    Leader --> D
    Leader --> S
    Leader --> L
    Leader --> W
    Leader --> M
    Followers --> W
    Followers --> M
    Instances --> R
    Workers --> U
    Workers --> M
    Backend --> RG

Flushing / cleanup

To wipe TaskRunner's state for a consumer group (useful in dev):

# Streams + group
XGROUP DESTROY <prefix>:<queue> <group>
DEL <prefix>:<queue>

# Delayed tasks
DEL taskrunner:<group>:delayed_tasks

# Tuning + metrics
DEL taskrunner:<group>:worker_tuning_capacity
DEL taskrunner:<group>:metrics

# Leadership (force a re-election)
DEL taskrunner:leader:<base32(sha1(<group>)>

# Replication (optional; will repopulate)
EVAL "return redis.call('del', unpack(redis.call('keys', ARGV[1])))" 0 "taskrunner:replication:*"

Don't flush in production

Flushing the stream drops in-flight messages. Flushing the leader key forces a re-election. Flushing unique locks can allow duplicate dispatch. Only do this in dev/staging.

Key count

For a deployment with G consumer groups, T registered tasks (across all groups), I instances, U in-flight unique jobs, and D delayed jobs:

streams:        G
ZSETs:          G
leader keys:    G
tuning keys:    <= G   (only present when tuning enabled + a value published)
metrics hashes: G
replication:    I
unique locks:   U
metric rings:   G

Even at moderate scale this is well under Redis's comfortable key limit (tens of millions). Memory is dominated by the stream length (bounded by deleteOnAck) and the PEL size (bounded by in-flight work).