Operations & Adoption

This is the page to read when you have been told to move your workers to TaskQ and want to know the footguns, the patterns to follow, and the antipatterns to avoid, before your first incident. It compresses the operational content of the whole guide set into one pass: what each knob actually does, how the sizing math works, how to shape fan-out jobs, and how to classify failures.

Mechanics live in the canonical guides (linked throughout); this page is about decisions.

Read order for a new adopter

1. This page, top to bottom.
2. retries.md: start_to_close vs schedule_to_close
3. deployment.md: Production Checklist
4. observability.md: setup

Know which TaskQ you are actually running

The published PyPI release can be far behind main: everything in the upgrading guide's Breaking API changes / Silent behaviour changes / Unreleased features sections (keyed rate-limit refs, cancel_where, enqueue tags, actor-config live tuning, the PG rate-limit backend on the dispatch path, dozens of operational fixes) exists only in git pins. A taskq-py>=0.2.2 range can silently resolve to the stale release, which carries known hazards the current docs no longer describe (for example: start_to_close failing to cancel the actor body, unique_for races, a payload-defect retry loop, and JSON results reviving UUID-shaped strings into UUID objects). If you pin a git rev, assert the installed distribution really resolves to it (a three-line test against importlib.metadata); if you use the release, read the docs for that release, not main.

---

Contents

1. The operating model in five minutes
2. Timeouts: start_to_close and schedule_to_close
3. Concurrency: process, actor, queue, fleet
4. Sizing: workers and Postgres connections
5. Fan-out at scale: chunks, cursors, idempotency
6. Classifying failures: terminal, retryable, transient
7. Waiting politely: rate limits, Snooze, RetryAfter, Retry-After
8. Observability and alerting
9. Porting workers from other queue systems
10. The footgun index
11. Adoption checklist
12. Scaling playbook: from signal to knob
13. TimescaleDB: plain Postgres or hypertables?

---

1. The operating model in five minutes

A worker is one asyncio process. It runs a producer task, exactly max_concurrency consumer coroutines, a heartbeat loop, a LISTEN/NOTIFY connection, and a leader-election loop. Scaling is processes, not threads: FOR UPDATE SKIP LOCKED makes concurrent workers safe against the same tables. See workers.md.

Queues route work. Every job sits in a queue; a worker only dispatches from the queues in its TASKQ_QUEUES. Partition fleets by queue to keep a deep backlog on one workload from starving another. See workers.md: Queue selection, and a queue with a deep strict_fifo backlog starves everything behind it, which is a priority problem, not only a partitioning problem.

One leader per schema. A schema-qualified advisory lock elects one worker to run the maintenance loops: sweeps (crash reclaim, deadline enforcement), scheduled-job promotion, cron. If the leader dies, another worker takes over at the next election. Nothing needs a dedicated process.

The database clock is the only arbiter. Every lease, deadline, and scheduled_at comparison uses Postgres clock_timestamp(). Worker clocks never decide anything, which is why absolute per-call schedule_to_close datetimes are deprecated in favour of interval budgets.

Delivery is at-least-once. A worker that dies mid-job has its jobs reclaimed after the lock lease expires and retried (or marked crashed if the budget is gone). Design actors to be idempotent; use the dedup tools in §5.

worker_group is a label, not a routing key. It is emitted as the OTel messaging.consumer.group.name span attribute and nothing else. It does not partition queues and carries no concurrency semantics. A "workgroup" in the deployment sense is the workgroup supervisor: a process orchestrator that spawns N child workers from one TOML file (workgroups.md). Note for managed-identity deployments: the stock supervisor spawns plain taskq worker subprocesses and cannot carry a --pg-credential-provider flag, so dual-auth-mode (DSN or Entra) deployments run their own supervisor, see managed-identities.md.

---

2. Timeouts: start_to_close and schedule_to_close

The two names sound alike and bound different things. Full mechanics: retries.md §7.

	start_to_close	schedule_to_close
Bounds	One attempt's wall-clock execution	The job's entire lifetime, all attempts
Type	timedelta (interval)	interval via actor retry.time_budget
Enforced by	asyncio.wait_for around the actor call	SQL guards at dispatch/retry/snooze + leader sweep
When it fires	Attempt is cancelled, recorded as TimeoutError, classified by the retry policy like any failure	Job fails permanently with error_class="DeadlineExceeded", but never kills a running attempt
Default	Unbounded	Unset (only kind="indefinite" actors get one, via time_budget)

Decide per workload class

Workload	start_to_close	schedule_to_close
Webhook / API call	30-60 s	RetryPolicy(kind="transient", max_attempts=5), no deadline needed
ETL chunk (see §5)	5-15 min	unset, or a generous time_budget (requires kind="indefinite"; time_budget is inert on any other kind)
Poll-until-ready (invoice, export)	short	unset: the actor Snooze loop owns timing
Batch finalizer	minutes	size generously; see the finalizer warning in §5

Set a fleet-wide safety net and override per actor:

# Every actor on this worker gets a per-attempt ceiling unless it declares its own.
export TASKQ_DEFAULT_START_TO_CLOSE=5m

from taskq import actor


@actor(start_to_close=timedelta(hours=1))  # opts this actor up
async def reindex_bucket(payload: Payload) -> None: ...

Gotchas

• There is no opt-out. The precedence chain is per-enqueue > @actor(...) > TASKQ_DEFAULT_START_TO_CLOSE. An actor that needs to run longer than the worker default must declare its own larger start_to_close; declaring None falls through to the worker default. Cron fires and admin "run now" currently do not propagate per-actor start_to_close (or schedule_to_close_interval); only the fleet-wide TASKQ_DEFAULT_START_TO_CLOSE protects those paths.
• A timed-out attempt consumes budget. attempt is incremented at dispatch, so a start_to_close timeout counts against max_attempts exactly like a raised exception.
• Sync actors are not stopped by start_to_close. def actors run via asyncio.to_thread; the timeout cancels the await, marks the job, and frees the slot, but the thread keeps running to completion, including its side effects. Poll ctx.should_abort() in sync actors (actors.md: Sync actors). The same applies to subprocess-spawning actors: a SIGKILL'd or force-exited worker leaves the child running: reap your own children.
• schedule_to_close never kills a running attempt. It gates future dispatches: expired pending jobs are never dispatched, retries/snoozes that would land past the deadline fail with DeadlineExceeded, and a leader sweep fails expired queued jobs. A running attempt that finishes after the deadline still succeeds.
• The enqueue-time heartbeat_timeout parameter reclaims silent holders. A running job whose holder stops heartbeating past its heartbeat_timeout is crash-reclaimed by the leader's sweep even while its lease (the global TASKQ_LOCK_LEASE, default 60 s) is still valid: the shorter of the two deadlines governs. Size it >= 2x the fleet's TASKQ_HEARTBEAT_INTERVAL; a value below one heartbeat interval reclaims a healthy job on a single missed beat. 2x is enough to absorb one transient beat because the heartbeat loop anchors its wait to each tick's start rather than its end, so a slow tick does not push the next beat out by its own duration, and because a failed tick is not a beat: it stamps nothing, so instead of also sleeping the full remaining interval it retries promptly on a bounded backoff (a quarter interval), keeping the gap between the good beats around one blip well inside the 2x floor. The same rule names the upper-bound trap: a heartbeat_timeout at or above TASKQ_LOCK_LEASE never governs: the lease deadline (last beat + lease) always precedes the heartbeat deadline (last beat + timeout), so the lease arm reclaims first and the per-job budget silently no-ops. Size it inside the window: >= 2x TASKQ_HEARTBEAT_INTERVAL and < TASKQ_LOCK_LEASE.
• The stored error_message of a genuine start_to_close timeout is the literal string "start_to_close": alert on error_class == "TimeoutError" if you want all timeouts, and remember an actor raising its own TimeoutError is indistinguishable by class.

How a worker death is recovered

lock_lease (60 s default) is the visibility timeout. The leader's sweep runs every sweep_interval (30 s default), so worst-case reclaim is ~90 s, then the job retries on its own RetryPolicy backoff curve (base, cap, backoff kind, jitter: the same curve an application-level failure would use, not a flat interval), or lands crashed if attempts are exhausted. Invariants that keep this safe: lock_lease >= max(heartbeat_interval, heartbeat_command_timeout) + (max_heartbeat_failures + 1) * (heartbeat_interval + heartbeat_command_timeout) (the worst coherent failed-beat cascade - the last good beat's tail plus the failed cycles - and the watchdog kills a stalled loop before its leases expire. Don't lower TASKQ_LOCK_LEASE without re-checking both (configuration.md: Validation Constraints).

The heartbeat does not rewrite a healthy lease every beat: it renews a held row only when the row's remaining lease drops to the renewal threshold, max(lock_lease / 2, max(heartbeat_interval, heartbeat_command_timeout) + (max_heartbeat_failures + 1) * (heartbeat_interval + heartbeat_command_timeout)), compared server-side on the clock that stamped the lease. Every term of that floor is enforced: the tick's whole command sequence (BEGIN, the writes, the cancel hook, COMMIT) runs under a single TASKQ_HEARTBEAT_COMMAND_TIMEOUT budget, its teardown (the bounded rollback-or-close) shares that budget's remainder rather than burning a second one, and the pool acquire is bounded by the heartbeat interval, so a failed beat costs at most interval + command_timeout, and the floor covers max_heartbeat_failures + 1 of them (the cascade the lock_lease invariant above exists to bound). At the default settings (60 s lease, 10 s interval, 2 s command timeout) the floor is 58 s, at or above what the lease has to harvest, so the renewal runs on every beat exactly as before; the rewrite savings apply from leases of roughly 70 s upward (2× at 70 s, 4× at 90 s, ~6× at 120 s; see perf-evidence-lease-renewal.md at the repo root for the sizing math, the enforced-bound measurement, and the fix-round history). A live worker's lease never comes closer to expiring than one worst beat-gap below the renewal threshold (threshold − interval − 2 × command_timeout, or lock_lease − that same gap on every-beat configs): renewal fires at or under the threshold, so the threshold itself is the renewal point, not the closest approach. Jobs carrying the enqueue-time heartbeat_timeout are renewed on every beat regardless, because the sweep's heartbeat arm reads their last_heartbeat_at freshness, not the lease.

---

3. Concurrency: process, actor, queue, fleet

Five different knobs are called "concurrency". Picking the wrong one is the most common misconfiguration:

Knob	Scope	Semantics
TASKQ_MAX_CONCURRENCY	per worker process	Hard bound on simultaneously executing jobs (default 8). One consumer coroutine each.
@actor(max_concurrent=N)	per actor, fleet-wide	Best-effort admission damper, not a hard cap (see below). Operator-tunable live. 0 = drain mode: no jobs of this actor dispatch, silently.
taskq queues set-max-concurrent	per queue, fleet-wide	Strict leased-slot cap. Read at worker startup; needs a restart to change.
ConcurrencyReservation	per bucket, fleet-wide, cross-actor	Strict leased slots in Postgres, declarable inline on the actors: the only strict mechanism that can serialize two different actors (e.g. against one thread-unsafe library). See rate-limiting.md.
@actor(singleton=True)	per actor, fleet-wide	At most one active job (pending/scheduled/running). Second enqueue raises SingletonCollisionError.

max_concurrent is not a hard cap

Dispatch reads the running-count once per round before locking rows. Two replicas dispatching concurrently each admit up to the cap; the over-dispatch bound is (num_producers - 1) * max_concurrent per round, and those jobs genuinely run. For memory- or GPU-bound actors use a strict leased-slot cap (per-queue or ConcurrencyReservation) instead. Full worked warning: deployment.md: max_concurrent and max_pending.

A reservation's lease must exceed the actor's worst-case runtime

Reservation slots are leased; a slot whose lease expired is acquirable by the next job. If an actor can run longer than its bucket's lease, two of them will hold one slot. Size the lease above your p-max runtime, not your p99.

When to add workers: saturation, not depth

Depth alone does not say whether the fleet is short of slots or the slots are idle. The capacity gauges answer it per process and per actor:

Signal	PromQL	Reads
Utilisation ≥ 0.9 for 10 m and oldest_pending_age rising	taskq_worker_active_jobs / taskq_worker_max_concurrency > 0.9 beside max by (actor) (taskq_jobs_oldest_pending_age_seconds) climbing	every slot is busy and pending work is ageing: add workers (or raise TASKQ_MAX_CONCURRENCY where the actors are I/O-bound).
Utilisation high, oldest_pending_age flat	as above, age not climbing	the fleet is busy but keeping up; no action.
Utilisation low, oldest_pending_age rising	... < 0.5 with the age climbing	slots are free and work still waits: not a capacity problem: an admission cap (max_concurrent, a reservation or rate limit: taskq_reservation_denials_total), an unserved queue (TaskQQueueUnserved), or a promotion stall (TaskQPromotionStalled). Adding workers changes nothing.
One actor holding the slots	sum by (actor) (taskq_jobs_running) dominated by one actor while others' oldest_pending_age climbs	that actor's max_concurrent (or a queue cap) is the lever, not the replica count.
Running age past the actor's p99 with timeouts flat	max by (actor) (taskq_jobs_oldest_running_age_seconds) beyond histogram_quantile(0.99, rate(messaging_process_duration_seconds_bucket[1h])) while taskq_jobs_timeouts_total{kind="start_to_close"} does not move	an actor with no start_to_close: the attempt has outlived what the actor normally takes and nothing will end it; declare the budget (see §2).

taskq_worker_active_jobs / taskq_worker_max_concurrency are one series per worker process (the health socket's taskq_active_jobs is the same number, but only a real scrape sees these two); taskq_jobs_running{actor} is the fleet-wide running count per actor, sampled by every worker beside taskq_jobs_by_status.

Capacity ownership: the seed-only trap

The @actor(max_concurrent=...) literal only seeds the actor_config row on first registration. Two production-shaped consequences:

• Stored rows win. Once a row exists, code changes to max_concurrent are ignored (logged as actor-config-capacity-override). Tune live without restarts:
• A stored NULL means uncapped, and pre-existing rows can null your literals. A schema first populated before an actor grew a capacity literal keeps max_concurrent NULL forever: every decorator literal in the repo is dead config, nothing fails, and the fleet runs uncapped. This has happened across whole fleets (rows seeded by an early deploy, literals added later, zero signal). Audit with:

taskq actor-config list    # treat any NULL capacity as a decision you never made
taskq actor-config set my_actor --max-concurrent 10

The exception is max_pending: a NULL stored value falls back to the code literal, so its declarations work on deploy. (max_pending rejects enqueues past the queued depth with MaxPendingExceededError, surfaced as taskq.backpressure.errors with kind="max_pending"; filter on the kind label, because the same counter also carries identity-serialization refusals (unique_for_lock_timeout, idempotency_lock_timeout) that are lock contention, not capacity pressure.)

Two viable ownership postures; pick one deliberately:

1. Operator-owned (TaskQ's default): literals are seeds; operators tune with the CLI; drift is logged. Requires someone to watch the log line.
2. Boot-time convergence: your startup code writes the literals into actor_config on every boot, making code the source of truth. An operator's live tune then reverts on the next deploy; document that.

The queue assignment is operator-owned once a row exists: move an actor between queues with taskq actor-config move-queue ACTOR NEW_QUEUE: the actor's pending/scheduled backlog moves to the target queue in bounded batches, then one final transaction flips the stored assignment and carries the old queue's mode/cap to the target (when the target has no row of its own), so old-queue strays drain through the target's consumers. Workers boot on either side of the matching code deploy (a stale literal logs actor-config-queue-override and adopts the stored queue); ensure workers consume the new queue, and keep consuming the old one until every producer runs the new literal. Running jobs the move leaves behind finish on the workers that claimed them; any that re-pend instead (failure retry, crash reclaim, operator retry) keep their old queue label as an audit trail but are routed at dispatch by the actor's current assignment, so the running-job tail drains through the target queue's consumers and never strands on the retired source queue. A mid-drain abort (timeout, lost connection) re-raises after the actor-queue-move-aborted event logs how many rows were already committed onto the target; re-running the move continues where it stopped. metadata is the remaining structural field: changing it after a row exists refuses to boot (ActorConfigDriftList); set TASKQ_FORCE_UPDATE_ACTOR_CONFIG=true for exactly one boot. Never leave it set. See workers.md: ActorConfig sync.

Starvation: priority and fairness

A deep backlog on one queue starves everything queued behind it, including small, urgent jobs, no matter how much concurrency you add. "No output from family X" with family X jobs sitting pending is head-of-line blocking, and it is fixed with ordering, not capacity:

• @actor(priority=N) / enqueue(priority=N): higher priority dispatches first. Priority is stamped at enqueue time: a standing backlog keeps whatever priority it was enqueued with, so after an incident you may need to bulk-UPDATE the backlog's priority (or cancel and re-enqueue) to promote the victim family.
• round_robin queue mode + fairness_key: interleaves dispatch across fairness_key cohorts (typically tenant or entity ids) so one tenant's 10,000-job sync cannot starve the others. Two traps: the mode is not the default and nothing in TaskQ sets it for you: run taskq queues set-mode <queue> round_robin once or seed the queues row in your bootstrap (otherwise fairness_key is a column TaskQ writes and never reads); and jobs enqueued without a fairness_key all collapse into one __null__ cohort where round-robin degenerates to plain FIFO; nothing warns you. See workers.md: Queue dispatch modes.

Thread-unsafe native libraries

A def actor dispatched through asyncio.to_thread runs on a real OS thread pool: several jobs in one process can drive one native library from several threads at once. Libraries that are not thread-safe (PDF rendering toolkits are a classic) will segfault the interpreter, kill every in-flight job in the process, and corrupt output. The fix is not a number in a config file; it is a strict ConcurrencyReservation with one slot shared by every actor that touches the library, across queues and worker processes. "Any safety property expressed as a number in a config file is one out-of-band edit away from being lost."

Sync actors are bounded by the thread pool, not by max_concurrency alone. A fleet of blocking def actors is limited by CPython's default executor (min(32, cpus + 4) threads per process), and each blocked thread still holds its job slot. Prefer async actors; for CPU work use a dedicated low-max_concurrency worker pool.

---

4. Sizing: workers and Postgres connections

Throughput math

Each worker process sustains, for one queue mix:

jobs_per_second ≈ max_concurrency / mean_job_seconds

I/O-bound actors multiplex cheaply (16-64 per process); CPU-bound actors block the event loop (2-4 per core). The workload table is in deployment.md: Concurrency tuning. Scale horizontally by adding processes; the fleet-wide caps from §3 still apply. Remember your own app-side pools: an actor that opens a DB session per job needs its pool sized to at least max_concurrency too; that counts against the same max_connections.

The connection budget

A worker holds up to four pools plus dedicated connections:

Resource	Count (default)	DSN	Notes
dispatcher_pool	4	direct	dispatch claims, sweeps, leader loops
heartbeat_pool	4	direct	heartbeat transaction every 10 s
worker_pool	int(max_concurrency * 1.5)	pooled (may traverse PgBouncer)	terminal writes, sub-enqueues, PG-backed rate limits
slot pool (conditional)	max_concurrency + 1	direct	per-slot transaction pool: only when a LOOP-scope asyncpg.Connection is registered and max_concurrency > 1; carries every per-job transaction (the actor's own writes via its injected slot connection, the terminal write, transactional sub-enqueues), one connection per consumer slot + one for the readiness probe, fully warmed at boot
notify_conn	1	direct	LISTEN (session-scoped; cannot pool)
leader_conn	1	direct	advisory-lock election (every worker)
leader-only monitor + cron	+2	direct	only on the elected leader

• Idle floor: pools open with min_size=1, so an idle worker holds 5 sessions (7 as leader), growing under load to the maxima.
• Default worker (max_concurrency=8): 10 direct + 12 pooled = 22 max; 24 on the leader.
• Per-slot path (LOOP-scope connection registered, max_concurrency > 1): add max_concurrency + 1 direct; at the default max_concurrency=8 that is 19 direct + 12 pooled = 31 max (33 on the leader).
• Per-TaskQ client (web/API pod): one pool, ≤ 5 connections.
• Workgroup supervisor with child health checks: health_workers + 1 extra.

Fleet formula:

direct  = M * (dispatcher_pool_size + heartbeat_pool_size + 2 + slot_pool) + 2   # +2 = the leader's extras
pooled  = M * int(max_concurrency * 1.5)
total   = direct + pooled + web_pods * 5 (+ supervisor health pool) + your app's own pools
# slot_pool: max_concurrency + 1 on the per-slot path, 0 otherwise

Worked example: 10 workers at max_concurrency=16, defaults elsewhere:

direct = 10 * (4 + 4 + 2) + 2 = 102    # 272 on the per-slot path (+ 10 * 17 slot connections)
pooled = 10 * 24              = 240
total  = 342  + your application's own connections    # 512 on the per-slot path

Check this against Postgres max_connections including the app's pools, and including the rolling-deploy doubling: orchestrators (Kubernetes rolling updates, Azure Container Apps revisions) bring the new pods up before draining the old ones, so your fleet briefly runs at ~2× steady state. Omitting that term is the single most common way this budget gets underestimated.

On the per-slot path there is a second doubling term: a credential rotation (SIGHUP / TASKQ_RELOAD_INTERVAL) rebuilds the slot pool, and for the drain window the worker briefly holds 2 × (max_concurrency + 1) slot-pool connections: the old pool draining plus the new pool warm, bounded by the reload drain timeout. Plan max_connections against that peak whenever rotations can coincide across the fleet (a scheduled TASKQ_RELOAD_INTERVAL is exactly that); against the steady state only when rotations are staggered.

The worker logs its own budget at startup (worker-startup-budget); the same arithmetic is taskq.worker.budget.compute_connection_budget(), which flags pgbouncer_recommended when the total exceeds 80 (conservative below the Postgres default max_connections=100). The startup line's direct count excludes the conditional slot pool; on the per-slot path the transactional_consume_per_slot startup event states its size (slot_pool_size). Small SKUs break that assumption: a 50-connection managed Postgres with ~15 reserved has ~35 usable and often cannot run PgBouncer at all: two default workers (44-48 connections) already exceed it. On small SKUs, lower TASKQ_MAX_CONCURRENCY (the derived worker_pool shrinks with it) until M × budget + app pools + rollout doubling fits.

Changing TASKQ_MAX_CONCURRENCY is a restart-only operation: SIGHUP does not re-read settings, and on the per-slot path the setting sizes the slot pool (max_concurrency + 1 direct connections, fully warmed at boot), so a change moves the direct-connection budget at the next startup. On that path it is also boot-blocking; a worker that cannot open the slot pool fails to boot, naming the pool and the DSN host.

PgBouncer: pooled DSN only

Only worker_pool may route through transaction-mode PgBouncer (TASKQ_PG_DSN_POOLED). LISTEN, advisory locks, and the sweeps require direct sessions (TASKQ_PG_DSN_DIRECT). Splitting the two DSNs is the supported pattern (see configuration.md and workers.md: PgBouncer compatibility).

Bring-your-own pools change the arithmetic

If you pass your own PoolFactory through WorkerConnections (managed-identities.md), TaskQ invokes it once per role: dispatcher, heartbeat, and worker pools become three distinct full-sized pools, not one shared pool, and a worker on the per-slot path adds a fourth, conditional pool: the per-slot transaction pool. That pool deliberately has no WorkerConnections slot (it is worker-internal, built by TaskQ on the direct DSN), so a fleet whose credentials live in its own pool factories must pass pg_credential_provider to worker_main / worker_main_async (the worker warns slot_pool_own_credentials at boot when caller-supplied pools are in play with no provider). Size and count accordingly.

Never hand TaskQ a pool your request handlers already use

asyncpg's Pool.acquire() has no default timeout. A request that holds one connection from a shared pool and then enqueues (which needs a second connection from the same pool) will, at pool_max concurrent requests, wait forever: a silent, permanent deadlock of the whole process, with green health probes. Give the TaskQ client its own small pool (≤5), always.

Managed identities and token rotation

With token-based auth (Entra ID/AAD, IAM, Vault), asyncpg pools keep the password they were built with: a worker that never rebuilds its pools looks perfectly healthy for the first hour and then cannot open a new connection. Size TASKQ_RELOAD_INTERVAL inside your token lifetime with margin (a common default is 40 minutes against ~60-minute tokens), or SIGHUP to force a reload.

A provider-backed slot pool rotates with the rest, keeping its boot-time size (TASKQ_MAX_CONCURRENCY is boot-only). In-flight jobs get the reload drain timeout to finish; past it the old pool terminates; the job's transaction rolls back, the row stays running, and lock-lease expiry reclaims and retries it: a loud, retryable infrastructure failure, never a false success. For the drain window's connection peak see the rotation note in §4.

Cover all five connection roles (dispatcher/heartbeat/worker pools, notify, leader) in your factories: a partial override yields a worker that starts, dispatches nothing, and never logs leader-elected. The slot pool is not one of them: it is worker-internal and takes its credentials from pg_credential_provider or the direct DSN. Full reference: managed-identities.md.

The notify reconnect lock's worst-case hold. A NOTIFY connection reconnect (health-check failure, or a SIGHUP credential reload) serializes on notify_reconnect_lock for the sum of every step inside it: the credential factory call (TASKQ_RELOAD_FACTORY_TIMEOUT, 30s default), each of the three channels' LISTEN execute and add_listener registration (TASKQ_NOTIFY_LISTENER_SETUP_TIMEOUT, 10s default, ×2 steps ×3 channels = 60s), and the failed connection's bounded close (5s): ~95s at shipped defaults. Every individual step is bounded, so this is never a hang, but for that whole span a concurrent SIGHUP reload reports the notify connection as failed and dispatch runs on the poll fallback instead of NOTIFY wakeups; to an operator watching a rotation that looks identical to a wedged worker. It resolves on its own; widening any of the three timeouts above raises this worst case by the same amount.

Database performance knobs

Four Postgres-side settings worth deciding deliberately: each closes a measured incident class.

jit = off for the queue role. During enqueue/dispatch churn the planner's cost model crosses jit_above_cost (measured at a planner cost of ~4.5M on a busy jobs table) and JIT compilation made dispatch 45× slower. Disable it once for the role TaskQ connects as:

ALTER ROLE taskq SET jit = off;

or per-pool, with no role-level access, by appending ?options=-c%20jit%3Doff (URL-encoded -c jit=off) to the DSN's query string. The repo's docker-compose.yml carries the equivalent commented block.

Statement-cache sizing: set for you. asyncpg caches prepared statements per connection in an LRU of statement_cache_size entries (asyncpg default: 100, evicted after max_cached_statement_lifetime seconds: 300). TaskQ's read paths render far more distinct SQL texts than 100 (list_jobs emits 384+ filter-combination variants, and the admin pages add more) so a client or dashboard whose filters vary thrashed the default cache: a measured 90-96 % steady-state miss rate, slower than disabling the cache entirely (each miss re-pays the Parse/Describe round trips, +10-40 ms/call at managed-PG RTT). Every pool TaskQ builds now passes statement_cache_size=512 and max_cached_statement_lifetime=3600, enough for the variant space with headroom, and no needless re-prepares on long-running workers. The write hot loop (dispatch, enqueue, terminal writes, sweeps) is textually stable and unaffected.

Both knobs are settings, not hard-coded constants: TASKQ_STATEMENT_CACHE_SIZE and TASKQ_MAX_CACHED_STATEMENT_LIFETIME (fields statement_cache_size / max_cached_statement_lifetime on TaskQSettings) flow into every pool TaskQ builds. 0 is meaningful on each: the cache disabled, or no lifetime cap. These knobs apply only to pools TaskQ builds: bring-your-own pools (WorkerConnections factories, a caller-supplied pool=) must set the same two create_pool kwargs themselves: TaskQ cannot resize a pool it did not build.

One deployment must set them to 0 regardless: behind a transaction-mode pooler (TASKQ_PG_IS_POOLED=true; PgBouncer pool_mode=transaction and friends remap server connections between statements, so a cached prepared statement's Parse and Bind can land on different server connections and fail with SQLSTATE 26000). TaskQ then forces statement_cache_size=0 and max_cached_statement_lifetime=0 on every pool it builds, overriding these two knobs, and treats the pooler-remap statement errors as transient.

Autovacuum overrides for the hot tables. jobs and job_events turn over hard; the default autovacuum_vacuum_scale_factor (0.2 of the table) lets bloat accumulate until vacuums are rare and huge. Per-table overrides, tuned for queue churn:

ALTER TABLE {schema}.jobs SET (autovacuum_vacuum_scale_factor = 0.01,
                               autovacuum_vacuum_insert_scale_factor = 0.05,
                               autovacuum_vacuum_cost_limit = 2000);
ALTER TABLE {schema}.job_events SET (autovacuum_vacuum_scale_factor = 0.01,
                                     autovacuum_vacuum_insert_scale_factor = 0.05,
                                     autovacuum_vacuum_cost_limit = 2000);

The insert scale factor is the one that matters for an append-heavy table: dead-tuple thresholds alone may never fire autovacuum (which also refreshes the statistics the planner needs).

fillfactor (70-90): consider, don't copy. Job rows are written once, then updated in place through their status transitions; a lowered fillfactor keeps those updates HOT (same page, no index churn). Try 80-90 on jobs only if profiling shows HOT-chain fragmentation; changing it rewrites the table (VACUUM FULL / pg_repack), so measure before and after.

---

5. Fan-out at scale: chunks, cursors, idempotency

The antipattern: one giant job

A "sync everything" job that runs for five hours:

• occupies a concurrency slot for the whole run and bottlenecks every pipeline behind it: nothing downstream proceeds until all the work succeeds;
• gets killed by start_to_close if you set one (and if you didn't, a hang holds the slot forever), then retried into the same death until max_attempts or schedule_to_close gives up;
• makes a resync repeat the entire five hours; the processing never proceeds.

Batch-shaped work must be chunked. Three supported shapes:

Pattern A: cursor chain (recommended default)

Process one page per job; each job enqueues the next page. Progress is durable, per-chunk retries are cheap, downstream consumers start as soon as the first chunk lands, and the queue drain rate is bounded by your concurrency/rate-limit settings instead of one job's runtime.

from datetime import timedelta

from pydantic import BaseModel

from taskq import JobContext, actor


class RowPayload(BaseModel):
    run_id: str
    row: dict  # whatever `fetch_page` returns per row


class SyncPagePayload(BaseModel):
    run_id: str
    cursor: int  # your own pagination cursor; TaskQ has none to pass for you


async def fetch_page(cursor: int, limit: int) -> tuple[list[dict], int | None]: ...


@actor(start_to_close=timedelta(minutes=10))
async def sync_page(payload: SyncPagePayload, ctx: JobContext[SyncPagePayload]) -> None:
    rows, next_cursor = await fetch_page(payload.cursor, limit=500)
    for row in rows:
        await ctx.jobs.enqueue(
            process_row,
            RowPayload(run_id=payload.run_id, row=row),
            # Stable per-row key: without this a parent retry re-enqueues every
            # child (autonomous sub-enqueues commit immediately and do not roll
            # back with the parent).
            idempotency_key=f"row:{payload.run_id}:{row['id']}",
        )
    if next_cursor is not None:
        # Stable key: re-enqueueing this page after any failure dedups instead of duplicating.
        await ctx.jobs.enqueue(
            sync_page,
            SyncPagePayload(run_id=payload.run_id, cursor=next_cursor),
            idempotency_key=f"sync:{payload.run_id}:{next_cursor}",
        )

Why the pieces matter:

• The cursor lives in the payload. TaskQ's cursor type is keyset pagination for read APIs (client.list(...)); there is no built-in job-continuation cursor. Your cursor is your data.
• The idempotency keys make re-dispatch safe. Both the per-row enqueues and the successor page carry stable keys. If the parent is retried after enqueuing children, the duplicates are dropped and the existing jobs returned (was_existing=True); the chain cannot fork and rows are not processed twice.
• Transactional vs autonomous sub-enqueue: know which one you are on. Two modes, decided by your DI setup, not by the call:
  • Autonomous (the default worker): no LOOP-scope asyncpg.Connection is registered, so ctx.jobs.enqueue commits each child immediately through the worker pool. A parent failure does not roll back its children; the startup log warns sub_enqueue_autonomous_fallback. The idempotency keys are your correctness backstop (a re-run parent re-enqueues the same children, which dedup). Registering an asyncpg.Pool at Scope.LOOP does not leave this mode: the transactional path keys on Connection, so a Pool-only registration silently keeps autonomous commit in force.
  • Transactional: register a LOOP-scope asyncpg.Connection and children join the parent's transaction (rollback on failure). At max_concurrency > 1 the worker opens a per-slot transaction pool: each job transacts on its own connection, and the actor receives that same slot connection by injection, so the actor's own writes join the transaction too, making transactional consume and throughput compatible at any concurrency. Keep the chaining actor on a dedicated TASKQ_MAX_CONCURRENCY=1 worker only when the registered LOOP-scope connection carries session state (SET ROLE, search_path, an RLS-driving GUC) that the transactional writes must inherit (on the per-slot path the actor's writes run on fresh slot connections that never carried that session state), or to keep the connection budget minimal (no max_concurrency + 1 slot pool).
• For a self-continuing poll loop (same job comes back), raise Snooze(delay) instead of enqueueing a successor; see §7.

Pattern B: fan-out batch + finalizer

Enqueue up to 1,000 children in one enqueue_batch(...) call (or stream chunks of 1,000) and gate completion with a finalizer that snoozes until all children are terminal. From inside an actor, pass an explicit batch_id and enqueue the finalizer as its own job:

from uuid import NAMESPACE_URL, uuid5

from taskq.batch import EnqueueItem


items = [
    EnqueueItem(actor_ref=process_chunk, payload=p, idempotency_key=f"chunk:{run_id}:{i}")
    for i, p in enumerate(chunks)
]
# Deterministic id: a parent retry must reuse the same batch id. `new_uuid()`
# (UUIDv7) mints a fresh id per attempt, so deduped children keep attempt one's
# batch id while the finalizer points at an empty batch id, and
# `wait_for_batch` then returns `is_complete=True` on the wrong id. `uuid5`
# keeps the id stable across attempts; the finalizer's own idempotency key
# dedups the second enqueue (`was_existing=True`).
batch_id = uuid5(NAMESPACE_URL, f"taskq-batch:{run_id}")
await ctx.jobs.enqueue_batch(items, batch_id=batch_id)
await ctx.jobs.enqueue(
    finalize_run,
    FinalizePayload(batch_id=batch_id, expected=len(items)),
    idempotency_key=f"finalize:{run_id}",
)

Always UUIDv7, never uuid4(), except retry-safe in-actor batch ids

TaskQ generates every internal id (jobs, workers, batches) as UUIDv7, time-ordered, so B-tree index inserts stay local and ids sort by creation time. Use new_uuid (exported from taskq) for any id you choose yourself, such as an explicit batch_id; unless the batch is enqueued from inside an actor that may retry, in which case derive the id deterministically from the run (e.g. uuid5) as above so every attempt reuses it.

The finalizer= parameter exists on the client's enqueue_batch (client.enqueue_batch(items, finalizer=...)); the in-actor SubJobEnqueuer.enqueue_batch takes only batch_id, so enqueue the finalizer separately as above. The finalizer's wait_for_batch(db, batch_id) raises Snooze on a 10 s interval until the batch completes, then runs once. Worked example: examples/actors/batch.py and jobs-clients.md: Batch finalizer.

Size the finalizer's budget generously

A snooze whose now + delay would land past schedule_to_close fails the job with DeadlineExceeded instead of rescheduling. A finalizer that snoozes for hours needs either no deadline or (on a kind="indefinite" actor; time_budget is inert on any other kind) a retry.time_budget that comfortably exceeds the expected batch duration (plus retries). Note a default-kind (transient) finalizer that snoozes has neither a deadline nor an attempt ceiling (a snooze refunds the attempt instead of consuming it), so make it indefinite with a budget if you want it bounded.

Pattern C: app-level run accounting + finalize sweep

When completion spans generations of jobs (a chunk enqueues chunks), a single batch_id cannot cover the chain. Keep a durable counter on your own run/entity rows: increment at enqueue (gated on was_existing; deduplicated enqueues must not double-count), decrement when a job lands terminal (in the actor, or via on_retry_exhausted), and run a periodic sweep cron that force-closes runs whose counter has been nonzero too long. This also catches wedged runs that a pure finalizer would wait on forever.

Choosing the enqueue API

API	Size	Idempotency	Notes
enqueue_batch	≤ 1,000	per-item keys honored; same-actor collisions return existing jobs, a collision with another actor's job refuses the whole batch (IdempotencyKeyActorMismatchError)	single transaction; max_pending enforced per actor; over-cap actors' items refused (typed BatchMaxPendingExceededError naming the actor + item indices), everyone else's admitted
enqueue_batch_streaming	unbounded (chunks of ≤ 1,000)	per-item keys honored	generator input; max_pending enforced per chunk with the same per-actor partition; with no caller connection each chunk is its own committed transaction; a failure leaves the committed prefix durable (retry via idempotency keys or the error's refused indices)
enqueue_batch_fast	≤ 50,000	none: any duplicate key aborts the whole COPY with DuplicateIdempotencyKeyError (IdempotencyKeyActorMismatchError when the holder is another actor's)	bulk-import semantics; returns a count only; max_pending enforced per actor (same partition; COPY stays all-or-nothing on constraint violations)

See jobs-clients.md for the full tradeoff table.

Choosing a dedup mechanism

Mechanism	Semantics	Use when
idempotency_key (+ idempotency_scope)	Exact. DB unique index; duplicates return the existing job, no error	re-enqueue after failure/crash; exactly-once scheduling
unique_for + identity_key	Windowed, single-flight. A transaction-scoped advisory lock serializes the preflight-then-insert per identity on pool and bare-caller connections alike, so concurrent enqueues dedupe against the winner's row	suppressing duplicate triggers inside a freshness window
singleton=True	One active job per actor	"never two of me at once" (mind the cron interaction below)

Idempotency-key discipline: every rule below was learned from a production incident:

• There is no TTL. A key dedups against any row that still exists; the horizon is retention: 30 d for succeeded rows, 90 d for failed rows by default. Need a time window? Encode it in the scope: idempotency_scope=f"daily-sync:{date}".
• Keys are status-blind. A terminal-failed job's key absorbs every re-post onto the dead job: your post-fix rerun of a failed batch will silently no-op. Encode an attempt or window axis (key=f"...:{attempt}", or bump a version in the key when the entity's state changes).
• Keys must encode every distinguishing payload dimension. TaskQ cannot see payload contents: a key that carries only one dimension of a composite identity collapses two different work items onto one job, and the second item's work silently never runs: work loss, not duplication.
• No stable key on a self-continuation successor. A continuation job carrying the same key as its (now-succeeded) predecessor collides with it and is silently dropped. Successors of the same actor need either a fresh key dimension (the next cursor) or no key.
• Namespace client-supplied keys. A raw user-supplied string as the whole key lets one caller squat another caller's dedup slot; mix the caller identity into the key.
• Length cap: 1024 bytes (TASKQ_IDEMPOTENCY_KEY_MAX_BYTES); oversize keys raise at enqueue. Hash over-cap values (keep legible keys below the cap).
• unique_for without identity_key at enqueue is a silent no-op (logged warning).
• unique_for on a thin root does not single-flight its fan-out: a root that finishes in seconds frees the window while its child chain still runs; a cron cadence faster than the chain will overlap runs. Single-flight the work, not the root (a running-run guard in your own state, or a strict reservation).
• metadata.batch_id is library-reserved and stripped from caller-supplied metadata; a caller-supplied metadata.singleton=True on an ordinary actor is not stripped: it survives onto the row and triggers real single-flight behaviour (enqueue preflight and the jobs_singleton_uniq index key on it).
• For fleet-wide completion reactions (rather than a finalizer), the durable-cursor pattern TaskQ.watch_reclaims() is documented in architecture.md: which component drives each transition.

tags are the group-operation tool: enqueue(..., tags=[f"run-{run_id}"]) then JobFilter(tags=(...)) for status polling and bulk cancel_where. Tags must match \A\w(?:[\w\-]*\w)?\Z (no colons), and invalid tags raise at enqueue, so a bad tag factory fails every trigger, not just one.

Cron and scheduled workloads

• scheduled_at is timezone-aware only (naive datetimes raise), and promotion runs on the leader's ~1 s tick; that's your timing precision.
• Cron fires on schedule regardless of the previous fire. A cron actor slower than its cadence piles up overlapping runs. Overlap suppression options:
  • singleton=True on the actor: a fire landing while the previous fire is still active is suppressed, not failed: no consecutive_failures strike, no auto-disable. Safe for actors that routinely outrun their cadence.
  • identity_key on the schedule + unique_for on the actor: fires inside the window dedup silently (no strike). The window must cover the cadence, and it single-flights the root only (see the dedup rule above).
  • An app-level running-run guard is the robust option for chains that outrun their cadence.
• Misses past the catch-up window are dropped, and counted. A fire overdue by more than TASKQ_CRON_CATCH_UP_WINDOW is skipped: the recompute lands next_fire_at in the future and the dropped occurrences are never replayed. The drop is instrumented, not silent — taskq.cron.skipped_slots (per actor, in fire units), the skip depth on taskq.cron.slots_behind, and skipped_slots on both the cron missed slots skipped warning and the cron fired line. A sustained skip rate is the runaway fan-out (jobs clearing slower than the cron period): the TaskQCronSkippedSlots alert pages on it, and the fires-vs-clearance SQL in taskq.insights (src/taskq/insights.py) names the schedule.
• Decorator registration is create-only. @cron(...) creates the schedule row on first sight and never updates it; a changed expression, timezone, or a removed decorator has no effect on an existing row. Manage live schedules with client.update_schedule / delete_schedule (an empty expression does not disable a previously-registered schedule).
• Auto-disable is permanent until re-enabled: check taskq.cron.disabled_schedules and the cron schedule auto-disabled log line (kind="cron_fire"; cron.auto_disabled is only an OTel span event, not a log event); a silently-disabled schedule is an outage with one log line.
• Give cron actors a start_to_close: an unbounded cron actor that hangs holds its queue. Note cron fires currently do not propagate per-actor start_to_close (only the fleet-wide TASKQ_DEFAULT_START_TO_CLOSE applies), so the safety net above is what protects cron.

---

6. Classifying failures: terminal, retryable, transient

TaskQ has no exception subclass that marks "permanent". You classify by configuration, in this order (first match wins):

1. non_retryable_exceptions=(MyError,) on the actor: isinstance match, subclasses included.
2. Auto-fail classes: PayloadValidationError, ResultTooLarge, pydantic ValidationError (a re-run reproduces the same problem).
3. The actor's RetryPolicy.kind: non_retryable → fail; transient → retry while attempt < max_attempts; indefinite → retry until schedule_to_close.
4. Per-instance refinement: a retry_classifier hook returning RetryOverride(kind=..., delay=...).

Doctrine: put each failure class where it belongs:

Failure class	Classification	Why
Malformed payload, schema violations, bad arguments	terminal (PayloadValidationError is auto-terminal)	deterministic: retries reproduce it
Auth failures, permission denied, "unknown resource" for the job's whole subject	terminal via non_retryable_exceptions or classifier kind="non_retryable"	the condition will not heal
Per-item faults inside a fan-out (one deleted file in a 500-item page)	per-item fault, skip and continue, not a job failure	one unreadable item must never fail the whole page/run; record it and move on
Downstream 429 / Retry-After	transient with the server's delay: RetryAfter or classifier RetryOverride(delay=...)	retrying now makes it worse; retrying then succeeds
5xx, timeouts, connection resets	transient with bounded max_attempts	healable, but bound the budget
"Down until Tuesday" (maintenance window)	Snooze (no budget consumed) or kind="indefinite" + time_budget	known-duration waits should not burn attempts

Size the retry window to span a routine provider blip. RetryPolicy(max_attempts=4, base=timedelta(seconds=10)) covers only ~70 s of cumulative delay (10 + 20 + 40 jitter-free); an ordinary object-store 5xx blip outlasts that, exhaustion is terminal, and (if your on_retry_exhausted marks entities dead) the entity silently disappears from downstream surfaces. For network-bound actors, prefer fewer attempts with a longer base over many fast ones.

Multiplicative jitter does not decorrelate a synchronized cohort. The default ±20% around a 5 s base re-clusters a fleet that failed together into a ~2 s window at T+5s; at large fan-out that is a synchronized retry wave. Disperse cohorts with a longer base and higher jitter, or shed load with max_pending before the wave forms.

Never catch a failure and return a result

An actor that catches the provider's 429/5xx and returns (ok=False, retryable=True) has succeeded as far as TaskQ is concerned: the job is succeeded, the retry policy, the classifier, the advertised Retry-After, and on_retry_exhausted never engage. Raise, or translate to your typed retryable exception and raise that.

There is also no poison state and no dead-letter queue. A job whose budget is exhausted lands in terminal failed with error_class MaxAttemptsExceeded / DeadlineExceeded / your exception's class, and stays queryable in jobs until retention prunes it. Routing to a real DLQ is your job, at the two designed hook points: on_retry_exhausted (per actor) and the ErrorReporter (per worker, e.g. Sentry forwarding). Note: on_retry_exhausted receives a bare JobRow with no DI scope: hooks must build their own connections and cannot inject. See retries.md: on_retry_exhausted and observability.md: Error reporting.

Two more states worth naming because they mean infrastructure, not your code:

• crashed: the worker died (or lost its heartbeat) mid-attempt with the retry budget exhausted. Crash labels are WorkerCrashed (assumed gone) and HeartbeatLost (alive but partitioned).
• abandoned: only ever produced by cancellation escalation: a force-cancel whose cleanup did not finish within the grace periods after an operator's cancel request. A timeout or exception never produces abandoned, and shutdown never produces it either: a deploy releases (interrupts) the job back to the fleet instead, the spent attempt standing.

Infra failures during the terminal write leave the job running, on purpose

If Postgres itself errors while recording the outcome, the write is retried for about a second (waits of 50, 200 and 800 ms, four attempts, within a 5 s wall-time budget); if it still has not landed, the row is left running and reclaimed by lease expiry (~90 s worst case with defaults) rather than misclassifying an infrastructure blip as your actor's failure. Don't be alarmed by a short-lived running row on a dead worker; the sweep owns it. The reclaim re-runs the attempt: the actor's work finished but its outcome was never recorded, so this is the at-least-once delivery case that does not involve a worker crash. Actors whose side effects are not naturally idempotent should guard them with idempotency_key / unique_for on the jobs they enqueue and idempotent writes of their own (§5).

A routine SIGTERM drain sets the same cancel event an operator cancel does

ctx.cancel_event / ctx.should_abort() fire during every rolling deploy's drain phase. What the row records is the actor's own outcome: if your actor treats "cancelled" as "close the run as cancelled and return", the job records succeeded (it returned normally), no retry runs, and successor enqueues never happen; a chain whose actor returns early on cancel dies on every deploy. Read ctx.cancel_origin to tell the deploy apart from the operator: on SHUTDOWN the attempt is released and re-run by the fleet with the interrupted attempt spent, so checkpoint via progress state and re-raise; on OPERATOR the job terminalises, so the partial result you return is the one kept. See cancellation.md: Shutdown is not an operator cancel.

---

7. Waiting politely: rate limits, Snooze, RetryAfter, Retry-After

Rate-limit denial is a snooze, not a failure

When an actor's rate limit or reservation denies admission, the actor body never runs: the job is rescheduled at now + retry_after (computed by the limiter store in the DB/Redis clock domain), the slot is freed, and no retry budget is consumed. A denial is admission control with HTTP-429 semantics ("come back later", indefinitely retryable), not an execution, so it writes no job_attempts and no job_events row: per-denial rows grow without bound under sustained contention, so contention is carried by the aggregated rate_limit_blocked_count on the job row (snooze_count for an actor-requested deferral), while metadata.awaiting names the bucket being waited on. A denied job waits in Postgres; this is not busy-spinning in the worker.

A denial never terminally fails a job either: the job reschedules until capacity frees, and its only terminal exit is its own schedule_to_close deadline, reached through the ordinary deadline path. To find jobs starving for admission rather than progressing, sort by rate_limit_blocked_count.

from taskq.ratelimit import SlidingWindow, TokenBucket, registry

# Downstream allows 50 req/s fleet-wide (`capacity` is the burst allowance,
# `refill_per_second` is the sustained rate):
registry.register(
    TokenBucket(name="partner_api", capacity=50, refill_per_second=50.0, backend="redis")
)


@actor(rate_limits=["partner_api"])
async def call_partner(payload: Payload) -> Result: ...

Per-tenant / per-entity quotas use keyed refs; the bucket name is derived per payload key:

from taskq.ratelimit import KeyedRateLimitRef


@actor(
    rate_limits=[
        KeyedRateLimitRef.typed(
            Payload,
            base_name="partner_per_tenant",
            key_fn=lambda p: p.tenant_id,
            capacity=10,
            refill_per_second=0.5,
            backend="redis",
        ),
    ]
)
async def call_partner_per_tenant(payload: Payload) -> Result: ...

Scope matching: rate_limits is per actor. A quota shared across several actors (one provider, many callers) needs a single shared bucket referenced by all of them; the limit registry is global, so register the bucket once and list it on every actor that spends it. This is separate from your HTTP-gateway rate limiting (429s at the edge); TaskQ limits govern background consumption.

Backends: redis (default for TokenBucket/SlidingWindow; requires TASKQ_REDIS_URL; the worker fails fast at startup if a Redis-backed limiter is served without Redis), postgres (needs only the standard TaskQ schema), or memory (per-process, tests only). On Redis connection errors the limiter falls back to the PG implementation by default (TASKQ_RATE_LIMIT_PG_FALLBACK_ENABLED). Full reference: rate-limiting.md.

A saturated reservation does not tax its neighbours

Dispatch is reservation-aware at claim time. The claim statement reads live slot occupancy in reservation_slots (which actors' running jobs currently hold live slots, and how many slots in those buckets remain acquirable) and folds it into per-actor admission: an actor whose held bucket is full is admitted nothing that round, and a partially free held bucket admits at most its free count (times the dispatch oversample). Without this gate, a saturated actor's pending jobs were claimed into the worker's shared TASKQ_MAX_CONCURRENCY consumer pool, denied by the post-claim acquire, and snoozed, every cycle, for as long as the backlog and the exhaustion both lasted, each denied job occupying a consumer slot while it was denied (measured: 25-45% throughput loss for a healthy actor sharing the worker; after the gate, none (see perf-evidence-dispatch.md).

What you observe instead:

• The saturated actor's rows stay pending (never claimed), so they appear in the per-actor backlog series (taskq.jobs.actor_backlog / taskq.jobs.oldest_pending_age_seconds) as what they are: work waiting on capacity. A saturated actor is more visible, not less.
• When capacity frees (a holder finishes, or its lease expires: the same acquirability definition the acquire path uses), the gate re-opens on the next dispatch round and the actor drains. The gate is a damper, never a starvation mechanism: the first claim of a never-running actor is never gated, so capacity can always be taken.
• The post-claim acquire remains the admission authority. A claim-to-acquire race (a peer takes the last slot in between) degrades to one bounded denial cycle per race, with the 429 semantics above intact: no retry budget spent, no per-denial rows, rate_limit_blocked_count bumped.

Because the gate derives pressure from holder state (reservation_slots.job_id → the holding job's actor) rather than from actor declarations, it covers static reservations, keyed reservations, and the per-queue concurrency cap (queues.max_concurrent) alike. Two boundaries to know: a bucket shared across different actors gates only the actors currently holding it (a co-declarant with nothing running is discovered by the ordinary post-claim denial path), and token-bucket / sliding-window rate limits are not read by the gate at all (their denials are discovered post-claim and snoozed at the limiter's retry hint, the behaviour described above).

A drained fixed-quota bucket repolls every 5 s, forever

A TokenBucket with refill_per_second=0 never refills and returns retry_after=None, so denials fall back to the 5 s default backoff. Without a schedule_to_close, such a job re-queues every 5 s indefinitely. Either give the bucket a refill rate, or give the actor a deadline (retry.time_budget requires kind="indefinite").

Dev/prod Redis asymmetry

A common compose/dev setup sets TASKQ_REDIS_URL while production deliberately runs Postgres-only. A Redis-backed limiter added in dev then refuses to start the production worker: fail-fast at bootstrap, naming the limiters. Decide the backend per environment, not per habit.

The three "come back later" signals

Signal	Consumes budget	Reschedules at	Use for
raise Snooze(delay)	No: the claim's attempt increment is released, so attempt returns to what it was and max_attempts never moves	now + delay	waiting for a condition (batch completion, external state)
raise RetryAfter(delay)	Yes (default); bounded by max_attempts, terminal MaxAttemptsExceeded when out	now + delay	a retry that should wait a known time (429s)
raise RetryAfter(delay, consume_budget=False)	No (snooze semantics)	now + delay	known-delay wait that must never exhaust the budget

A deferral on a row carrying an operator's cancel request is refused (noop) and the cancel ladder terminalises the row; a deferral whose schedule_to_close lapses on such a row terminalises cancelled, never DeadlineExceeded: operator intent outranks the deadline.

A classifier RetryOverride(delay=...) is the type-level variant: it applies to a whole exception class and is clamped by max_retry_backoff; RetryAfter's delay is not clamped (only schedule_to_close gates it).

Honoring Retry-After (and x-retry-after) headers

TaskQ does not automatically honor any HTTP header; there is no setting or integration that does this. The supported pattern is to read the header yourself and translate it into a signal:

from taskq import RetryAfter, RetryPolicy, actor


@actor(retry=RetryPolicy(kind="transient", max_attempts=8))
async def call_api(payload: Payload) -> Result:
    resp = await http_post(payload.url)
    if resp.status == 429:
        # Standard header (seconds) or your gateway's x-retry-after; same translation.
        wait = float(resp.headers.get("Retry-After") or resp.headers.get("x-retry-after") or 60)
        raise RetryAfter(delay=timedelta(seconds=wait))
    if resp.status in (404, 410):
        raise UnknownResource(payload.url)  # listed in non_retryable_exceptions
    return parse(resp)

For per-status-code policy (429 → wait the header's delay, 404 → terminal, 5xx → bounded transient), use the retry_classifier example in retries.md §5; it carries the header value through RetryOverride(delay=...) and inherits the max_retry_backoff clamp, so a malformed Retry-After: 999999999 cannot strand the job.

Scheduled re-enqueue as the "snooze until it can run" fallback

When the wait is longer than any job should hold a slot, don't snooze: enqueue a successor with a scheduled_at (timezone-aware; promotion tick ~1 s) plus an idempotency_key, and return success. The job queue is your timer, and dedup makes the pattern crash-safe.

---

8. Observability and alerting

Wiring an exporter

TaskQ emits via the OpenTelemetry API and never overrides standard OTel vars. Two supported shapes:

• OTLP (collector, Jaeger, Tempo, Datadog, Sentry, PostHog): install taskq-py[otel] and set OTEL_EXPORTER_OTLP_ENDPOINT (:4317 gRPC / :4318 HTTP), OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES. taskq worker installs the SDK providers from those variables at startup and logs otel-exporter-configured traces=otlp metrics=otlp source=env (the console renderer's shape; under the default TASKQ_LOG_FORMAT=json the same fields ride the otel-exporter-configured JSON event) (opentelemetry-instrument taskq worker is the equivalent launcher; an embedded worker calls taskq.obs.configure_exporters(settings) or initializes the SDK in-process ( examples/otel_setup.py).
• Vendor SDK in-process (e.g. Azure Monitor / App Insights): call the vendor's configure_azure_monitor(...) inside the worker process before it starts; the worker logs otel-exporter-preconfigured and leaves the vendor's providers alone.

A vendor connection string on the container does nothing by itself

APPLICATIONINSIGHTS_CONNECTION_STRING on a worker container exports nothing unless the worker process itself calls the vendor's configure function; it is not one of the standard OTEL_* variables the worker wires from. The standard variables set without the [otel] extra export nothing either; the worker says so at startup (otel-exporter-unavailable, naming the extra). Verify the startup line and that telemetry actually arrives (one trace, one metric series) before trusting the pipeline.

Every job emits an enqueue PRODUCER span, a process CONSUMER span (linked, with lifecycle.* events), and an attempt.N child. Prometheus scraping and vendor receivers: observability.md and deployment.md: Observability Setup.

The metrics that catch the failure modes in this page

Metric / signal	Catches
taskq.queue.depth (by queue; counts pending and scheduled)	backlog growth, starved queues, fan-out storms
taskq.jobs.by_status (by status; pending and scheduled reported separately)	which side of promotion the backlog sits on
taskq.jobs.actor_backlog (by actor and queue; pending only; exact below the sampler's 1000-row per-pair cap, reading 1000 at/above it)	the unconsumed actor: one actor's series rises while its queue-mates stay flat, invisible in any queue-summed view
taskq.jobs.queue_wait_seconds (histogram by actor and queue)	what dispatched jobs actually waited; a rising p99 with flat depth is dispatch starvation (priority/fairness, admission caps), not backlog
taskq.jobs.oldest_pending_age_seconds (by actor and queue)	the same condition as a head-of-line age growing with wall clock; the series TaskQQueueDepthHigh fires on
taskq.jobs.oldest_due_age_seconds	how long the oldest due scheduled job has waited for promotion
taskq_maintenance_leader_sweep_last_success_seconds (by sweep)	per-sweep stalls: a sweep that stops completing
taskq_maintenance_leader_sweep_timeouts_total (by sweep)	batches aborted by deadlines or server-side cancels, and gauge-sampler reads that did not complete; an absent or frozen gauge beside a rising rate is a dead sampler, not a resolved alert
taskq_worker_loop_unexpected_errors_total (by loop)	a long-lived loop's backstop (UnexpectedLoopErrorGuard) caught a non-transient error — the leader maintenance loops, the worker producer loop, the progress flush loop; anything above zero warrants investigation
taskq.queue.live_workers (by queue, same tick as depth)	a queue with work and no live worker; TaskQQueueUnserved joins it against taskq.queue.depth
taskq.jobs.stranded (by actor and reason)	jobs that can never dispatch: no_actor_config (no actor_config row) or unserved_queue (no live worker on the routing queue)
taskq.dispatch.duration	dispatch contention (PgBouncer/pool trouble)
taskq.jobs.oldest_running_age_seconds (by actor)	an attempt that outlived what the actor normally takes with taskq.jobs.timeouts flat; an actor with no start_to_close
taskq.worker.active_jobs / taskq.worker.max_concurrency (one series per process) and taskq.jobs.running (by actor)	slot saturation: utilisation ≥ 0.9 with oldest_pending_age rising is "add workers"; which actor holds the slots (see §3: When to add workers)
taskq.worker.slot_pool.acquire_failures / taskq.worker.slot_pool.connections_in_use	per-slot pool exhaustion and saturation; an acquire failure is infrastructure (the job is left for lock-lease reclaim, not failed); the gauge pinned at the pool maximum with zero acquire failures is saturation, visible below the acquire-failure cliff
messaging.process.duration (by outcome)	actor latency, slow chunks; filter outcome="succeeded" for the healthy percentiles
taskq.jobs.attempt_failures (by actor, error_type, retryable)	a dependency failing under retry cover (retryable="true" rising while the terminal-failed share stays flat); the series TaskQRetryRateHigh fires on
taskq.jobs.abandoned (by actor)	an actor that ignores cancellation: an operator cancel outlasted both graces; never produced by a deploy
taskq.lock.expires_in_seconds	heartbeat trouble before it becomes crashed jobs
taskq.jobs.timeouts (by actor and kind)	start_to_close hits per attempt; schedule_to_close hits however the deadline was enforced (sweep or handler arm); taskq.deadline_exceeded_sweep.jobs_failed is the sweep's own count
taskq.backpressure.errors (filter kind to the capacity kinds)	max_pending rejections: producer pressure
taskq.cron.disabled_schedules	a cron outage with one log line
taskq.cron.skipped_slots (by actor) + taskq.cron.slots_behind (depth, fire units)	the runaway fan-out: a cron-triggered sync whose jobs clear slower than the cron period — the leader is dropping fire slots past the catch-up window, counted, never replayed; run the fires-vs-clearance SQL in taskq.insights (src/taskq/insights.py) to name the schedule
taskq.maintenance_leader.is_leader summed != 1	leader split-brain / no leader

When metrics are unavailable or suspect, query the tables, not the logs; during one production outage the console logs looked clean while the worker was down:

-- Replace {schema} with your TASKQ_SCHEMA_NAME.
SELECT id, worker_label, last_seen_at,
       last_seen_at < clock_timestamp() - interval '60 seconds' AS stale
FROM {schema}.workers ORDER BY last_seen_at DESC;

SELECT queue, status, count(*) FROM {schema}.jobs
WHERE status IN ('pending','scheduled','running')
GROUP BY queue, status ORDER BY count(*) DESC;

The per-actor backlog sampler: every worker, every interval, depth-bounded

The two per-(actor, queue) series above are fed by the backlog-detection sampler, which runs on every worker every TASKQ_QUEUE_DEPTH_INTERVAL (default 15 s), deliberately not leader-gated, unlike the maintenance sweeps: a detector hosted behind the leadership gate would emit nothing under exactly the failure it exists to expose (an election loss or a stuck lock mutes every leader-gated loop at once). Your scrape target deduplicates the identical per-worker series.

Because N workers pay the read each interval, the read itself is bounded so its cost never grows with backlog depth: a loose index scan over jobs_actor_dispatch_idx enumerates the distinct pending (actor, queue) pairs (one index seek per pair, never one per row), and a per-pair probe reads at most 1000 rows in dispatch-head order (priority DESC, scheduled_at). One tick costs at most (#pairs holding pending work) × 1000 row visits (shaped by the deployment's actors and queues, never by how deep a backlog gets) and touches no terminal rows. Measured on a single 100k-deep pair (local Postgres 18): ~14 ms and ~6000 shared buffers per tick before the bound, ~0.4 ms and ~100 buffers after, flat against the 10k case (see perf-evidence-dispatch.md).

What the cap does to the numbers you read:

• taskq.jobs.actor_backlog is exact below 1000 pending for a pair and reads 1000 at or above it: a lower bound, never an under-count. A pair pinned at the cap is "at least this deep"; dashboard over it with that meaning.
• taskq.jobs.oldest_pending_age_seconds is the age of the oldest row within the pair's dispatch-head sample: exactly the oldest pending row whenever the pair is below the cap (or priorities are uniform), and in every case the head-of-line age an unconsumed actor grows without bound, so the TaskQQueueDepthHigh alert's semantics are unchanged by the cap.
• When you need the exact depth of a pair at/above the cap, query the table directly: SELECT count(*) FROM {schema}.jobs WHERE status='pending' AND actor='...' AND queue='...'; the right tool for a one-off investigation, which is exactly what a fleet-wide every-15-s sampler cannot afford to be.

Watch: large scheduled backlogs

A very large set of due scheduled jobs drains one bounded batch per one-second tick (TASKQ_EVENT_WRITER_BATCH_SIZE rows, default 100, each batch its own committed transaction with a server-side statement_timeout). Promotion always makes progress, but a backlog larger than the drain rate still grows, and the fleet still looks healthy from the outside while it does (workers heartbeat normally, dispatch reports count: 0). An earlier design promoted the whole due set in one transaction under a single dispatcher_command_timeout deadline, which wedged into timeout-rollback-timeout past a backlog cliff; the bounded batches replaced it (see maintenance-sweeps.md). The detection guidance below catches any failure mode that grows the scheduled backlog. Symptoms: scheduled depth grows, pending stays empty, throughput is zero, and health probes stay green. Detect it before it bites:

SELECT count(*) FROM {schema}.jobs
WHERE status = 'scheduled' AND scheduled_at <= clock_timestamp();

Alert on that count, on taskq.jobs.oldest_due_age_seconds, and on the TaskQScheduledBacklogGrowing / TaskQPromotionStalled rules in src/taskq/contrib/prometheus/rules.yaml (runbooks: runbooks.md). Prevent it with max_pending on high-fan-out actors, retry policies that disperse cohorts (longer base, higher jitter; see §6), and bounded fan-out per job (chunk sizes in the hundreds, not the tens of thousands).

Alert rules

Ship-ready alert rules for the metrics above exist in the repo and are ready to import: src/taskq/contrib/prometheus/rules.yaml (24 rules: queue depth, heartbeat misses, terminal-failed share, retried-failure share, abandoned jobs, lock TTL, leader split-brain, dispatch latency, claim-latency degradation, progress failures, disabled cron, scheduled-backlog growth, promotion stall, sweep timeouts, sweep unexpected errors, sweep degraded tier, maintenance-lock contention, rate-limit dependency outage, cron lock contention, cron budget deferrals, cron skipped slots, unserved queue, stranded jobs, expired-lease zombies) and the equivalent PrometheusRule CRD at src/taskq/contrib/kubernetes/prometheus_rule.yaml. Importing them is not enough; make sure something scrapes the workers (TASKQ_METRICS_PORT, every pod; see deployment.md: Prometheus scrape): the rules read worker-side series the admin process's /jobs/health/metrics never carries. The scrape listener binds nothing until the port is set, refuses startup (exit 1) if it cannot bind or configure, and serves an unauthenticated endpoint: pin it to the pod network with a TASKQ_METRICS_HOST=127.0.0.1 sidecar scraper, or scope it by network policy. Per-platform env blocks are in deployment.md: Listener deployment recipes.

---

9. Porting workers from other queue systems

TaskQ's core concepts and their operational equivalents for teams moving from background job systems:

Concept	TaskQ feature	Notes
Concurrency model	TASKQ_MAX_CONCURRENCY	asyncio coroutines, not processes; CPU-bound work must move to threads/executors
Soft timeout per attempt	start_to_close	cooperative cancellation of the attempt
Hard process kill per job	n/a	no per-job process kill; see the sync-actor caveat in §2
Visibility timeout for dead workers	lock_lease (60 s) + leader sweep	worker death → reclaim → retry
At-least-once delivery	default behavior	lease reclaim on worker loss
Backoff algorithm	RetryPolicy(max_attempts, backoff, base, cap, jitter)	jitter is ±20% multiplicative, not full jitter, and it does not decorrelate synchronized cohorts (see §6)
Retry configuration	default (all exceptions retry under the policy)	list the terminal ones instead: non_retryable_exceptions
Rate limiting per actor	TokenBucket / SlidingWindow on the actor	fleet-wide, Redis- or PG-backed; keyed refs for per-tenant quotas
Delayed scheduling	scheduled_at	timezone-aware; ~1 s promotion precision
Time budget for entire job lifetime	retry.time_budget → schedule_to_close (time_budget requires kind="indefinite")	interval from enqueue, server clock
Deduplication of reruns	idempotency_key (+ scope)	DB-enforced; duplicates return the existing job; mind the key-discipline rules in §5
Job dependency chains (many-to-one)	batch finalizer + wait_for_batch, or app-level run accounting	§5 Patterns B/C
Job workflows (chaining)	ctx.jobs.enqueue(...) from the actor body	transactional on a LOOP-scope conn (per-slot connections make it safe at any max_concurrency; session-state inheritance still needs a single-slot worker), autonomous otherwise; see §5
Task routing	@actor(queue=...) + worker TASKQ_QUEUES	routing is by queue, not by broker exchange
Job priority	@actor(priority=...) / enqueue(priority=...)	higher dispatches first; stamped at enqueue; see §3
Per-tenant fairness	round_robin mode + fairness_key	the mode must be set explicitly; see §3
Worker labeling	worker_label (+ worker_group OTel label)	labels only; no routing semantics
Scheduled job execution (cron)	@cron(...)	fires on schedule regardless of the previous fire; overlap suppression has traps; see §5: Cron
Dead-letter queue routing	on_retry_exhausted + ErrorReporter	no built-in queue; route where you want it

---

10. The footgun index

The condensed "know this before your first incident" list. Each row links to the deep dive.

Configuration & routing

Footgun	Symptom	Fix / deep dive
worker_group expected to route or partition	jobs "not reaching" a group	it's an OTel label only; route with queues (§1)
Worker consumes no queue an actor enqueues to	jobs sit pending forever	check actors-on-unconsumed-queues warnings; see workers.md
API and worker on different TASKQ_SCHEMA_NAME	every job stuck pending, no error anywhere	one source of truth for the schema, refuse-don't-default
round_robin expected but never set (or no fairness_key)	fairness silently off; __null__ cohort = plain FIFO	set the queue mode; key every enqueue; see §3
TASKQ_MIGRATE_ON_START=true expected to run in the worker	warning migrate-on-start-ignored-by-worker	run migrations in a pre-deploy job / admin UI (deployment.md)
taskq queues set-max-concurrent changed, nothing happened	cap unchanged	queue caps are read at worker startup; restart workers
@actor(max_concurrent=...) added in code, no effect	actor-config-capacity-override log	stored actor_config wins; NULL = uncapped; audit with taskq actor-config list (§3)
Actor's metadata changed after first registration	worker refuses to boot (ActorConfigDriftList)	one-boot TASKQ_FORCE_UPDATE_ACTOR_CONFIG (§3); a queue change is a move, not drift; taskq actor-config move-queue
.env files expected to lose to (or beat) process env	config resolves "backwards"	dotenvmodel 1.x: process env beats .env unless DOTENV_OVERRIDE=true (configuration.md; and mind the loading CWD
Actor registry passed as a generator (or generated actors missing from it)	zero/n actors registered	pass a mapping; test that every actor name is in the registry; see workers.md: Actor registry

Timeouts & retries

Footgun	Symptom	Fix / deep dive
No start_to_close anywhere; an actor hangs	slot held forever	set TASKQ_DEFAULT_START_TO_CLOSE + per-actor overrides (§2)
Expecting schedule_to_close to kill a running attempt	long attempt survives past deadline	it only gates future dispatches (§2)
start_to_close expected to kill a sync actor's thread	job marked timed out, side effects continue anyway	sync actors keep running; poll ctx.should_abort() (actors.md)
heartbeat_timeout set below one heartbeat interval	healthy jobs reclaimed on a single missed beat	size it >= 2x TASKQ_HEARTBEAT_INTERVAL: the per-job analogue of the lock_lease >= 4 x heartbeat_interval invariant (§2)
heartbeat_timeout at or above the fleet's TASKQ_LOCK_LEASE	the per-job budget silently never governs: the lease deadline (last beat + lease) always precedes the heartbeat deadline (last beat + timeout), so the lease arm reclaims first and the knob no-ops; the reclaim event names cause='lock_expired' directly	size it inside the window: >= 2x TASKQ_HEARTBEAT_INTERVAL and below TASKQ_LOCK_LEASE (§2)
Retry window shorter than routine provider blips	terminal exhaustion on an ordinary 5xx	fewer attempts, longer base (§6)
Snoozing finalizer with a tight time_budget	DeadlineExceeded mid-batch	size the budget to batch duration + retries; time_budget requires kind="indefinite" (§5)
max_retry_backoff raised, long backoffs still capped	retries at 24 h ceiling	the ceiling is min(policy.cap, TASKQ_MAX_RETRY_BACKOFF) (retries.md)
Catching failures and returning a result	job succeeded, retry machinery never engaged	raise; see §6
Actor returns normally during a drain-cancel	job records succeeded, chain dies on every deploy	drain sets the same cancel event; never return normally on cancel (§6)
Cron actor overlap "suppressed" with singleton=True	slot skipped, no strike, next due tick re-evaluates	suppression semantics; see §5: Cron

Fan-out & dedup

Footgun	Symptom	Fix / deep dive
One giant "sync everything" job	pipeline blocked for hours; killed by start_to_close; resync repeats	chunk it; see §5 Pattern A
No idempotency_key on chunk jobs	retries fork the chain, duplicate work	stable business key per chunk (§5)
Stable key on a self-continuation successor	continuation silently dropped (collides with succeeded predecessor)	fresh key dimension or no key (§5)
Key missing a payload dimension / re-posting after terminal failure	work silently lost / rerun no-ops onto the dead job	keys are payload-blind and status-blind; encode dimensions and attempts (§5)
unique_for without identity_key	dedup silently off	pass identity_key at enqueue; see actors.md
unique_for on a thin root expected to single-flight the chain	overlapping runs despite the window	single-flight the work, not the root (§5)
enqueue_batch_fast with duplicate keys	whole COPY aborts	pre-dedup or use enqueue_batch (§5)
Blind retry after BatchMaxPendingExceededError	duplicate jobs: the within-cap actors' items were already committed	every enqueue tier enforces max_pending per actor; retry only the error's refused item indices, or use idempotency_keys (§5)
Huge synchronized scheduled cohort (mass retry wave)	promotion stalls; health green; throughput zero	disperse cohorts, max_pending, scheduled-depth alert; see §8
Tag factory emitting colons	every enqueue 500s	tags must match \A\w(?:[\w\-]*\w)?\Z; see §5

Runtime & infrastructure

Footgun	Symptom	Fix / deep dive
Everything through transaction-mode PgBouncer	LISTEN/advisory-lock failures	only worker_pool may pool; split the DSNs (§4)
Fleet sized without connection math	too many connections under load	per-worker 22 (default), rollout ×2: the budget formula (§4)
BYO PoolFactory counted as one pool	three full-sized pools (plus the conditional per-slot pool on that path), budget blown	factory is invoked once per role (§4)
TaskQ client handed the request handlers' pool	silent whole-process deadlock at pool_max	dedicated small pool; see §4
Token auth without reload	healthy for an hour, then cannot connect	TASKQ_RELOAD_INTERVAL inside token lifetime, all five roles (§4)
NOTIFY reconnect mid-rotation looks like a hang	dispatch on poll fallback, SIGHUP reload reports notify as failed, for up to ~95s	notify_reconnect_lock's worst-case hold is the sum of its bounded steps, not a hang; see §4
terminationGracePeriodSeconds < shutdown worst case	SIGKILL mid-drain, crashed jobs	grace ≥ cancellation + cleanup + ~42 s tail (deployment.md)
DDL run against live tables mid-deploy	brief InvalidCachedStatementError (0A000) in worker logs	transient by design: the pool re-prepares next tick, no restart (deployment.md)
Migration hangs "waiting for lock" and the whole fleet's queries freeze with it	a queued ACCESS EXCLUSIVE parks every later statement on the table behind itself	the runner's 30 s ddl_lock_timeout gives up cleanly (MigrationLockTimeoutError); find the holder, re-run migrate up (deployment.md)
One job longer than the shutdown budget	watchdog force-exits; sibling in-flight jobs die too	size the grace to your slowest actor, or cap it with start_to_close (§2)
Actors longer than cancellation_grace_period + cleanup_grace_period	interrupted on every deploy and re-run from scratch (interrupt_count climbs, work restarts); only schedule_to_close ends the loop	bound them with schedule_to_close, or checkpoint via progress state and resume on re-claim (§6, cancellation.md)
Drained refill_per_second=0 bucket, no deadline	job re-queues every 5 s forever	add refill or schedule_to_close (§7)
Redis in dev, none in prod	worker refuses to start in prod only	decide the limiter backend per environment (§7)
Worker without a Redis client expected to publish progress	SSE/progress silently empty, no error	progress fanout needs Redis; PG snapshotting still works; see progress.md
max_concurrent=0 set as a "disable"	actor silently stops dispatching; jobs pile up pending	0 is drain mode: use it deliberately, drain with it, then restore
Redis-backed limiter without TASKQ_REDIS_URL	worker refuses to start	fail-fast by design: configure Redis or use backend="postgres" (§7)
Suppressing asyncio.CancelledError in an actor	cancel phase escalates to abandoned	never catch it; see cancellation.md
Two workers pinned to the same TASKQ_HEALTH_SOCKET_PATH	second worker's health server fails	the default is per-process unique (/tmp/taskq_health_<pid>.sock); give explicitly pinned paths unique values
leader_lease below 4 * heartbeat_interval	no error, no boot refusal	silently raised to the floor: the same lock_lease >= 4 x heartbeat_interval invariant, honoured rather than enforced, so raising heartbeat_interval alone raises the effective lease instead of shortening the renewal margin (architecture.md)
Cancel expected to survive a crash-retry	retried attempt runs uncancelled	retry arms reset cancel state by design; re-cancel if still needed
Exporter env var set, telemetry expected	nothing collected, silently	exporter must be configured in-process; see §8
PyPI release assumed current	fixed bugs recur; documented features missing	check the CHANGELOG, pin a rev, assert the pin (see the intro warning)

---

11. Adoption checklist

Everything to verify before pointing production traffic at a new TaskQ fleet. (The deployment mechanics of each item: deployment.md: Production Checklist.)

• Version: git pin asserted by test, or release docs matched to the release (see the intro warning)
• Migrations run as a pre-deploy step (the worker ignores TASKQ_MIGRATE_ON_START); schema name is one source of truth shared by API and worker
• Timeouts: TASKQ_DEFAULT_START_TO_CLOSE set as a safety net; every long-runner declares its own start_to_close; every kind="indefinite" actor has a time_budget
• Failure classification: terminal exception types listed in non_retryable_exceptions (or classified via retry_classifier); per-item faults skip-and-continue; default RetryPolicy() (3 attempts) reviewed per actor; no actor catches failures and returns
• Dedup: batch-shaped work chunked (§5); chunk/continuation jobs carry stable, dimension-complete idempotency_keys; self-continuations carry a fresh key dimension
• Concurrency: TASKQ_MAX_CONCURRENCY matched to workload class; strict caps (queue or ConcurrencyReservation) where a soft damper is not enough; thread-unsafe libraries serialized by a shared reservation
• Capacity audit: taskq actor-config list; every actor's max_concurrent/max_pending is the value you decided (NULL = uncapped); an ownership posture chosen (operator-owned vs boot-time convergence)
• Queue modes: round_robin + fairness_key set where tenants share queues; priorities assigned to latency-sensitive actors
• Cron: every schedule registered by this deploy still intended; actors have start_to_close (note cron fires currently ignore per-actor values; the fleet-wide TASKQ_DEFAULT_START_TO_CLOSE is what protects cron); overlap strategy chosen with its trap understood (§5: Cron)
• Connection budget: computed for the target fleet size against max_connections (including app pools, BYO factory fan-out, rollout doubling, and the per-slot rotation peak where that path is in use); pgbouncer_recommended checked
• DSN split: TASKQ_PG_DSN_DIRECT (worker core) vs TASKQ_PG_DSN_POOLED (worker_pool)
• Credentials: reload interval inside token lifetime; all five connection roles covered
• Shutdown budget: supervisor/terminationGracePeriodSeconds ≥ cancellation_grace + cleanup_grace + ~42 s (default model: 82 s), and ≥ your slowest actor
• Health probes: taskq health live/ready wired (exec probes; TCP TASKQ_HEALTH_PORT only where httpGet is forced); unique socket path AND unique probe port per process (a bind collision fails the listener loudly but the boot continues; the orchestrator's failed probes are the fail-closed backstop)
• Metrics scrape: TASKQ_METRICS_PORT set on every worker (off unless set; scrape endpoint unauthenticated, so pod network, loopback sidecar via TASKQ_METRICS_HOST, or SG-scoped); a bind or config failure exits the worker, so a bad port shows up at deploy
• Redis provisioned iff using Redis-backed limiters or real-time progress; PG fallback decision made; dev/prod symmetry checked
• Observability: exporter wired in-process and verified to arrive; the alert rules from §8 imported and scraped; someone watches taskq.queue.depth, scheduled depth, and taskq.backpressure.errors
• DLQ routing decided: on_retry_exhausted / ErrorReporter target chosen (there is no built-in dead-letter queue)
• Idempotency of actor bodies audited; delivery is at-least-once, including crash-reclaim re-dispatch

---

12. Scaling playbook: from signal to knob

Symptom-first: what the operator actually sees (an alert, an admin UI view, a metric series), the knob(s) that address it, and how to confirm the knob worked. This table only routes: the arithmetic behind each row is in the section it links; do not re-derive it here. Incident response (what to do while it burns) lives in runbooks.md; this section is about which knob to reach for once the fire is out. Admin routes referenced below are in admin-ui.md.

You see	First knob(s)	Confirm it worked	Failure mode of over-tuning
Actor saturated: taskq_worker_active_jobs / taskq_worker_max_concurrency > 0.9 sustained while taskq.jobs.oldest_pending_age_seconds climbs; /admin/actors shows the actor pinned at its cap. No shipped alert: this one is a dashboard check.	taskq actor-config set ACTOR --max-concurrent N (live; the stored row wins over the code literal; see §3 seed-only trap); TASKQ_MAX_CONCURRENCY or more replicas when no per-actor cap binds (the math: §3 when to add workers, §4 throughput)	Utilisation drops below 0.9 and oldest_pending_age flattens on the other actors first	Pushing max_concurrent high "to use the fleet" removes the damper that was protecting a downstream dependency; adding replicas when a strict queue cap or ConcurrencyReservation is the real bound buys idle pods
Worker CPU / event-loop lag: watchdog tier-1 loop-lag warnings with faulthandler dumps; /admin/workers shows beats arriving late. No shipped alert. The stall ATTRIBUTION names the actor: taskq_worker_loop_stall_attributions_total (and the worker's Stall hotspots column) tells you which actor's synchronous code is blocking the loop, and as blocking_call (released the GIL: move it to a thread) or gil_held (held the GIL: chunk it or move it off the loop)	More worker processes (asyncio is one thread per process; CPU-bound actors block the loop, §3); lower TASKQ_MAX_CONCURRENCY per process. TASKQ_WATCHDOG_LOOP_LAG_WARN_BUDGET / TASKQ_WATCHDOG_LOOP_LAG_BUDGET only after the host is provably not loaded	Lag warnings stop without having touched the budgets	Raising the lag budgets converts a designed trip-and-restart into a wedged worker that keeps serving slowly; the terminal detector exists so the supervisor restarts you
Queue depth growing, live workers idle: TaskQQueueDepthHigh fires while utilisation stays low; taskq.jobs.queue_wait_seconds p99 rises with flat depth; /admin/queues/{queue} shows depth sitting on one queue	Queue mode + fairness: taskq queues set-mode <queue> round_robin and fairness_key on the hot producers (§3 starvation); the actor's max_concurrent damper or a reservation holding slots; TASKQ_DISPATCH_OVERSAMPLE when identity collisions or many dispatchers poll the same actor+queue	queue_wait_seconds flattens while depth stays flat, then drains	oversample above the number of dispatchers that routinely poll the same actor+queue enlarges every candidate window for nothing; round_robin set without fairness_key collapses to FIFO with zero signal
Pending backlog with stranded > 0: TaskQStrandedJobs; taskq.jobs.stranded carries the reason (no_actor_config / unserved_queue); /admin/actors misses the actor	No worker knob fixes the rows: register the actor on a worker consuming that queue (TASKQ_QUEUES / --queues), or taskq actor-config move-queue (§3 capacity ownership). TASKQ_STRANDED_JOBS_INTERVAL is detection cadence only	The stranded gauge returns to 0 per actor	Shortening the interval polls Postgres more often and recovers nothing; it is a detector period, not a repair loop
Rate-limit denials: taskq_reservation_denials_total climbing; /admin/rate-limits and /admin/reservations show saturated buckets; TaskQRateLimitDependencyOutage when the PG fallback path is the one struggling	Limiter rates and reservation capacity (application config, not env; see §7); TASKQ_MAX_KEYED_RESERVATIONS / TASKQ_MAX_KEYED_RATE_LIMITS when denials are cardinality-driven; TASKQ_TOKEN_BUCKET_LOCK_TIMEOUT_MS / TASKQ_SLIDING_WINDOW_LOCK_TIMEOUT_MS when fallback row locks are the bottleneck	The denials counter flattens and queue_wait_seconds resumes	Widening the lock budgets makes a saturated bucket wait longer instead of denying: latency replaces refusals; adding workers changes nothing because admission sits upstream of the worker
Cron piling up at tick: TaskQScheduledBacklogGrowing / TaskQPromotionStalled with scheduled depth growing; /admin/schedules shows the fire history; TaskQCronLockContention on the lock	Spread cron expressions across minutes rather than aligning every schedule at * * * * *; TASKQ_CRON_TICK_LIMIT for a catch-up burst larger than one tick plans (§5 cron, §8 watch); TASKQ_CRON_CATCH_UP_WINDOW bounds what a restart replays	Scheduled depth drains one bounded batch per tick and taskq.jobs.oldest_due_age_seconds falls	The tick limit is consumed by the single leader; raising it moves the pile downstream into dispatch instead of clearing it faster; only one process pays the knob no matter how many replicas run
terminal-write-failed logs / disowned jobs: the worker fails every attempt of a terminal write, disowns the row (lease stops renewing), and TaskQRunningLeaseExpired catches the reclaim; usually beside TaskQHeartbeatMisses or TaskQLockExpiringSoon	Not actor config: database health. PgBouncer saturation, max_connections exhaustion, a failover (§4 connection budget). Fix the budget (TASKQ_MAX_CONCURRENCY down, a pooled DSN for worker_pool), never the actor	terminal-write-failed stops and no rows sit running past their lease	Raising TASKQ_MAX_CONCURRENCY grows the derived worker_pool and makes the exact connection pressure that caused the writes to fail worse
p95 duration far above p50: messaging.process.duration{outcome="succeeded"} tail; /admin/history sorted by started_at shows the long-running jobs	Review the outlier actors' start_to_close (§2) and chunk sizes (§5); TASKQ_DEFAULT_START_TO_CLOSE as the fleet-wide safety net; fairness_key when one cohort causes the tail	p95 tracks p50 and taskq.jobs.timeouts stays flat	A fleet-wide start_to_close sized above everyone's p95 silently kills legitimately slow attempts (and sync actors keep running past it anyway; only the slot is freed)

Two rows in this table are deliberately "no knob": stranded rows and terminal-write failures are routing and infrastructure conditions, and both failure modes above where an operator reached for a per-process number to fix a fleet-wide condition are the ones §3 and §4 exist to prevent.

---

13. TimescaleDB: plain Postgres or hypertables?

TaskQ's retention tables (jobs_archive, job_attempts_archive, job_events) are plain Postgres by default, and that is the right default for most fleets. On a server with the TimescaleDB extension they can convert to hypertables with Timescale-managed retention policies and a columnstore on the two archive tables (TASKQ_TIMESCALEDB_HYPERTABLES=true, opt-in at the taskq migrate up deploy step). The conversion mechanics, the DDL, and the retention-interplay rules are timescaledb.md's subject. This section is the decision and the day-2 operations, built from the repo's own measured A/B runs (benchmarks/timescale_tradeoffs.py and benchmarks/timescale_compression.py, results in benchmarks/results/timescale-*.json; every number below is read off those artifacts).

The mode dimension: three first-class storage modes

Every trade in this section sits on a dimension the section used to leave implicit: the detected storage mode. TaskQ detects it from the connected server (never from settings) — vanilla (no extension installed), timescale-apache (extension installed, license apache), or timescale-tsl (extension installed, full Timescale-license feature set) — and taskq doctor leads every report with the detected mode and its capability consequences. The full mode × capability matrix, cited from the detection code, is timescaledb.md's support matrix and mirrored in deployment.md. The cells that matter to the decision framework below:

• Compression exists in exactly one mode. The columnstore — and therefore every compression number in this section (the 6.20x storage reduction, the 4.6x/9.3x cold-read wins, the decompression GUC prerequisite, the first-tick DML-decompression catch-up) — is timescale-tsl only. Under the timescale-apache license the server refuses every compression API (measured on 2.30.1), and vanilla has no extension at all: in both, the archive tables stay rowstore.
• Policy-driven chunk-drop retention also exists in exactly one mode. The retention policies — and therefore the policy floor, the faster drains, the first-run aged-tail drop, and the watermark gap — are timescale-tsl only too: the apache license refuses add_retention_policy alongside the compression APIs (measured on 2.30.1).
• In timescale-apache and vanilla, retention is the row-level sweeps, full-range — bounded batch deletes, expire_at honored exactly, the event-prune watermark kept on every deletion. The two modes differ in one cell only: the apache mode's hypertables (faster recent-history reads, chunk partitioning) with no policy machinery on top.

So when reading the decision table below: its "hypertable" column is the timescale-tsl mode's behavior; if your server detects timescale-apache, take the table's read column (where it cites hypertable reads — those hold, the partitioning is Apache-licensed) but the plain column for every retention-drain and compression row.

The decision table

Your situation	Decision	The measured evidence
Archive below ~10,000 rows	Stay plain.	10k is the smallest swept scale, and no measured advantage for the hypertable justifies the machinery there: the archive page is already 4.9× faster, but at that size it is single-digit milliseconds either way. Plain is the simplest machinery and keeps every row-exact deletion guarantee.
Big archive + read-heavy admin surfaces	Hypertables win clearly.	The admin archive tab's newest-first page at the 1M-job corpus: 8.6 ms vs 45.0 ms plain (~5.2×) in the committed run (timescale-tradeoffs.json; plain p95 49.1 ms — the plain side carries the run-to-run spread, the hypertable side held 8.6 ms across runs). Since the retention-policy floor, the retention drains run FASTER than plain: event TTL 0.04 s vs 0.30 s (~6.8×), archive expiry 0.32 s vs 1.14 s (~3.6×). The columnstore shrinks archive storage −84 % (6.18×) on the 400k-row corpus.
Very high throughput fleets	Mind the prune→archive move — it is a flat ~8× slower on the hypertable.	Per 100k jobs moved: 4.5 s plain vs 35.9 s hypertable at the 1M corpus. Scale-invariant above ~100k rows: the drain-total penalty is 7.8× at 100k, 8.0× at 400k, 8.1× at 2M, and the per-batch p50 carries the same ~8-9× at both 400k and 2M. The DELETE half of the move runs on the never-converted jobs table in both modes, so the entire penalty is the INSERT half — chunk routing and per-chunk index maintenance — which is inherent to the feature: no setting removes it.
Forensic-grade audit needs	Read the guarantee gradient first.	On hypertables, expire_at is honored exactly inside chunk lifetime (the expiry sweep stays row-exact there), but the aged end is chunk-dropped silently, watermark-blind: the policy advances no pruned_through_id, so a stale watch_reclaims cursor loses events with no gap signal — keep the cursor strictly inside event_retention_period (or one chunk interval, whichever is larger). The re-archive fold guard can lose its witness: "archived at most once" holds only while the archive chunk lives. And job_attempts_archive's dropped cascade means parentless attempt rows stay queryable up to one chunk interval after the parent's chunk drops. Vanilla keeps all four guarantees; hypertables trade them for chunk-drop retention.

Two write-path facts that do not decide the trade, but bound it: enqueue parity holds (enqueue_batch_fast within ~3 % — jobs is plain in both modes), and the hypertable's job_events insert pays ~6 %. There is no write cliff; the penalty that matters is the prune→archive move in the third row.

The shape of the scale curve: read/write mix decides, not scale

The scale sweep (10k / 100k / 400k / 2M archive rows, fresh identical container pair per scale) measured both curves:

• The read win exists from ~10k rows and decays slowly. The archive tab's newest-first page is faster on the hypertable at every swept scale: 4.9× at 10k, 8.2× at 100k, 7.2× at 400k, 3.2× at 2M. It never crosses over — it narrows.
• The write penalty saturates at a flat ~8× above 100k rows. The prune→archive drain-total penalty goes 4.9× at 10k → 7.8× → 8.0× → 8.1×, and the per-batch p50 sits at the same ~8-9× at both 400k and 2M. Above ~100k rows the penalty is a constant ratio, not a growing one.

So the decision above ~10k archive rows is your read/write mix: deployments whose admin surfaces read recent history heavily win on the hypertable at any size; deployments that only write the archive and rarely read it pay the flat insert tax for nothing. Live-table reads are structurally untouched either way (jobs is never converted; the live pages measured at parity). One recorded exception to re-check against your own fleet: the all-statuses count read 51.3 ms vs 14.1 ms on the hypertable engine at 1M — a planner artifact on a table that is plain in both modes, so treat a count anomaly there with suspicion, not as an engine cost.

Planning the enabling deploy

1. Size the window for the constraint scans and the migrate_data copy together. Both take ACCESS EXCLUSIVE: the copy rewrites existing rows into chunks, and before it runs the constraint surgery (the FK drop, the pkey drops, every ADD CONSTRAINT ... UNIQUE) each takes ACCESS EXCLUSIVE and full-scans its table to validate the uniqueness it asserts. On a big archive those scans are part of the deploy window, not a footnote to it.
2. Deploy in a maintenance window when no worker archives. The pkey drop and the unique add run one per transaction on the deploy connection's autocommit, and the migration advisory lock serializes migrators only — workers keep running through the whole conversion. Between a table's pkey drop and its unique add, that table holds no unique constraint on its id columns; a duplicate insert landing in that window makes every subsequent deploy fail on the unique add until the duplicates are removed by hand. The window is two adjacent statements — tiny and bounded, but not zero.
3. Size the post-deploy window for the first policy run. Every chunk older than the retention interval is drop-eligible the moment the enabling deploy finishes, so the first background policy run drops the whole aged tail in one sweep — potentially gigabytes of IO where vanilla mode expires the same rows gradually.
4. Raise the decompression budget before relying on the columnstore. timescaledb.max_tuples_decompressed_per_dml_transaction defaults to 100000, and at that default the archive-expiry sweep hard-errors on compressed chunks with ConfigurationLimitExceededError (measured: one 10k-row expiry batch decompressed 356633 tuples). The deploy probes the setting and logs hypertable-decompression-budget-at-default when it sees the default — raise it (or set 0 = unlimited) server-wide, don't ignore the warning.
5. Bounded runs convert anyway. migrate up --phase pre, --target, and --max-steps all run the full hypertable work and re-register the policies when the flag is on: a run intended to apply one unrelated migration does the conversion too. Don't aim a bounded run at a fleet's schema casually.

What to watch after enabling

• The policy background jobs. timescaledb_information.jobs carries one row per registered policy (proc_name = 'policy_retention', plus the archive tables' compression-policy jobs); each row's config holds the drop_after / compress_after interval the last deploy registered. A policy that stops firing shows up as chunk counts growing without bound on the aged end — alert on chunk-count growth per hypertable (timescaledb_catalog.chunk joined on hypertable_id, or the friendlier timescaledb_information.hypertables view). Note the two failure shapes differ: a policy that is paused (scheduled = false) owns nothing — the row-level sweeps resume the aged end (row-exact, bounded batch deletes) instead of deferring to it — while a policy that is registered, scheduled, and failing on every run (the license-downgrade shape, taskq doctor's storage-mode drift arm names it) still holds the floor and strands the aged end.
• The sweeps' baseline changes shape after the policy-floor fix — read it as health, not breakage. Above the floor the aged end belongs to the policy (silent, chunk-granular), so the event-TTL and archive-expiry sweeps' aged-end deleted counts fall toward zero and their durations collapse to the floor probe plus an empty-window index scan. "The expiry sweep deletes nothing" is the floor working on this mode — but it is not the ONLY zero, and the two must be told apart. On a compressed chunk, an expiry DELETE's first execution can silently skip the qualifying rows that chunk holds (measured on 2.30.1, deterministic, independent of max_tuples_decompressed_per_dml_transaction and of every enable_dml_decompression-family GUC: a probe with 40 qualifying rows — 12 in young chunks, 28 in compressed ones — deleted 12 on the first execution, then 28 on the second, then 0; with the qualifying set entirely inside freshly compressed chunks the first tick deletes 0 outright; the probe is a recorded measurement in benchmarks/results/timescale-compression.json's first_tick_probe key, from benchmarks/timescale_compression.py, which also records the bare DELETE WHERE expire_at < now contrast — that shape deletes all 40 on the first execution). The mechanism is TimescaleDB's DML-decompression path trailing the chunk's compression state by one execution — the next sweep tick (the same statement, executed again) catches up, and there is no GUC workaround: enable_dml_decompression = off does not sidestep the lag, it hard-errors the delete. So a first-tick 0 on a compressed chunk can be this one-tick catch-up, while the floor's zero is the ZERO-DELETES-BECAUSE-NOTHING-QUALIFIED case. Distinguish them by counting what still qualifies: rows below the floor (the policy's) never appear in this count, so what it returns is the sweep's own work that did not happen yet (0 = the floor working; > 0 on a compressed chunk after a first 0-tick = the catch-up; > 0 persisting across ticks = a stuck sweep, page):

-- read drop_after from timescaledb_information.jobs.config (the policy's own horizon)
SELECT count(*) FROM "{schema}".jobs_archive
WHERE expire_at < statement_timestamp()
  AND finished_at >= statement_timestamp() - '{drop_after}'::interval;

The signals that still mean what they meant are the stall families: taskq_maintenance_leader_sweep_last_success_seconds (a sweep that stops completing) and taskq_maintenance_leader_sweep_timeouts_total (batches aborted by deadlines). Watch those; do not alert on zero aged-end deletes. - The deploy-time decompression warning. hypertable-decompression-budget-at-default in the deploy step's logs is the one configuration trap that fires later, at the first expiry batch on a compressed chunk — fix the GUC when you see it, not after the sweep pages you.

Turning it back off

The symmetric disable path is wired into the CLI: taskq migrate disable-hypertables (the CLI form of disable_hypertables in src/taskq/timescale.py, the mirror of the deploy step's enable side). The procedure and its semantics:

1. Flip TASKQ_TIMESCALEDB_HYPERTABLES=false first — the mirror of enable's gate. With the flag still true the command exits 1 with the remedy (the library-level gate is a zero-statement no-op; the CLI refuses to let a mistyped invocation look like a completed disable).
2. Run the disable under the same migration advisory lock context the enabling deploy used. It removes every registered retention and compression policy (refusing to swap tables under a live policy), then per hypertable, RENAME-FIRST: copies the rows into a restore heap and count-verifies the copy (this exercises the full read path — compressed chunks decompress — before any name moves), renames the hypertable to {table}__hypertable_trash (metadata-only: from that moment the rows exist in two places and the vanilla name is free), moves the migration-built vanilla table into place from the {schema}__vanilla scratch schema the bundled migrations themselves build — the restored pkeys, indexes, foreign keys, and defaults are the migrations' own output, never re-typed by hand — returns the rows FROM THE TRASH twin-verified against both copies, and only then drops the trash and the heap. The vanilla behaviors are restored idempotently: the bare primary keys reject duplicates again, the job_attempts_archive → jobs_archive cascade works again, and the event id sequence re-anchors past the restored maximum (forward-only, never backward; the re-anchor's one honest gap — an EMPTY restored table can re-issue an event id the prune watermark already claims — is stated in timescaledb.md).
3. A crashed run converges on re-run, structurally. A crash at ANY swap stage leaves every row in at least two places, and the next disable's first act — before any DROP SCHEMA CASCADE — finishes every crashed table's move under the same count/twin verification: a stranded trash renames back, stranded copies are absorbed into the live table, and nothing holding a row whose live twin is missing is ever dropped. There is no hand-finishing step.
4. Run it in a maintenance window when no worker archives anyway — the same caveat the enabling deploy carries. Workers keep running through the whole disable (the advisory lock serializes migrators only); the rename-first swap returns the rows from the trash, so a row committed mid-swap lands in the trash and comes back — the copy window loses nothing — but the swap's locks and the widened-uniqueness edge (below) are sized for a quiet archive all the same. One loud refusal: rows the hypertable's widened uniqueness admitted (same id, different partition-column value) cannot come back into the restored table's bare primary key; the swap names the conflicts and stops rather than choosing which row survives.

Roll-forward semantics hold on both paths: the flag-off deploy issues zero statements and a previously converted schema keeps its chunk retention at the last-registered intervals until a disable run actually executes — nothing between the flip and the disable degrades a live fleet.

See timescaledb.md for the conversion DDL, the retention-interplay rules, and the test coverage pinning all of it; configuration.md for the retention settings the chunk intervals and policies derive from.

---

Related documentation

• retries.md: retry policies, start_to_close vs schedule_to_close, classifier hooks
• workers.md: worker internals, concurrency model, WorkerSettings reference
• jobs-clients.md: enqueue/batch APIs, idempotency, dedup mechanics
• actors.md: @actor options, unique_for, sync actors, sub-job enqueuing
• rate-limiting.md: primitives, backends, reservations, queue-level strict caps
• cron.md: schedule registration, DST strategies, per-property dedup
• managed-identities.md: credential providers, BYO connections, token rotation
• deployment.md: production checklist, Kubernetes/systemd/Compose, scaling
• observability.md: OTel setup, metrics reference, error reporting
• timescaledb.md: the hypertable opt-in, conversion mechanics, measured trade-offs
• troubleshooting.md: symptom-indexed diagnosis for everything above going wrong