Troubleshooting¶

A problem-solution reference for common TaskQ operational issues. Each entry covers symptom, cause, diagnosis, and fix. Replace {schema} in SQL queries with your TASKQ_SCHEMA_NAME (default taskq).

1. Jobs stuck in `pending`¶

Symptom¶

Jobs show status = 'pending' but no worker picks them up. The pending count grows without bound.

Cause¶

Cause	Detail
No worker running	No `taskq worker` process is consuming the queue.
Wrong queue name	The actor declares `@actor(queue="email")` but the worker's `TASKQ_QUEUES` does not include `email`.
Actor not in registry	The job's `actor` field matches no `ActorRef.name` in the registry. The consumer logs `dispatch-actor-not-found` and leaves the row in `running` until the lock expires.
Stranded jobs	The actor was removed from the registry but jobs still reference it: no `actor_config` row exists.
`max_concurrent` saturated	All dispatch slots for the actor are occupied by in-flight jobs.

Diagnosis¶

SELECT queue, actor, count(*) AS cnt
FROM {schema}.jobs WHERE status = 'pending'
GROUP BY queue, actor ORDER BY cnt DESC;

Check actor config, stranded jobs, and whether any worker is consuming:

SELECT ac.actor, ac.queue, ac.max_concurrent, ac.max_pending
FROM {schema}.actor_config ac ORDER BY ac.actor;

SELECT j.actor, count(*) AS stranded FROM {schema}.jobs j
WHERE j.status IN ('pending','scheduled')
  AND NOT EXISTS (SELECT 1 FROM {schema}.actor_config ac WHERE ac.actor = j.actor)
GROUP BY j.actor;

SELECT id, hostname, pid, last_seen_at FROM {schema}.workers ORDER BY last_seen_at DESC;

Check worker logs for dispatch-actor-not-found or stranded-jobs-no-actor-config. For the "wrong queue name" cause, the worker also logs actors-on-unconsumed-queues at bootstrap when a registered actor targets a queue that worker does not consume; see workers.md for the warning's semantics (it fires even in legitimate split-queue topologies). A worker logging worker-consumes-no-queues at bootstrap has an empty TASKQ_QUEUES and will never dispatch anything; see workers.md.

Fix¶

No worker: start one: taskq worker --actors myapp.actors:registry.
Wrong queue: add the actor's queue: TASKQ_QUEUES=default,email taskq worker --actors myapp.actors:registry.
Actor not in registry: ensure the actor is decorated with @actor and exported from the registry module. Verify the module:attr string resolves to a Mapping[str, ActorRef] or Iterable[ActorRef] at import time.
Stranded jobs: re-add the actor to the registry and restart, or cancel the orphaned jobs via JobsClient.cancel(). The detector only warns; it does not delete or reassign.
max_concurrent saturated: run taskq actor-config set <actor> --max-concurrent N; takes effect on the next dispatch cycle, no restart. See workers.md.
Generator registry: if the registry attribute is a generator, the CLI iterates it twice and silently builds an empty registry. Use a list, tuple, or dict instead.

2. Jobs stuck in `scheduled`¶

Symptom¶

Jobs remain scheduled even though their scheduled_at has passed.

Cause¶

The scheduled_to_pending sweep (Sweep 3) runs every 1 second on the leader only, promoting due scheduled jobs to pending (scheduled_at <= statement_timestamp()). One tick promotes at most one bounded batch (TASKQ_EVENT_WRITER_BATCH_SIZE rows, default 100); a larger due backlog drains across the one-second ticks. If no leader is elected, jobs are never promoted. (scheduled_at still in the future is expected, not a bug.)

Cause	Detail
No leader elected	No worker holds the `taskq:maintenance_leader:<schema>` advisory lock.
Leader process died	Watchdog released the lock but no other worker has won election.
PgBouncer in transaction mode	`leader_conn` drops the session-scoped advisory lock between transactions.
Sweep batches keep timing out	A leader is healthy, but the database cannot finish bounded batches inside the batch `statement_timeout`; promotion still progresses batch by batch, but slower than jobs arrive. See the warning below and TaskQSweepTimeouts.

A growing scheduled backlog with a healthy leader means promotion is not keeping up

Each one-second tick promotes at most one bounded batch of due jobs (default 100 rows) in its own committed transaction, with a server-side statement_timeout (default 1750 ms) as the enforcement: a batch that cannot finish is aborted by the server and retried on the next tick, and a larger backlog drains across ticks. The unbounded-livelock failure this section used to document: one transaction doing one job_events round trip per promoted row, rolling back entirely past dispatcher_command_timeout and retrying forever while the backlog grew, is gone; see maintenance-sweeps.md for the design.

What can still stall promotion, in order of likelihood:

Sustained batch timeouts: the database cannot finish even bounded batches inside the per-batch statement_timeout (a lock pile-up, or plain slowness: I/O, bloat, plan regression). Aborted batches make no progress; if aborts outpace the one-batch-per-second drain, the overdue count grows.
The reduced tier: after repeated batch cancellations the batch-size breaker latches to quarter-size batches (25 rows at defaults). Promotion still progresses, just slower; the latch does not clear until the worker restarts.
No leader / stale leader: the cause table above.

Recognising it: this remains the dangerous part, because the fleet looks healthy from the outside. Workers heartbeat normally, dispatch runs cleanly reporting count: 0, and no job is in a failed state. The signature is all four of:

scheduled-wake-failed repeating on the leader about once a second (_scheduled_wake_loop sleeps 1.0s between ticks, so an aborted batch is retried at that cadence). The error is "TimeoutError()" (client deadline) or a QueryCanceledError (the server-side batch timeout).
taskq_maintenance_leader_sweep_timeouts_total{sweep_name="scheduled_to_pending"} rising, the metric counterpart of the log line, and the one the TaskQSweepTimeouts alert fires on.
workers heartbeating normally (last_seen_at fresh)
the scheduled overdue count flat or growing, never falling (taskq_jobs_by_status{status="scheduled"} climbing while {status="pending"} is flat, taskq_jobs_oldest_due_age_seconds climbing; the TaskQPromotionStalled and TaskQScheduledBacklogGrowing alerts)

All four shipped alerts (TaskQSweepTimeouts, TaskQSweepDegraded, TaskQPromotionStalled, TaskQScheduledBacklogGrowing) are documented with confirm/remediate steps in runbooks.md.

Do not look for a leader-retry line alongside it: that event belongs to the election loop and is logged by a worker that did not win the lock, with next_retry_secs = heartbeat_interval. It is unrelated to sweep failure, and its presence in a fleet with a healthy leader is normal.

Sweep 2 (sweep_deadline_exceeded) fails with the same shape and has its own signal: it runs in the 30-second sweep loop (no asyncio.timeout wrapper: each batch is bounded by the server-side statement_timeout and the pool's command_timeout instead) and logs sweep-deadline-exceeded-failed with the sweep_name="deadline_exceeded" timeout metric. A mass-expiry cohort (many jobs sharing a schedule_to_close) hits it the same way a promotion backlog hits sweep 3.

Any workload that accumulates a five-figure scheduled cohort can reach this: deferred jobs, retry-heavy actors, a paused-then-resumed fleet, or a concurrency increase that reschedules a large backlog at once.

Mitigation: fix the database, not the knobs' direction: identify the wait from the SQL in TaskQSweepTimeouts (lock pile-up: clear the blocker; slowness: bloat/indexes). If the database is genuinely slower than the batch budget, lower TASKQ_EVENT_WRITER_BATCH_SIZE (each batch gets smaller; the drain takes more batches) rather than raising TASKQ_EVENT_WRITER_STATEMENT_TIMEOUT_MS past the reclaim_event_visibility_delay margin. Do not raise dispatcher_command_timeout; cannot be raised far and will fail settings validation and crash-loop the worker. Do not scale out: promotion is a leader-side sweep, and extra workers add database load without promoting anything.

-- Emergency lever only: promote a bounded slice manually while the
-- underlying database problem is being fixed. Mirrors Sweep 3's own
-- transition. Repeat until the overdue count reaches zero.
WITH snap AS (
    SELECT id FROM {schema}.jobs
    WHERE status = 'scheduled' AND scheduled_at <= clock_timestamp()
    ORDER BY scheduled_at
    LIMIT 1000
    FOR UPDATE SKIP LOCKED
)
UPDATE {schema}.jobs j SET status = 'pending'::{schema}.job_status
FROM snap WHERE j.id = snap.id;

-- Then wake the producers. The channel embeds a hash of the schema
-- name (default schema shown; substitute yours in the literal, or use
-- taskq.constants.wake_channel(schema) from Python).
SELECT pg_notify('taskq_wake_' || left(encode(sha224('taskq'::bytea), 'hex'), 10), '');

FOR UPDATE SKIP LOCKED keeps this from contending with the leader's own retrying sweep. Verify the overdue count falls between runs. This skips the per-row job_events audit rows the sweep would have written; transition itself is complete and correct, but those promotions will not appear in job event history.

Diagnosis¶

SELECT ml.worker_id, w.hostname, w.pid, ml.last_seen_at, ml.expires_at,
       ml.expires_at >= clock_timestamp() AS lease_live
FROM {schema}.maintenance_leader ml
JOIN {schema}.workers w ON ml.worker_id = w.id;

SELECT actor, count(*) AS overdue FROM {schema}.jobs
WHERE status = 'scheduled' AND scheduled_at <= clock_timestamp()
GROUP BY actor;

Leadership is a lease on that row: the holder writes an expires_at of its own choosing and renews it every heartbeat_interval, and any pod may take the row over once that instant has passed. No rows from the first query = no leader. lease_live false = the lease has lapsed and the next election cycle on any running pod will take it. Check the admin UI at /admin/leader, which shows the same verdict.

Fix¶

No leader: ensure at least one worker is running. Failover is bounded by leader_lease + heartbeat_interval plus one round trip (50s at defaults) for a leader that went silent, and by about one round trip for one that exited cleanly (the resign is the shutdown's first act, before the drain). For one that kept winning the row but could not open its leader connections the bound is the same pair plus one failing cycle's connection attempts (each bounded by TASKQ_RELOAD_FACTORY_TIMEOUT, 30s at defaults, so roughly 80s, not 50s; the bound rests on the hand-back, not the lapse, because sub-lease re-wins keep refreshing expires_at so the row never lapses while the pod keeps winning). See leader-resigned-unassumable below.
PgBouncer: set TASKQ_PG_DSN_DIRECT to bypass PgBouncer. See PgBouncer compatibility.
Lapsed lease nobody takes: the surviving pods cannot reach or write {schema}.maintenance_leader. Check their logs for election-attempt-failed, and check that the application role still holds INSERT/UPDATE/DELETE on that table. Nothing else is needed to recover the role: no privilege over other sessions, and no manual intervention in the database.
leader-resigned-unassumable in the logs: that pod won elections but could not open the dedicated leader connections (connection-count pressure, a credential-factory outage), and the WARN means the hand-back landed: the lease row was actually deleted, so a peer takes over on its next election cycle (bounded by leader_lease + heartbeat_interval plus one failing cycle's connection attempts, each capped by TASKQ_RELOAD_FACTORY_TIMEOUT, roughly 80s at defaults, since a slow-failing factory open stretches the horizon past the flat 50s). If the WARN keeps repeating on one pod, that pod's connection budget or credential factory is the fault to fix: the fleet is leading from elsewhere. The shape to tell apart: repeated leader-dedicated-conn-failed without leader-resigned-unassumable (often with leader-resign-failed) means the hand-back could not reach the database; the row is still that pod's, it goes back on the next won cycle whose resign can land, and if the pod stops re-winning the row lapses as the backstop (sub-lease re-wins keep refreshing expires_at, so the lapse alone cannot fire while it keeps winning).
leader-advisory-lock-refused in the logs: a managed Postgres that restricts advisory-lock functions to admin/superuser roles refuses the election winner's courtesy pg_try_advisory_lock probe. The WARN names the refused function and logs once per refusal streak; leadership proceeds on the lease row alone, and election, watchdog, cron, sweep, and prune all keep running. The one exposure is a version roll alongside a release that predates the lease design and knows only the advisory lock, could lead beside the lease holder beside the lease holder. Grant EXECUTE on pg_try_advisory_lock to the application role to restore the roll protection, or schedule the cutover so old and new releases never run together.
A lease that never lapses while nothing runs: the holder is alive and renewing but its leader-gated loops are not progressing. That is a different fault; see TaskQPromotionStalled.

3. Jobs in `crashed` state¶

Symptom¶

Jobs appear with status = 'crashed' and error_class = 'WorkerCrashed' or error_class = 'HeartbeatLost'.

Cause¶

`error_class`	Mechanism	Trigger
`WorkerCrashed`	Reclaim sweep (Sweep 1)	Worker died (OOM, SIGKILL, eviction). `lock_expires_at` passed; the sweep reclaimed the job.
`HeartbeatLost`	`isolate_self`	Heartbeat failed > `max_heartbeat_failures` times. Worker self-isolated and shut down.

Both transition running → crashed only when the job is non-retryable (retry_kind = 'non_retryable' or attempt >= max_attempts). Retryable jobs are re-pended with scheduled_at = clock_timestamp() + 5s.

Diagnosis¶

SELECT id, actor, attempt, max_attempts, error_class, error_message, finished_at
FROM {schema}.jobs WHERE status = 'crashed'
ORDER BY finished_at DESC LIMIT 20;

SELECT j.locked_by_worker, w.hostname, w.pid, w.last_seen_at,
       clock_timestamp() - w.last_seen_at AS stale_for
FROM {schema}.jobs j
LEFT JOIN {schema}.workers w ON j.locked_by_worker = w.id
WHERE j.id = $1;

Check container/OS logs for OOM kills or SIGKILL on the worker host.

Fix¶

OOM kills: increase the container memory limit or reduce TASKQ_MAX_CONCURRENCY.
Retry crashed jobs: use taskq job retry, the embedded admin UI's Retry button (TASKQ_ADMIN_ACTIONS_ENABLED=true; the standalone taskq ui serve's button answers 503 — no Backend there), or backend.retry_job().
Prevent recurrence: set retry_kind="transient" with appropriate max_attempts so the sweep re-pends instead of crashing. The reclaim sweep is leader-only (it runs in the leader's sweep loop, every TASKQ_SWEEP_INTERVAL); its SQL keeps FOR UPDATE SKIP LOCKED so it stays row-safe if a sweep is ever run concurrently, e.g. two leaders of the same schema during a rolling deploy across the advisory-lock rename.

4. Jobs in `abandoned` state¶

Symptom¶

Jobs appear with status = 'abandoned'. The event log shows a state_change with cancel_phase_from=2.

Cause¶

The job was cancelled but the actor did not exit within the combined grace period. The three-phase protocol escalated:

Phase	Trigger	Worker action
`COOPERATIVE` (1)	`cancel()` writes cancel flag	Sets `ctx.cancel_event`; actor may return cooperatively
`FORCED` (2)	`cancellation_grace_period` (default 30s) elapsed	Writes `cancel_phase=2`, calls `task.cancel()`
`ABANDON_PENDING` (3)	`cleanup_grace_period` (default 10s) elapsed	Queues job for `mark_abandoned` post-transaction

Total time before abandonment: cancellation_grace_period + cleanup_grace_period (default 40s).

Diagnosis¶

SELECT id, status, cancel_phase, cancel_requested_at, finished_at
FROM {schema}.jobs WHERE id = $1;

SELECT kind, detail, created_at FROM {schema}.job_events
WHERE job_id = $1 ORDER BY created_at DESC;

Check whether the actor suppresses asyncio.CancelledError: a try/except asyncio.CancelledError: pass pattern prevents the forced-cancel path from working.

Fix¶

Always re-raise asyncio.CancelledError: never swallow it. Let it propagate so the consumer can call mark_cancelled.
Check cancellation boundaries: ensure the actor observes ctx.cancellation_requested at natural loop boundaries. For single long await calls, use ctx.cancel_event.wait().
Increase grace periods: if the actor needs more cleanup time, raise TASKQ_CANCELLATION_GRACE_PERIOD and TASKQ_CLEANUP_GRACE_PERIOD. Constraints: cancellation + cleanup < lock_lease and < termination_grace_period - 5.0.
Re-running an abandoned job: abandoned jobs: an operator cancel the actor did not honour within the cancellation and cleanup grace periods, so the worker gave up on the attempt (not a worker restart: that releases the job as pending/scheduled, see below), can be retried via backend.retry_job() or the admin UI's Retry button, the same as failed, crashed, cancelled, and succeeded jobs. Only a running job (a live attempt) or one already queued as pending/scheduled is refused.

Shutdown never lands here: read `interrupt_count` instead¶

A deploy that interrupts a running job does not produce abandoned: the job is released back to the fleet (pending, or scheduled behind the remaining termination budget when the actor never unwound) with the spent attempt standing and interrupt_count bumped. You see it on the row and the timeline, not in a terminal state:

SELECT id, status, attempt, interrupt_count FROM {schema}.jobs WHERE id = $1;
-- one state_change event per release carries detail->>'reason' = 'interrupted'

A job whose interrupt_count climbs without ever finishing is too long for your deploy cadence: it is re-run from scratch on every deploy. Bound it with schedule_to_close (the deadline fails it terminally instead of releasing it forever), or checkpoint through `ctx.progress(), released row carries the last checkpoint, and resume on re-claim.

5. NOTIFY connection failures¶

Symptom¶

Worker logs notify-conn-error and repeated notify-reconnect-attempt. Dispatch latency increases as the producer falls back to polling.

Cause¶

The NOTIFY listener holds a dedicated direct connection (notify_conn) subscribed to the schema's wake channel (wake_channel(schema)). A health-check issues SELECT 1 every notify_health_check_interval (default 5s). On failure, it reconnects with bounded exponential backoff (initial 1s, doubling, max 30s, each delay multiplied by a random factor in [0.75, 1.25] so a fleet that lost PG simultaneously does not retry in lockstep). Common triggers: pg_terminate_backend, network partition, PgBouncer in transaction mode (LISTEN is session-scoped), or Postgres restart (AdminShutdownError is treated as reconnectable).

Diagnosis¶

Verify the worker uses the direct DSN and check for active LISTEN connections:

echo $TASKQ_PG_DSN_DIRECT

SELECT pid, client_addr, state FROM pg_stat_activity
WHERE query LIKE '%LISTEN%taskq_wake%';

If TASKQ_PG_DSN_DIRECT is empty, notify_conn falls back to TASKQ_PG_DSN, which may route through PgBouncer.

Fix¶

Set TASKQ_PG_DSN_DIRECT to a direct Postgres endpoint that bypasses PgBouncer.
Wait for reconnection: the listener auto-reconnects with backoff. After reconnect, it re-registers LISTEN and fires a simulated wake notify to drain jobs that arrived while disconnected.
Reduce health-check interval: set TASKQ_NOTIFY_HEALTH_CHECK_INTERVAL=2.0 for faster detection.
Disable NOTIFY: if your environment cannot maintain a long-lived direct connection, set TASKQ_NOTIFY_ENABLED=false to use poll-only dispatch with TASKQ_POLL_INTERVAL (default 1.0s). Trades latency for resilience.

6. Migration errors¶

Symptom¶

Worker fails to start, taskq migrate up reports errors, or queries raise UndefinedTableError.

Cause¶

Error	Detail
Checksum mismatch	An applied migration file was modified after recording. Runner logs `migration-checksum-drift`.
Forward-only constraint	No `down` operation. Reverting requires a database backup restore.
Schema not migrated	`schema_migrations` table or TaskQ tables do not exist. Enqueue/get/list/cancel paths translate the raw `UndefinedTableError` into `SchemaNotMigratedError`, whose message carries the schema name and the fix; the original driver exception is chained via `__cause__`.
Concurrent migration races	Two workers starting simultaneously both attempt migrations.

Diagnosis¶

taskq migrate status

SELECT version, checksum FROM {schema}.schema_migrations ORDER BY version;
SELECT schema_name FROM information_schema.schemata WHERE schema_name = '{schema}';

Search worker logs for migration-checksum-drift.

Fix¶

Schema not migrated: taskq migrate up against the correct TASKQ_PG_DSN and TASKQ_SCHEMA_NAME. Workers never self-migrate: run the migration from a pre-deploy job or init container (the SchemaNotMigratedError message says the same). The full when/why/what-to-do row for this class is in the Exceptions API reference.
Checksum mismatch: restore the original migration file from git. Migration files are append-only; never modify an applied migration. If intentional, restore the database from backup and re-apply. Checksums are SHA-256 of the rendered SQL; a mismatch risks silent query failures at runtime.
Forward-only revert: restore from a pre-migration backup snapshot. There is no rollback.
Concurrent races: apply_pending_locked serializes appliers on a schema-qualified advisory lock, taskq:migrate:{schema}, acquired with pg_advisory_lock(hashtextextended($1, 0)) (migration_lock_name in migrate.py; the old shared fixed bigint key was deliberately removed - it made every schema in a database compete on one lock). The wait is bounded, not indefinite: DEFAULT_MIGRATION_LOCK_TIMEOUT is 120 s, and a loser raises SystemExit naming the contention instead of hanging until the container platform's startup probe kills it. A timeout message means another process held the lock - a wedged holder, or a deployment racing yours. Name the holder before touching anything:

SELECT l.pid, l.granted, a.usename, a.query, a.state
FROM pg_locks l LEFT JOIN pg_stat_activity a ON a.pid = l.pid
WHERE l.locktype = 'advisory';

Only force-release when the holder is confirmed gone (a dead session's locks are released with it, so an unclaimed lock usually means a live holder you have not found). The unlock takes the same qualified key:

SELECT pg_advisory_unlock(hashtextextended('taskq:migrate:<schema>', 0));

7. Heartbeat failures¶

Symptom¶

Worker logs heartbeat-tick-failure with increasing consecutive_failures, then isolate-self-complete and shutdown.

Cause¶

The heartbeat loop ticks every heartbeat_interval (default 10s). If a tick fails with a transient error (TimeoutError, PostgresConnectionError, QueryCanceledError, OSError) or with any other unexpected, non-transient error (a revoked UPDATE grant, a driver contract violation), heartbeat_failures increments: both arms share one threshold, one gauge, and one reset-on-success rule, so a persistent refusal isolates the worker instead of looping forever on a green /ready. When heartbeat_failures > max_heartbeat_failures (default 3), isolate_self is called: it opens a fresh direct connection, transitions running jobs (retryable → pending with 5s delay, non-retryable → crashed), writes attempts with error_class='HeartbeatLost', and exits. An early warning fires at max_heartbeat_failures // 2.

Diagnosis¶

SELECT id, hostname, pid, last_seen_at,
       clock_timestamp() - last_seen_at AS stale_for
FROM {schema}.workers ORDER BY last_seen_at DESC;

A healthy worker's stale_for should be under heartbeat_interval (default 10s). Check worker logs for the failure progression: heartbeat-tick-failure → heartbeat-failures-approaching-limit → isolate-self-complete.

Fix¶

Connection issues: verify TASKQ_PG_DSN_DIRECT resolves to a reachable Postgres. Check heartbeat_pool_size (default 4) is sufficient.
Increase tolerance: set TASKQ_MAX_HEARTBEAT_FAILURES higher (e.g. 5) to absorb transient blips. Keep lock_lease >= max(heartbeat_interval, heartbeat_command_timeout) + (max_heartbeat_failures + 1) * (heartbeat_interval + heartbeat_command_timeout), raising TASKQ_MAX_HEARTBEAT_FAILURES or TASKQ_HEARTBEAT_COMMAND_TIMEOUT raises the floor with it.
Pool exhaustion: if heartbeat_pool.acquire() times out, increase TASKQ_HEARTBEAT_POOL_SIZE.
After self-isolation: restart the worker via your process supervisor. Its running jobs were already transitioned: retryable jobs are re-pended with a 5s delay. HeartbeatLost is intentionally distinct from WorkerCrashed (Sweep 1): a heartbeat-lost worker may still be alive but partitioned.

8. Leader election issues¶

Symptom¶

No leader is elected, maintenance sweeps do not run, or /admin/leader shows no leader or a stale leader.

Cause¶

Issue	Detail
No leader elected	`maintenance_leader` table is empty; no worker has won the row's lease.
Lapsed lease nobody takes	The holder went silent and its `expires_at` has passed, but no surviving pod can reach or write the row.
PgBouncer interference	`leader_conn` routes through transaction-mode pooling, which the election, renewal, and resign statements all require a direct session for.

Diagnosis¶

SELECT ml.*, ml.expires_at >= clock_timestamp() AS lease_live
FROM {schema}.maintenance_leader ml;

Check the admin UI at /admin/workers: the is_leader column and /admin/leader's watchdog_healthy reflect the same lease verdict this query computes.

Fix¶

No leader: ensure at least one worker is running with a valid TASKQ_PG_DSN_DIRECT. Election is attempted every heartbeat_interval.
PgBouncer: set TASKQ_PG_DSN_DIRECT to bypass PgBouncer. See PgBouncer compatibility.
Lapsed lease nobody takes: the surviving pods cannot reach or write {schema}.maintenance_leader. Check their logs for election-attempt-failed, and check that the application role still holds INSERT/UPDATE/DELETE on that table. Recovery needs no privilege over the previous holder's session and no manual intervention in the database: a stuck or dead leader is displaced on the lease alone.
Multiple schemas: each schema gets its own maintenance_leader row. Verify TASKQ_SCHEMA_NAME is consistent across all workers. Failover is bounded by leader_lease + heartbeat_interval plus one round trip (50s at defaults) for a leader that went silent, and by about one round trip for one that exited cleanly (the resign is the shutdown's first act, before the drain); if slower, check that leader_conn uses a direct DSN and the watchdog health check (every 5s) is not blocked.

9. Admin UI not loading¶

Symptom¶

taskq ui serve exits with RuntimeError, or the UI loads but shows stale data or a "polling mode" badge.

Cause¶

Issue	Detail
Auth failure	`create_router()` raises `RuntimeError` if no `auth_dependency` and `TASKQ_ENVIRONMENT` is not `dev`/`development`. Fail-closed by default.
Redis not configured	`TASKQ_REDIS_URL` not set. UI falls back to polling mode: functional but less fresh.
`[fastapi]` extra missing	Admin UI requires the `fastapi` optional dependency.
Health token required	In non-dev, `taskq ui serve` fails closed if `TASKQ_HEALTH_TOKEN` is empty and `TASKQ_HEALTH_REQUIRE_TOKEN=true`.

Diagnosis¶

A RuntimeError with "admin UI requires auth_dependency" means the fail-closed check triggered. Check the mode badge in the top-right corner: real-time mode (Redis reachable), polling mode (no Redis), or polling mode (Redis unavailable).

Fix¶

Auth (dev): TASKQ_ENVIRONMENT=development taskq ui serve.
Auth (production): pass an auth_dependency to create_router(), or set TASKQ_ADMIN_UI_REQUIRE_AUTH=false and TASKQ_PROGRESS_REQUIRE_AUTH=false behind a reverse proxy that enforces auth.
Redis: TASKQ_REDIS_URL=redis://redis:6379/0 taskq ui serve.
Missing [fastapi] extra: uv add "taskq-py[fastapi]".
Health token: set TASKQ_HEALTH_TOKEN to a strong token, or TASKQ_HEALTH_REQUIRE_TOKEN=false if relying on network policy.
Polling fallback is safe: all pages remain functional (default 2.0s refresh).

10. Rate limiter not working¶

Symptom¶

Rate limits are not enforced, jobs are denied with ReservationUnavailable unexpectedly, or rate-limit state is inconsistent across workers.

Cause¶

Issue	Detail
Redis not available	Backend raises `ConnectionError`; PG fallback (if enabled) kicks in but is slower.
`[redis]` extra missing	`TokenBucket(backend="redis")` without the `redis` package raises `ImportError` at acquire time.
In-memory backend	`backend="memory"` is per-process only; state not shared across workers.
Primitives not registered	Actor references names not in the `RateLimitRegistry`. DI validation raises `MissingProvider` at startup.
Reservation slots not synced	`reservation_slots` table has the wrong row count for the configured `slots`.

Diagnosis¶

Check the admin UI at /admin/rate-limits for Postgres and Redis state. Check reservation slots and snoozed jobs:

SELECT bucket_name,
       count(*) FILTER (WHERE job_id IS NOT NULL) AS held,
       count(*) FILTER (WHERE job_id IS NULL) AS free,
       count(*) AS total
FROM {schema}.reservation_slots GROUP BY bucket_name;

SELECT actor, count(*) AS snoozed FROM {schema}.jobs
WHERE status = 'scheduled' AND snooze_count > 0
GROUP BY actor;

A denial writes no job_events and no job_attempts row, so the aggregated rate_limit_blocked_count on the job row is the per-job record of how much contention a job has absorbed. Use it to tell a job that is starving for admission from one that is merely slow:

SELECT id, actor, rate_limit_blocked_count, scheduled_at
FROM {schema}.jobs
WHERE status = 'scheduled' AND rate_limit_blocked_count > 0
ORDER BY rate_limit_blocked_count DESC
LIMIT 50;

Fix¶

Redis not available: verify TASKQ_REDIS_URL and connectivity. PG fallback (TASKQ_RATE_LIMIT_PG_FALLBACK_ENABLED=true, the default) keeps limits functional but slower.
Missing [redis] extra: uv add "taskq-py[redis]".
In-memory backend: switch to backend="redis" or backend="postgres" for multi-worker deployments. Memory is for tests only.
Primitives not registered: register all primitives on the registry singleton before the worker starts. DI validation checks each actor's rate_limits/reservations names at startup.
Reservation slots out of sync: call sync_slots() after changing slot counts. Sustained rate limiting accumulates jobs as scheduled (no retry budget consumed: a denied job is rescheduled until capacity frees or its schedule_to_close deadline expires through the ordinary deadline path; no denial terminalises a job on its own); monitor queue depth, as there is no built-in backpressure beyond max_pending.
One job is slow while the fleet looks healthy: a denial writes no job_events and no job_attempts row, so the aggregated rate_limit_blocked_count column on the job row is the per-job record of how much contention that job absorbed. Order by it to find the starving job; the fleet-wide OTel denial counters only tell you the fleet is shedding admissions.

from taskq.ratelimit import sync_slots

result = await sync_slots([my_reservation], pool=pg_pool)

11. Worker won't start¶

Symptom¶

The worker process exits immediately with a non-zero exit code and an error or traceback on stderr.

Cause¶

Failure	Detail
Migration not applied	TaskQ tables do not exist; queries raise `UndefinedTableError`.
Actor registry import error	`module:attr` does not resolve: module not found, attribute missing, or wrong type.
DI validation failure	`MissingProvider`, `ScopeViolation`, or `DependencyCycle` during `registry.validate()`.
`ActorConfigDriftList`	Registered `queue` or `metadata` differs from the stored `actor_config` row (structural drift); `--force-update-actor-config` not set. `max_concurrent` / `max_pending` / `result_ttl` never cause this: those are operator-owned and cannot drift; see ActorConfig sync.
Timing invariant violation	`lock_lease < max(heartbeat_interval, heartbeat_command_timeout) + (max_heartbeat_failures + 1) * (heartbeat_interval + heartbeat_command_timeout)`, or `cancellation + cleanup >= termination_grace - 5.0` or `>= lock_lease`.

Diagnosis¶

Read the stderr output. ActorConfigDriftList produces a clean one-line error; other failures produce a traceback. Check migration status and actor config, and compare against your @actor decorator parameters (only queue and metadata participate in drift; max_concurrent, max_pending, and result_ttl are operator-owned once a row exists):

taskq migrate status
taskq actor-config list
taskq actor-config diff --actors myapp.actors:registry

SELECT actor, max_concurrent, max_pending, queue FROM {schema}.actor_config ORDER BY actor;

Fix¶

Migration not applied: taskq migrate up, then restart.
Import error: verify the module:attr string resolves to a Mapping[str, ActorRef] or Iterable[ActorRef]:
```
python -c "from myapp.actors import registry; print(type(registry))"
```
DI validation failure: MissingProvider = missing provider. ScopeViolation = wider scope depends on narrower. DependencyCycle = provider cycle. Register the missing provider or fix the scope/cycle. See dependency-injection.md.
ActorConfigDriftList: this is always a queue or metadata mismatch. Deploy the first pod with --force-update-actor-config, then remaining pods without it. Do not leave it set permanently. If you only meant to change max_concurrent / max_pending / result_ttl, you don't need this flag at all: deploy normally, then run taskq actor-config set <actor> ....
Timing invariant violations: adjust settings so lock_lease >= max(heartbeat_interval, heartbeat_command_timeout) + (max_heartbeat_failures + 1) * (heartbeat_interval + heartbeat_command_timeout) and cancellation + cleanup < termination_grace - 5.0 and < lock_lease.

12. Performance issues¶

Symptom¶

Dispatch throughput is lower than expected, job latency is high, or the database shows high CPU/lock contention.

Cause¶

Issue	Detail
Dispatch oversampling	`TASKQ_DISPATCH_OVERSAMPLE` (default 2) gathers `residual × oversample` candidates per actor. High oversample with many actors increases query cost.
Pool too small	`dispatcher_pool_size` (default 4) or `heartbeat_pool_size` (default 4) insufficient for concurrency.
`max_concurrent` too high	Worker spawns `max_concurrency` consumers; `worker_pool_size = int(max_concurrency * 1.5)`. Too high exhausts PG connections and event-loop capacity.
`max_concurrent` too low	Actor's `max_concurrent` bottlenecks throughput even when capacity is available.
Queue depth starvation	`strict_fifo` mode lets a deep queue of one actor starve others at the same priority.

Diagnosis¶

SELECT actor, queue, status, count(*) AS cnt
FROM {schema}.jobs WHERE status IN ('pending','scheduled','running')
GROUP BY actor, queue, status ORDER BY cnt DESC;

SELECT j.actor, count(*) AS in_flight, ac.max_concurrent
FROM {schema}.jobs j JOIN {schema}.actor_config ac ON j.actor = ac.actor
WHERE j.status = 'running'
GROUP BY j.actor, ac.max_concurrent ORDER BY in_flight DESC;

Check dispatch latency via OTel or the /metrics endpoint (taskq health metrics | grep taskq_dispatch_duration).

Fix¶

Reduce oversampling: TASKQ_DISPATCH_OVERSAMPLE=1 if you do not use identity_key and run a single-producer deployment.
Tune pool sizes: increase TASKQ_DISPATCHER_POOL_SIZE and TASKQ_HEARTBEAT_POOL_SIZE if acquire() timeouts appear. Keep worker_pool_size derived.
Tune max_concurrent: run taskq actor-config set <actor> --max-concurrent N to match external resource capacity. Takes effect on the next dispatch cycle, no restart.
Switch to round_robin: for multi-tenant queues where one tenant starves others:
```
taskq queues set-mode multi round_robin
```
Takes effect within the queue-mode cache's 5 s TTL; no worker restart needed (dispatch resolves queue modes through a per-worker TTL cache; the process running set-mode invalidates its own caches immediately, and out-of-process workers pick the flip up on their next cache refill). In SQL, note the UPSERT: queues are implicit; no runtime path inserts a queues row (only the admin surface does, and set-mode above is itself an upsert), so a plain UPDATE ... WHERE name = ... matches zero rows and silently does nothing on a fresh deployment.
```
INSERT INTO {schema}.queues (name, mode) VALUES ('multi', 'round_robin')
ON CONFLICT (name) DO UPDATE SET mode = EXCLUDED.mode, updated_at = clock_timestamp();
```
This is inert on its own. Cohorts come from fairness_key, which is set at enqueue time; with no keys every job lands in one __null__ cohort and the queue behaves exactly like strict_fifo. See workers.md: Queue dispatch modes.
Scale horizontally: add worker processes. FOR UPDATE SKIP LOCKED prevents duplicate dispatch. Use unique --health-socket-path per worker on the same host.
Offload CPU-bound work: the worker is asyncio-based: CPU-bound actors block the event loop. Use run_in_executor(). Monitor taskq.dispatch.duration and messaging.process.duration via OTel: rising dispatch duration with flat process duration = DB contention; rising process duration = actor bottleneck.

13. Worker hangs or wedged shutdown¶

Symptom¶

The worker stops dispatching but the process stays alive, or it ignores SIGTERM and does not exit within termination_grace_period. /ready returns 503 with a stale_loops reason, or loop_tick_ages in the /ready body keeps growing.

Cause¶

Something has wedged inside the process: a blocked event loop (CPU-bound or synchronous work inside an actor), a deadlocked interval-driven sibling loop, a sibling that returned silently while no shutdown was in progress, or a shutdown that cannot complete because a sibling is hanging the TaskGroup exit. The in-worker watchdog (TASKQ_WATCHDOG_ENABLED=true, the default) covers all four cases. On a trip it logs worker-watchdog-trip at CRITICAL (labelled by detector), dumps every live asyncio task, and force-exits with code 2. In-flight jobs are reclaimed by the leader sweep once lock_lease expires, and the non-zero exit makes the supervisor restart the worker.

Diagnosis¶

Get the live task stacks first; they name what every task is waiting on:

# On-demand dump to the worker's log and stderr (not available on Windows):
kill -USR2 <worker-pid>

# Same payload as JSON, when the endpoint is enabled:
curl --unix-socket /tmp/taskq_health_<pid>.sock http://localhost/tasks

The socket path is the worker's bound one: an unconfigured worker binds the per-process default /tmp/taskq_health_<pid>.sock (logged at boot as health-server-started's socket_path); a worker with TASKQ_HEALTH_SOCKET_PATH set uses that value verbatim.

GET /tasks is privileged and disabled by default (TASKQ_HEALTH_TASKS_ENABLED=false): the dump reveals code structure, file paths, and task names (never locals or payload values). Enabling it also tightens the health socket to mode 0600. While disabled, the endpoint returns 404, indistinguishable from a missing route.
Read loop_tick_ages and shutdown_elapsed_seconds in the /ready body to see which loop went silent and how long shutdown has been in progress.
Correlate with the OTel metrics taskq.worker.watchdog_trips_total (by detector), taskq.worker.loop_tick_age_seconds (by loop), and taskq.worker.sibling_crashes_total (by loop).

Fix¶

Read the await-site frames in the dump to find the blocked call; move CPU-bound or blocking work off the event loop with run_in_executor().
If trips fire under legitimate load (large GC pauses, host starvation), raise TASKQ_WATCHDOG_LOOP_LAG_BUDGET, TASKQ_WATCHDOG_TICK_GRACE_FACTOR, or TASKQ_WATCHDOG_STALE_FLOOR rather than disabling the watchdog. See configuration.md. Note these interact with TASKQ_DISPATCHER_COMMAND_TIMEOUT: lowering TASKQ_WATCHDOG_STALE_FLOOR or TASKQ_WATCHDOG_TICK_GRACE_FACTOR can shrink the staleness budget below the configured timeout and make the worker fail settings validation at startup. See Dispatcher command timeout vs staleness budget.
Ensure the supervisor restarts on any non-zero exit; watchdog trips always exit with code 2.

14. Worker looks healthy in logs but is doing no work¶

Symptom¶

Logs are clean: telemetry emits, no exceptions, and a shutdown (if any) looked orderly, yet no jobs complete. Dashboards built on log volume look normal.

Cause¶

Logs are not a liveness signal. A worker can emit well-formed telemetry, log a tidy shutdown sequence, and still be dispatching nothing: the process can be alive with its consumer loops parked, isolated after heartbeat loss, holding no leader lock, subscribed to queues nothing publishes to, or capped to zero concurrency. None of those produce an error line.

The database is the source of truth. Worker liveness lives in {schema}.workers.last_seen_at, and progress lives in job state transitions. Both are observable independently of anything the worker chooses to log.

Cause	Detail
Worker not actually registered	No row in `{schema}.workers`, or `last_seen_at` is stale: the process is up but its heartbeat is not.
No state transitions	Workers fresh, but no job has changed state: dispatch is finding nothing eligible (see §1) or capacity is zero.
Capacity pinned to zero	A stored `actor_config.max_concurrent = 0` is drain mode; a queue cap of `0` is rejected, but an actor's is not.
Queue mismatch	`TASKQ_QUEUES` does not include the queue jobs are enqueued on, so this worker is healthy and irrelevant.

Diagnosis¶

Never conclude from logs. Run all three:

-- 1. Which workers does the DB believe are alive? stale_for should be
--    under heartbeat_interval (default 10s).
SELECT id, hostname, pid, queues, last_seen_at,
       clock_timestamp() - last_seen_at AS stale_for
FROM {schema}.workers ORDER BY last_seen_at DESC;

-- 2. Is anything actually progressing? Empty = no work completed,
--    regardless of what the logs say.
SELECT status, count(*) FROM {schema}.jobs
WHERE finished_at > clock_timestamp() - interval '5 minutes'
GROUP BY status;

-- 3. Is capacity pinned to zero, or unset?
SELECT actor, queue, max_concurrent, max_pending FROM {schema}.actor_config
ORDER BY actor;

Re-run query 2 a minute apart: unchanged counts mean no progress, whatever the logs show. Cross-check that workers.queues overlaps the queues jobs are enqueued on.

Fix¶

Stale or missing worker rows: the process is not heartbeating; treat it as down and restart it, then see Heartbeat failures.
Workers fresh but nothing progressing: work through Jobs stuck in pending and Jobs stuck in scheduled, including the promotion stall, whose whole signature is healthy workers plus zero promotion.
max_concurrent = 0: drain mode. taskq actor-config set <actor> --max-concurrent N to restore; effective next dispatch cycle.
max_concurrent unexpectedly NULL: the decorator literal never reached this deployment: capacity fields are seed-only. See ActorConfig sync.
Queue mismatch: align TASKQ_QUEUES with the queues actually used, and confirm via workers.queues.
Alert on the DB, not on logs: page on max(clock_timestamp() - last_seen_at) across {schema}.workers and on job-completion throughput. A log-based liveness alert cannot detect this failure mode, which is what let it run unnoticed.

15. Worker refuses wire operations after a fork¶

Symptom¶

A job body (or the process around it) forks, and in the child every TaskQ wire operation raises ForkedInheritedProcessError - a typed refusal naming the resource - while the worker's own logs carry a fork-detected-in-worker-process event.

Cause¶

TaskQ's fork guard: every wire resource TaskQ builds (asyncpg pools and connections, Redis clients, the NOTIFY listener) records the process that created it, and a forked child's first touch of one raises ForkedInheritedProcessError BEFORE a byte reaches the socket. Two processes sharing one TCP stream corrupt both protocol state machines silently; the refusal trades a loud error for that corruption. The worker's loops also consume an at-fork stamp and emit fork-detected-in-worker-process once per fork, so the log carries one honest signal that a fork happened and what it put at risk. The parent keeps running.

Fix¶

Do not use TaskQ from the forked child. Restructure so the child does no wire work (hand results back to the parent, or enqueue a follow-up job), and prefer asyncio.to_thread or a process pool seeded after TaskQ.open() over forking a live worker. What is and is not guarded — including application-built pools the guard cannot see - is in Fork safety.

16. `taskq doctor` reports a STARVED, OVERPROVISIONED, SLOW DRAIN or CRON LAG finding¶

Symptom¶

taskq doctor names a live-traffic condition on a queue or cron schedule: STARVED, STRANDED WORK, OVERPROVISIONED, SLOW DRAIN or CRON LAG. The stored-config families (a missing actor row, a stale queues row, incoherent caps) all read clean — the configuration is fine, the traffic is not.

Cause¶

These are the operational-insight families: doctor reads the live fleet through the same statements the operational-insights SQL layer documents, over the 24h window. Each finding's threshold is derived in the code (and in cli.md: The operational-insight findings); the short form:

Finding	The data	The threshold
`STARVED`	`fetch_queue_imbalance`: utilization = due depth ÷ effective capacity (`sum(max_concurrent)` × live workers)	> 2x — a full second claim wave survives after the first drains; one wave is a burst the dispatcher absorbs
`STARVED` (strand)	the same read's `oldest_due_age_s` against `fetch_wait_distribution`'s clean-segment p95	> 4x p95 with a 60s floor — past the whole observed distribution including its tail; the floor is 2x the 30s worker-liveness window
`OVERPROVISIONED`	`fetch_overprovisioning`'s verdict: live workers, zero due depth, terminalisations < workers over the window	fewer than one completion per worker across the whole window
`SLOW DRAIN`	`fetch_drain_estimates`: `eta_seconds` = due depth ÷ the window's completions/second	eta > the 24h window — the depth exceeds everything the entire window completed
`CRON LAG`	`fetch_cron_ledger`: fires vs cleared per schedule, both windows, plus the windowless `outstanding`	`runaway_trending` (fires > cleared in BOTH windows), or outstanding > the schedule's best demonstrated clearance

The confidence caveats are part of the finding, not fine print: a SLOW DRAIN eta is a throughput extrapolation resting on the window's realised traffic (has_traffic — a no-traffic window renders no eta at all, never zero), and a CRON LAG verdict should be read against the schedule's dst_strategy (an allof schedule legitimately doubles a fire in the DST-overlap hour) and its budget deferrals (a deferred fire enqueues no row, so it reads as zero fires here).

Diagnosis¶

The finding is the diagnosis — that is what doctor is for. Re-run it after acting to confirm the shape cleared; the single-window verdicts become decisive when they persist across windows.

Fix¶

STARVED / STRANDED WORK: the only two levers that add dispatch capacity — start another worker serving the queue, or raise the serving actor's max_concurrent (the finding names the actors serving it). See ops.md: When to add workers.
OVERPROVISIONED: consolidate the workers serving the queue in the workgroup config (fewer [[workers]] entries, or a smaller fleet on the queue's subscriber list). Nothing is deleted; a single-window TRUE is a hypothesis, three consecutive windows is a fleet to shrink.
SLOW DRAIN: the same capacity levers as STARVED, or slow the producer; if the window carried no traffic the eta is undefined — widen the window (up to the archive retention floor) before trusting anything else.
CRON LAG: slow the cron (widen the interval or raise its budget), or add workers for the actor (raise its max_concurrent so more fires clear per wave). Those are the only two honest levers; the backlog itself is never the thing to delete.

Troubleshooting¶

1. Jobs stuck in pending¶

Symptom¶

Cause¶

Diagnosis¶

Fix¶

2. Jobs stuck in scheduled¶

Symptom¶

Cause¶

Diagnosis¶

Fix¶

3. Jobs in crashed state¶

Symptom¶

Cause¶

Diagnosis¶

Fix¶

4. Jobs in abandoned state¶

Symptom¶

Cause¶

Diagnosis¶

Fix¶

Shutdown never lands here: read interrupt_count instead¶

5. NOTIFY connection failures¶

Symptom¶

Cause¶

Diagnosis¶

Fix¶

6. Migration errors¶

Symptom¶

Cause¶

Diagnosis¶

Fix¶

7. Heartbeat failures¶

Symptom¶

Cause¶

Diagnosis¶

Fix¶

8. Leader election issues¶

Symptom¶

Cause¶

Diagnosis¶

Fix¶

9. Admin UI not loading¶

Symptom¶

Cause¶

Diagnosis¶

Fix¶

10. Rate limiter not working¶

Symptom¶

Cause¶

Diagnosis¶

Fix¶

11. Worker won't start¶

Symptom¶

Cause¶

Diagnosis¶

Fix¶

12. Performance issues¶

Symptom¶

Cause¶

Diagnosis¶

Fix¶

13. Worker hangs or wedged shutdown¶

Symptom¶

Cause¶

Diagnosis¶

Fix¶

14. Worker looks healthy in logs but is doing no work¶

Symptom¶

Cause¶

Diagnosis¶

Fix¶

15. Worker refuses wire operations after a fork¶

Symptom¶

Cause¶

Fix¶

16. taskq doctor reports a STARVED, OVERPROVISIONED, SLOW DRAIN or CRON LAG finding¶

Symptom¶

Cause¶

Diagnosis¶

1. Jobs stuck in `pending`¶

2. Jobs stuck in `scheduled`¶

3. Jobs in `crashed` state¶

4. Jobs in `abandoned` state¶

Shutdown never lands here: read `interrupt_count` instead¶

16. `taskq doctor` reports a STARVED, OVERPROVISIONED, SLOW DRAIN or CRON LAG finding¶