CLI reference¶
The taskq CLI is the primary operational interface for managing migrations, running workers, probing health, rendering the operational insights, and serving the admin UI. All commands load settings from TASKQ_* environment variables or .env files via dotenvmodel; environment variables take precedence over .env files.
Installation¶
The taskq command is installed as part of the taskq-py package:
The console-script entry point is taskq.cli:main. See ../getting-started/quick-start.md for initial environment setup.
Global environment variables¶
These variables are read by all commands via TaskQSettings.load() or WorkerSettings.load().
| Variable | Default | Description |
|---|---|---|
TASKQ_PG_DSN |
postgresql://taskq:taskq@localhost:5432/taskq |
Postgres connection string used by migrate and health commands |
TASKQ_SCHEMA_NAME |
taskq |
Postgres schema for all TaskQ tables |
Worker-specific variables (pool sizes, heartbeat timing, cancellation grace periods, etc.) are documented in workers.md.
taskq dev¶
Starts a worker in development mode with automatic restart on file changes. Useful during local development: it saves manually killing and restarting the worker after every code edit.
Arguments:
| Argument | Description |
|---|---|
MODULE:ATTR |
module:attr reference to the actor registry, same syntax as taskq worker --actors. |
Options:
| Option | Type | Default | Description |
|---|---|---|---|
--watch |
str (repeatable) |
current working directory | Path to watch for changes. Pass the flag multiple times to watch several directories. |
--grace-period |
int |
5 |
Seconds between SIGTERM and SIGKILL when stopping the worker before a restart. 0 means send SIGTERM then kill immediately if still alive. |
How it works:
- Imports
MODULE:ATTRat startup and exits with a clear error if the module is not found or the attribute does not exist. - Spawns
taskq worker --actors MODULE:ATTRas a child subprocess. The child inherits the parent's full environment, soTASKQ_*variables set in the shell are picked up normally. - Watches the specified paths with
watchfiles.awatch()usingDefaultFilter(ignores.git,__pycache__, editor temp files, etc.) and a 400 ms debounce window. - On any file change: sends SIGTERM to the running worker, waits up to
--grace-periodseconds, then SIGKILL if still alive. Re-validates the import, then spawns a fresh worker. If the re-import fails (e.g. a syntax error) it logs a warning and waits for the next change rather than crashing. Ctrl-C(SIGINT) stops the worker cleanly and exits with code 0.
Each restart gets a fresh Python interpreter; no importlib.reload() is involved, so
module-level state (Pydantic models, actor registries, config objects) is always clean.
Requirements:
taskq dev requires the reload extra:
This installs watchfiles. Without it the command prints an install hint and exits with code 1.
Example: watch a single package directory
Example: watch multiple directories
Example: fast restart (no grace period)
Exit codes:
| Code | Condition |
|---|---|
0 |
Clean exit via Ctrl-C |
1 |
Bad MODULE:ATTR syntax, import failure, or watchfiles not installed |
Note:
taskq devis intended for local development only. Do not run it in production; usetaskq workerdirectly under a process supervisor (systemd, Docker, Kubernetes).
taskq migrate status¶
Shows which migrations have been applied and which are pending.
Connects to TASKQ_PG_DSN, queries {schema}.schema_migrations, and prints one line per discovered migration file.
Example output:
Options:
| Option | Type | Default | Description |
|---|---|---|---|
--pg-credential-provider |
str \| None |
None |
module:attr reference to a PgCredentialProvider. The connection is opened through it instead of the DSN's static password. Overrides TASKQ_PG_CREDENTIAL_PROVIDER. See Managed identities. |
A ✔ marker indicates the migration has been applied. An empty marker indicates it is pending. Migrations are discovered from the built-in migration directory in the package.
Without the flag the command uses TASKQ_PG_DSN and TASKQ_SCHEMA_NAME from the environment.
taskq migrate up¶
Applies pending migrations in order.
Options:
| Option | Type | Default | Description |
|---|---|---|---|
--phase |
pre \| post \| None |
None |
Restrict to only pre or only post phase migrations. When absent, applies both phases in order. |
--target |
str \| None |
None |
Stop after applying this migration version (inclusive). Version format matches the filename prefix, e.g. 01.00.00_01. |
--max-steps |
int \| None |
None |
Maximum number of migrations to apply in this invocation. |
--ddl-lock-timeout |
float |
30 |
Seconds a transactional migration waits for a table lock before it fails and rolls back (SET LOCAL lock_timeout; see Upgrading: the migration gave up waiting for a table lock). 0 waits indefinitely, parking every statement on the table behind the queued DDL. Values below one millisecond and non-finite values (nan, inf) are refused: a sub-millisecond bound would truncate to lock_timeout = 0 and wait indefinitely, the opposite of what was asked. |
--allow-checksum-drift |
bool |
false |
Proceed past a drifted checksum ledger: an APPLIED migration's ledger checksum differs from the bundled file, which refuses the run by default. Verify the edit is safe first; the drift is still logged as a migration-checksum-drift warning on every run and the ledger keeps the stored checksum. See Upgrading: the checksum ledger. |
--pg-credential-provider |
str \| None |
None |
module:attr reference to a PgCredentialProvider. The connection is opened through it instead of the DSN's static password. Overrides TASKQ_PG_CREDENTIAL_PROVIDER. See Managed identities. |
The command is idempotent: each migration is recorded in {schema}.schema_migrations and is skipped on subsequent runs. Running taskq migrate up with no options applies all pending migrations.
On failure (exit code 1) the command never prints a traceback. When a migration fails during apply, the command diagnoses itself: it names the failed migration, reports the state the schema was left in: a clean rollback for transactional migrations, or for -- taskq:no-transaction migrations the statements that remain applied plus any INVALID indexes found, and prints the single action to take. When the database connection itself cannot be established, there is nothing to diagnose, so it prints a short report naming the connection error and the same re-run action.
Example: apply all pending:
Example output:
applied 2 migration(s):
01.01.00_01_pre_add_reservation_slots.sql
01.01.00_02_post_add_reservation_slots.sql
When there is nothing to apply:
Example: apply only pre-phase migrations up to a target version:
taskq migrate disable-hypertables¶
Converts the TimescaleDB hypertables back to plain vanilla tables — the disable mirror of the migrate up deploy step's conversion. See TimescaleDB: turning it back off for the full mechanics and the crash-safety guarantees.
Options:
| Option | Type | Default | Description |
|---|---|---|---|
--pg-credential-provider |
str \| None |
None |
Same as migrate up's. |
The gate: flip TASKQ_TIMESCALEDB_HYPERTABLES=false first. With the flag still true the command exits 1 with the remedy (the library-level gate it wraps is a zero-statement no-op, and the CLI refuses to let a mistyped invocation look like a completed disable). On a server without the extension it is a no-op success (nothing to disable).
The command runs under the same migration advisory lock migrate up holds, re-reads the same settings cascade the workers run with, and on failure prints the same self-diagnosing report (exit 1, never a traceback). A crashed run converges on the re-run.
Example output:
disabled hypertables on 3 table(s):
restored to plain: jobs_archive
restored to plain: job_attempts_archive
restored to plain: job_events
removed retention policy: jobs_archive:365 days
removed retention policy: job_attempts_archive:365 days
removed retention policy: job_events:7 days
removed compression policy: jobs_archive
removed compression policy: job_attempts_archive
When there is nothing to disable:
taskq worker¶
Starts a TaskQ worker process. Blocks until SIGTERM or SIGINT.
Options:
| Option | Type | Default | Env var override | Description |
|---|---|---|---|---|---|
| --actors | str | required | n/a | module:attr reference to the actor registry |
| --queues | list[str] | None | TASKQ_QUEUES | Queue names to consume; repeat the flag once per queue name |
| --max-concurrency | int | None | TASKQ_MAX_CONCURRENCY | Upper bound on concurrent jobs |
| --poll-interval | float | None | TASKQ_POLL_INTERVAL | Producer loop fallback polling cadence (seconds) |
| --worker-group | str | None | TASKQ_WORKER_GROUP | Consumer group name for observability spans |
| --worker-label | str | None | TASKQ_WORKER_LABEL | Human-readable label stored in the workers table |
| --workgroup-instance | str | None | TASKQ_WORKGROUP_INSTANCE | UUIDv7 identifying the workgroup orchestrator that launched this worker |
| --health-socket-path | str | None | TASKQ_HEALTH_SOCKET_PATH | Unix socket path for the health server (use unique paths when running multiple workers) |
| --force-update-actor-config | bool | False | TASKQ_FORCE_UPDATE_ACTOR_CONFIG | Overwrite metadata-drifted actor-config rows at startup (queue and capacity are never boot-rewritten) |
| --until-idle | bool | False | n/a | Run until all subscribed queues are drained, then exit. Exit 0 if all jobs succeeded, 3 if any failed, 4 if idle-max-runtime was exceeded. Incompatible with cron-driven workloads. |
| --idle-settle-window | float | None | TASKQ_IDLE_SETTLE_WINDOW | Seconds to wait after queues appear empty before declaring drained. Default 2.0. Only used with --until-idle. |
| --idle-poll-interval | float | None | TASKQ_IDLE_POLL_INTERVAL | How often to check queue depth. Default 1.0. Only used with --until-idle. |
| --idle-max-runtime | float | None | TASKQ_IDLE_MAX_RUNTIME | Maximum wall-clock seconds before forcing exit (code 4). Only used with --until-idle. |
| --pg-credential-provider | str | None | TASKQ_PG_CREDENTIAL_PROVIDER | module:attr reference to a PgCredentialProvider (an instance, a zero-arg factory, or the provider class also work). Every Postgres pool and dedicated connection is then built through it, so SIGHUP / TASKQ_RELOAD_INTERVAL rotate real credentials. See Managed identities. |
| --redis-credential-provider | str | None | TASKQ_REDIS_CREDENTIAL_PROVIDER | module:attr reference to a RedisCredentialProvider, in the same shapes as --pg-credential-provider. Requires TASKQ_REDIS_URL. |
All other worker settings are read from environment variables. See workers.md for the full list.
--actors format¶
The value must be a module:attr string with exactly one colon separator. Both module and attr must be non-empty.
myapp.actors:registry # module=myapp.actors, attr=registry
myapp.workers.email:handlers # module=myapp.workers.email, attr=handlers
The module is imported at startup via importlib.import_module. The attribute must resolve to one of:
Mapping[str, ActorRef]: keys are actor names, values areActorRefinstances.Iterable[ActorRef]: names are read fromActorRef.nameon each element.
Any other type (including a plain list of non-ActorRef objects) prints an error and exits with code 1.
Example: iterable form
# myapp/actors.py
from taskq.actor import actor
from pydantic import BaseModel
class EmailPayload(BaseModel):
to: str
subject: str
@actor(queue="email")
async def send_email(payload: EmailPayload) -> None: ...
registry = [send_email]
Example: mapping form
--queues flag¶
--queues is a multi-value flag. Pass it once for each queue name:
Do not pass a comma-separated string directly to --queues on the command line. Use the environment variable for comma-separated input:
On the command line the flag must be repeated once per queue.
--force-update-actor-config¶
At startup the worker compares each registered actor's metadata value against the stored row in {schema}.actor_config. If it differs and this flag is absent, the worker refuses to start and prints:
ActorConfigDriftList: ...
Re-run with --force-update-actor-config to overwrite, or set TASKQ_FORCE_UPDATE_ACTOR_CONFIG=true.
max_concurrent, max_pending, result_ttl, and queue never appear in this error: they are operator-owned once a stored row exists (the @actor(...) literal only seeds them the first time) and can never drift into a refusal. A differing queue literal logs actor-config-queue-override at WARNING and the boot adopts the stored assignment (the rolling-deploy window of taskq actor-config move-queue); a differing capacity literal logs actor-config-capacity-override at info. Use taskq actor-config set to change capacity, taskq actor-config move-queue to move an actor's queue; see ActorConfig sync.
Use this flag on the first new pod of a rolling deploy when an actor's metadata has changed. Remove it for subsequent pods; it is not safe to run permanently as it allows silent metadata drift. See workers.md for the full drift protocol.
taskq actor-config¶
Inspect and tune stored {schema}.actor_config capacity fields without a code change or a worker restart:
taskq actor-config list
taskq actor-config get checkout_charge
taskq actor-config set checkout_charge --max-concurrent 20
taskq actor-config set checkout_charge --max-pending 5000
taskq actor-config set checkout_charge --clear-max-pending
All three capacity fields are live. --max-concurrent and --result-ttl take effect immediately for every worker in the fleet: the dispatch query re-reads max_concurrent every dispatch cycle, and the terminal-write path re-reads result_ttl on every job completion. --max-pending is enforced by every enqueue-side process (your web app, your workers' sub-job enqueues) through a TTL-bounded cache of the table (default 5s staleness), so a change propagates fleet-wide within seconds, with no redeploy and no worker restart.
Clearing a field writes NULL, which means different things per field on purpose: --clear-max-concurrent makes the actor unlimited (the dispatch SQL cannot see the code literal once a row exists), while --clear-max-pending / --clear-result-ttl revert to the @actor(...) literal; clearing undoes your override and restores the code default. For result_ttl the reverted literal is applied from each job's completion timestamp by the completing worker (never re-pinned to the enqueue time), so a job that sat in the queue longer than its TTL still gets its full result lifetime.
To see why a change is (or isn't) taking effect, diff the stored rows against the code literals in your registry:
Per actor and capacity field this prints the @actor(...) literal, the stored value, and the value the engine currently enforces (effective). It also flags queue mismatches (assignment drift: boot adopts the stored queue, but the cron leader's fires follow it while producers enqueue by their own literal, so the two routing halves disagree until you run taskq actor-config move-queue or deploy the matching literal), metadata mismatches (startup-blocking), actors with no stored row yet, and leftover rows whose actor is no longer registered. Note the no-row case is field-dependent: an actor with no stored row does not dispatch at all (max_concurrent shows effective=0; the dispatch capacity gate reads only actor_config rows until a worker startup seeds it), while max_pending / result_ttl fall back to the code literal.
taskq actor-config set requires the actor to have a stored row already (created by a worker startup that registered it). queue is moved by taskq actor-config move-queue; metadata is structural and only ever changes by redeploying a new @actor(...) registration (plus --force-update-actor-config if a stored row already exists).
taskq actor-config list¶
Lists every stored actor_config row, one line per actor:
checkout_charge: max_concurrent=20 max_pending=5000 queue=default result_ttl=None updated_at=2026-01-01 00:00:00+00:00
An empty table prints no actor_config rows. The list is the authoritative view of what can dispatch: rows exist only once a worker startup registered the actor, and the dispatch capacity gate reads only stored rows, so an actor whose row is missing does not appear here and does not dispatch at all. For stored-vs-code drift per field, use taskq actor-config diff instead.
taskq actor-config get¶
Shows the stored actor_config row for one actor:
The actor is named positionally, and unlike set, get does not need the actor to be in the loaded registry - it reads the row by name.
Exit codes:
| Code | Condition |
|---|---|
0 |
Row printed |
1 |
No stored actor_config row for that actor |
A missing row is a finding, not a cosmetic absence: the dispatch capacity gate reads only stored rows, so jobs for that actor accumulate pending with no error anywhere until a worker startup seeds the row.
taskq actor-config deregister¶
Deregister an actor: delete its actor_config row with safety checks.
<ACTOR>: actor name (positional argument)--force: cancel pending/scheduled jobs, disable enabled cron schedules, and proceed despite non-terminal jobs. Running jobs still block.--purge-queue: also delete the orphanedqueuesrow if no other actor references it.
Exit code 0 on success, 2 on refusal (with error message), 3 on unknown actor.
taskq actor-config move-queue¶
Move an actor to a different queue in one operator action, the one-step
replacement for the old four-write lockstep (code literal + stored row +
consumed-queue set + target queues row) whose fail-closed half refused
worker boot mid-move:
The actor's pending/scheduled backlog is rewritten onto the target queue as
bounded committed batches, then one final transaction locks the stored
assignment, carries the old queue's queues row (mode + max_concurrent) to
the target when the target has no row of its own, and flips it;
old-queue strays drain through the target's consumers, and a crash mid-drain
re-runs cleanly (the abort logs actor-queue-move-aborted with the count
already committed). Running jobs finish on the workers that claimed them;
any that re-pend instead (failure retry, crash reclaim, operator retry) keep
their old queue label as an audit trail but are routed at dispatch by the
actor's current assignment, so the running-job tail drains through the
target queue's consumers; other actors
on the old queue are untouched; cron fires follow the moved assignment from
the flip on.
Deploy the matching @actor(queue=...) literal before, during, or after
the move, in any order. Workers boot on either side of the window (a
stale literal logs actor-config-queue-override and adopts the stored
assignment). Ensure workers consume the new queue, and keep consuming the
old queue until every producer runs the new literal; stale producers
keep enqueueing to it.
The command reports how many pending jobs still carry the old queue label. That count is the residual stale producers keep adding to, and it is what tells you when the retired queue can stop being consumed; keep its consumers up until it reaches zero.
Exit code 0 on success, 2 on refusal (invalid queue name, the actor is already on that queue, the assignment changed concurrently, or a drain batch exceeded its statement timeout: batches committed before the abort are kept, so the move is incomplete and safe to re-run), 3 on unknown actor.
taskq queue migrate¶
The same move under the queue noun, for when you are thinking about queue
lifecycle rather than about the actor_config table the assignment is
stored in:
The target queue is named by an explicit --to rather than positionally.
Both arguments of a move are plain strings, and two bare positionals are
easy to transpose under pressure, with the consequence that the backlog
drains onto a queue that was never the target.
Behaviour, reporting and exit codes are identical to
taskq actor-config move-queue.
taskq doctor¶
A read-only health report for the misconfigurations that produce no error anywhere:
The report's first finding family is the detected storage mode —
vanilla, timescale-apache, or timescale-tsl, detected from the
connected server (the extension catalog, then the extension's
timescaledb.license setting), never read from settings — with that
mode's capability consequences in one glance (which retention mechanisms
exist, what the sweeps own). The family's red arms: the flag on in an
environment whose server detects vanilla gets named as drift, because
the next taskq migrate up refuses; and a license downgrade — an
apache-licensed server still carrying conversion-era policy jobs from an
earlier timescale-license deployment (they fail on every run under the
downgraded license, and the sweeps defer the aged end to them, so aged
rows strand) gets named as drift with the license-restore remedy. The
mode × capability
matrix is timescaledb.md's support matrix.
TaskQ refuses boot only on structural stored-config drift, so a whole
family of capacity and configuration problems fails silently, so their only
symptom is work that quietly does not happen. doctor names them together:
- a registered actor with no stored row, which never dispatches (the
dispatch capacity gate reads only
actor_configrows); - pending/scheduled jobs whose actor has no stored
actor_configrow (a renamed or removed actor that old producers or old rows still reference): they wait forever with no error anywhere, and no registry walk can name them, because the registry no longer knows the name. Re-register the actor and seed its row (a worker startup does this), or purge the jobs if the actor was retired; - pending/scheduled jobs routed to a queue no live worker serves: the routing queue is the job's own queue, or the actor's stored assignment once the row is re-pended. Start a worker subscribed to the queue or move the actor onto a served one (see runbooks.md: TaskQStrandedJobs);
- a stale
queuesrow whose queue no actor is assigned to; it is inert now, but silently applied to the next actor moved onto that name; - incoherent capacity combinations: a
max_pendingbelowmax_concurrent(the actor may queue fewer jobs than it may run at once, so its cap is unreachable), and an actor cap above its queue's cap (the queue binds first, so raising the actor cap changes nothing).
The report's environment family sits upstream of all of these: every
TASKQ_-prefixed environment variable that matches no settings field is a
typo applying its intended setting's default silently (the settings
loader reads only the names it defines — see
configuration.md for the full variable tables). Each
finding names the offender, states the consequence, and offers the closest
real setting name as the remedy hint (a typo is nearly always one character
of drift). The wrong value was in force before any worker registered, so
this family is checked before every stored-row condition in the report.
The two stranded-jobs families are the same computation the leader's
stranded-jobs sweep runs every minute, issued here on demand: doctor is
the surface an operator reaches for mid-incident, and it cannot wait on a
leader tick. The result is per actor, bounded by the distinct-actor count,
never by backlog depth.
When a live worker's workers row metadata carries a non-empty event-loop
stall tally, doctor reports one finding per attributed actor: which
worker recorded it, the actor, the kind counts (blocking_call: a sync
call that released the GIL; gil_held: sync work that held it), and the
remedy. The tally is the rolling top-20 the worker's lag watchdog
attributed (see runbooks.md: Event-loop stall attribution);
the worker's own event-loop-stall-attributed warnings name the exact
file:line.
A stored max_concurrent=0 is labelled drain mode and a stored NULL
is labelled uncapped, so a deliberate drain is distinguishable from an
accidental zero and a real "no actor-level cap" from missing data.
Pass --platform-grace-seconds (Kubernetes terminationGracePeriodSeconds,
an ACA/ECS stop timeout, compose stop_grace_period, systemd
TimeoutStopSec) and doctor compares it against the worker's modelled
worst-case shutdown: a platform grace below the worst case gets SIGKILLed
mid-teardown, degrading every shutdown to crash reclaim (leases expire,
in-flight work re-runs); the report names the shortfall and the fix. The
worker itself cannot see the platform's number, so this is the one check
that needs the operator to supply it.
The operational-insight findings¶
Beyond stored config, doctor reads the live fleet through the operational-insights SQL layer — the same statements the insights guide documents, run over the 24h window (a member of that layer's closed window set: 24h spans one full diurnal traffic cycle, the shortest window that cannot mistake a nightly lull for a fleet to shrink, while staying far inside the default 30-day prune retention). Four families, each naming its data and its remedy:
- STARVED (queue): the queue's utilization — due depth ÷ effective
capacity, where effective capacity is
sum(max_concurrent)over the actors routed to the queue × live workers — exceeds 2x, or the oldest due job's age exceeds the queue's own p95 wait by 4x (with a 60s floor). Threshold derivations: 2x means a full second claim wave of due work survives after the first drains entirely — one wave is a burst the dispatcher absorbs by design, two sustained waves is a fleet too small for its arrival rate, and the remedy's own granularity is one wave (one worker, or onemax_concurrentstep). "Sustained" is load-bearing: the utilization arm only fires when the oldest due job has outlived the 60s floor, because a depth snapshot younger than that is a burst in the act of being absorbed — an arrival-rate claim needs a depth that persisted past the claim-in-flight window. The 4x p95 factor survives the burst confound because the p95 is computed over the same window the burst inflates; the 60s floor is 2x the 30s worker-liveness window, so a claim already in flight is never reported. The remedy names the only two levers that add dispatch capacity: start another worker serving the queue, or raise the serving actor'smax_concurrent.utilization IS NULL(due work nothing can serve) has two shapes, reported by different lines: no live worker on the queue is the stranded-jobs family's shape; a live worker behind a storedmax_concurrent = 0is the deliberately-stopped drain mode, named by the stored-capacity line itself — neither is reported twice. - OVERPROVISIONED (queue): live workers on a queue with zero due
depth and fewer terminalisations across the whole window than workers —
fewer than one completion per worker (the verdict
fetch_overprovisioningcomputes). The remedy is to consolidate the workers serving the queue in the workgroup config — never anything destructive; the finding says explicitly that nothing is deleted. - SLOW DRAIN (queue): the drain estimate (
fetch_drain_estimates'seta_seconds— due depth ÷ the window's completions per second) exceeds the 24h observation window itself, with the due depth having persisted past the 60s persistence floor (the imbalance read's oldest-due age). Derivation: the eta is a throughput extrapolation whose only honest input is the traffic the window actually carried, so the window is the longest horizon the rate has evidence for — and it is the operator's own "will this be done by tomorrow?" period; an eta beyond it means the depth exceeds everything the entire window completed. The persistence floor is what keeps the claim honest for an idle-capacity fleet, whose observed rate is demand-limited, not capacity-limited: a depth younger than the claim-in-flight window is a burst the idle workers are absorbing, and its "eta" is fiction (a 51-job burst against an idle 32-slot worker read as ~1.0 days without it). The finding states the eta and the confidence caveat: the estimate rests on the window's realised traffic (has_traffic), assumes the next window looks like the last one, and does not include the armed wave. A window with no traffic renders no estimate at all (eta_secondsis NULL — never zero, which would read as "already drained"), so no finding fires there. - CRON LAG (schedule): a schedule whose fan-out outruns its
clearance (
fetch_cron_ledger) — eitherrunaway_trending(fires > cleared in both the current and the prior window; one window is a burst, two consecutive is the runaway shape) or an outstanding backlog above one catch-up window's slot capacity, defined as the schedule's own demonstrated clearance in its best window (max(cleared_window, cleared_prior), required positive): what the fleet actually cleared, not a theoretical ceiling — and never zero, because a schedule with no demonstrated clearance has nothing to compare against and its in-flight fires are honest work in flight, not a lag (a capacity of zero would read every first in-flight fire as an uncatchable backlog). The best-of-two guards the ledger's right-edge confound (clearance lags fires at the window's edge, so a burst of fresh fires does not read as uncatchable), and the backlog arm requires the schedule to have fired within the two-window horizon — a weekly cron with one long-running fire is work in flight, not a lag. The finding names the schedule id, actor and cron expression, sizes the backlog, and gives the two honest remedies: slow the cron (widen the interval or raise its budget), or add workers for the actor (raise itsmax_concurrentso more fires clear per wave).
These families read, they never write — the same read-only contract as
the stored-config families above. DST's allof double-fire and
budget-deferred fires are the ledger's documented confounds
(insights.md carries them per metric); read a CRON LAG
verdict against the schedule's dst_strategy before acting on it.
It issues no writing statement, so it is safe to run against production
mid-incident. It always exits 0: every condition it reports is one a worker
keeps running through, and a diagnostic that fails the shell gets wrapped
in || true and then ignored. Gate CI on drift with
taskq actor-config diff, which exits non-zero by design.
Exit codes¶
| Code | Meaning |
|---|---|
0 |
Clean shutdown (SIGTERM received and shutdown completed) |
1 |
Startup failure or runtime error |
3 |
--until-idle: some jobs failed during the drain |
4 |
--until-idle: idle-max-runtime exceeded before drain completed |
ActorConfigDriftList is caught by the CLI and produces a clean one-line error message on stderr. Other bootstrap failures (import errors, wrong attribute type, etc.) produce a Python traceback on stderr. Both exit with code 1.
taskq job show¶
Shows one job's stored row, reading {schema}.jobs first and falling back to {schema}.jobs_archive: a pruned terminal job is shown from the archive, not reported missing.
Arguments:
| Argument | Type | Description |
|---|---|---|
JOB_ID |
UUID |
The job's id. A non-UUID value is rejected as a usage error before any database access. |
Prints the operator-facing fields of the stored row: id, actor, queue, status, priority, attempt, max_attempts, retry_kind, the four timestamps, and (only when set) error_class/error_message and idempotency_key. payload, result and error_traceback are deliberately not printed by default, keeping a terminal read from dragging arbitrarily large blobs onto the wire; error_traceback and payload are available under the opt-in flags below. A row found in jobs_archive is marked archived: yes; see jobs-clients.md for the archival lifecycle.
Under retry_kind="indefinite" the stored max_attempts ceiling is inert: the retry path never consults it, so the command renders it as (indefinite), the same framing the admin UI uses; a bare number would advertise a budget the job is not enforcing. See retries.md.
Example output:
id: 018f1c7e-5a2b-7c3d-8e4f-9a0b1c2d3e4f
actor: send_email
queue: default
status: succeeded
priority: 0
attempt: 168
max_attempts: (indefinite)
retry_kind: indefinite
created_at: 2026-01-01 00:00:00+00:00
scheduled_at: 2026-01-01 00:00:00+00:00
started_at: 2026-01-01 00:00:01+00:00
finished_at: 2026-01-01 00:00:02+00:00
archived: yes
Exit codes:
| Code | Condition |
|---|---|
0 |
Row found in jobs or jobs_archive |
1 |
Invalid job id (not a UUID), or no row with that id in either table |
Uses TASKQ_PG_DSN and TASKQ_SCHEMA_NAME from the environment.
Options:
| Option | Type | Default | Description |
|---|---|---|---|
--traceback |
flag | off | Also print the stored error_traceback. The default output stays blob-free; this flag is the operator asking for the traceback by name. |
--payload |
flag | off | Also print the stored payload jsonb. |
A blob the job does not carry prints as (none), so an absent value is distinguishable from a flag that did nothing. Both blobs print in full and unbounded.
taskq job cancel¶
Cancels one job, through the same backend mechanism the admin UI's cancel route calls.
A pending or scheduled job moves straight to terminal cancelled. A running job gets a cooperative cancel request (cancel_phase=1): the worker running it stops the attempt at its next cancellation checkpoint and writes the terminal state itself, so the command prints cooperative cancel requested and the job's status stays running until that write lands. See cancellation.md.
Options:
| Option | Type | Default | Description |
|---|---|---|---|
--reason |
str |
None |
Recorded on the job's cancel_request event. |
Exit codes:
| Code | Condition |
|---|---|
0 |
Cancel applied (directly, or requested on a running job) |
1 |
Invalid job id (not a UUID), no job with that id, or the job is already in a terminal state |
taskq job retry¶
Re-pends a resting job, the same backend operation the admin UI's retry route calls.
Every terminal status is a valid source: failed, crashed, cancelled, and also succeeded (a re-run after a bad deploy - the status records what the actor returned, never that its side effects were right) and abandoned. A pending, scheduled, or running job is refused: re-pending a row while an attempt is live can execute the job twice concurrently, and a job that is already queued has nothing to put back.
The attempt counter is not reset: the re-pended job climbs to fresh attempt numbers, and max_attempts rises to at least attempt + 1 so at least one fresh execution is possible.
Exit codes:
| Code | Condition |
|---|---|
0 |
Job re-pended (prints the read-back status, pending) |
1 |
Invalid job id, no job with that id, the job is not in a terminal state, or it left a terminal state between the pre-check and the write (a concurrent claim won the race; no rows changed) |
taskq job cancel-where¶
Bulk-cancels every job matching the given filters in one set-based operation, through JobsClient.cancel_where's bounded drain.
taskq job cancel-where [--queue NAME] [--status S]... [--actor NAME] [--tag T]... [--older-than DURATION] [--reason TEXT] [--dry-run]
Pending/scheduled matches move straight to terminal cancelled; running matches get cancel_phase=1 (cooperative cancel). The write executes as bounded committed batches, so a mid-operation failure leaves partial progress rather than rolling everything back; re-running continues where it stopped (already-cancelled rows are skipped).
Guardrail: with no filter at all the command refuses to run - a filter with no predicates would cancel the entire table, and the CLI offers no allow_empty_filter bypass. Add at least one of --queue, --status, --actor, --tag, or --older-than.
Options:
| Option | Type | Default | Description |
|---|---|---|---|
--queue |
str |
None |
Only jobs on this queue. |
--status |
list[str] |
[] |
Only jobs in this status. Repeatable. Valid: pending, scheduled, running, succeeded, failed, cancelled, crashed, abandoned. |
--actor |
str |
None |
Only jobs for this actor name. |
--tag |
list[str] |
[] |
Only jobs carrying this tag. Repeatable; matches any of the given tags. |
--older-than |
duration | None |
Only jobs enqueued before now minus this duration. Formats: 45 (seconds), 30m, 2h, 7d, 2w. |
--reason |
str |
None |
Recorded on every matched job's cancel_request event. |
--dry-run |
flag | off | Print the matching count and up to 5 sample ids (newest first) and write nothing. |
Example:
# Preview, then apply: the same predicate set, so the preview answers for the write.
taskq job cancel-where --queue email --status pending --older-than 7d --dry-run
taskq job cancel-where --queue email --status pending --older-than 7d --reason "stale backlog"
Exit codes:
| Code | Condition |
|---|---|
0 |
Bulk cancel applied (or the dry run completed) |
1 |
No filter predicate given, an unknown --status value, an unparseable --older-than duration, or the backend's empty-filter guardrail refused the write |
taskq job events¶
Lists one job's job_events timeline: occurred_at, the event kind (state_change, cancel_request, heartbeat_miss, progress), and the detail jsonb rendered as one bounded line.
A long detail is truncated with the dropped-character count named. The job must exist in jobs or jobs_archive; a job with no events prints no job_events rows.
Exit codes:
| Code | Condition |
|---|---|
0 |
Timeline printed (possibly empty) |
1 |
Invalid job id, or no job with that id in either table |
taskq queues¶
Read the stored queues rows and tune the two settings they hold: the dispatch ordering mode and the fleet-wide leased-slot concurrency cap. Queues are implicit - enqueueing onto a name creates the work without creating a row - and a queue with no row runs on the defaults: strict_fifo ordering (so fairness_key has no effect) and no concurrency cap. See workers.md: Queue dispatch modes for the modes and rate-limiting.md: Queue-level concurrency cap for the cap's mechanics.
taskq queues depth¶
Per-queue depth from one grouped pass over {schema}.jobs: pending/scheduled/running/failed counts plus the age of the oldest pending job.
The oldest-pending age is computed server-side (clock_timestamp()), never by subtracting the CLI process's clock from a database timestamp. failed is included alongside the live statuses because the command runs once on demand; a - oldest-pending age means the queue has no pending rows. Queues are ordered newest activity first. Read-only, like queues list and queues get.
Exit codes:
| Code | Condition |
|---|---|
0 |
Depth table printed (or no jobs ... nothing to report on an empty table) |
taskq queues list¶
Lists every configured queues row, ordered by name:
Queues absent from this list are not missing - queues are implicit and are created by enqueueing onto them. An absent queue runs on the defaults: strict_fifo ordering (so fairness_key has no effect) and no concurrency cap. An empty table prints no configured queues (all queues run on defaults: mode=strict_fifo, max_concurrent=unlimited).
taskq queues get¶
Shows one queue's stored row:
A queue with no stored row is not an error: the command prints the same runs-on-defaults explanation an absent list row implies - including that fairness_key has NO effect on a strict_fifo queue - and exits 0.
Exit codes:
| Code | Condition |
|---|---|
0 |
Row printed, or the runs-on-defaults message for a queue with no row |
taskq queues set-mode¶
Sets a queue's dispatch ordering mode, creating the row if needed:
The mode is a positional argument, strict_fifo or round_robin. round_robin is what makes fairness_key do anything: on the default strict_fifo the key is accepted, stored, and ignored. The write is an UPSERT and takes effect on the next dispatch cycle - no worker restart.
Exit codes:
| Code | Condition |
|---|---|
0 |
Row written and printed |
1 |
Mode is not one of strict_fifo, round_robin (the accepted modes are in --help), or the queue name is not a valid queue name |
taskq queues set-max-concurrent¶
Sets or clears a queue's fleet-wide leased-slot concurrency cap:
taskq queues set-max-concurrent email --max-concurrent 8
taskq queues set-max-concurrent email --clear
Exactly one of --max-concurrent and --clear is required; passing both or neither is refused with exit code 1.
Options:
| Option | Type | Default | Description |
|---|---|---|---|
--max-concurrent |
int |
None |
New per-queue leased-slot cap. Must be >= 1 (a lower value is rejected by the option parser as a usage error). |
--clear |
flag | off | Remove the cap (unlimited): writes NULL. |
There is no 0 state: NULL (via --clear) is uncapped, and an emergency drain to 0 belongs to actor-config set --max-concurrent 0, which is per-actor. Unlike a per-actor change via actor-config set, the write is not live the moment it lands: each running worker read the cap into memory at startup, so every worker keeps enforcing the old value until it restarts, and a cap change lands fleet-wide only after the workers serving the queue have been restarted. taskq queues get shows the database value; it cannot tell you which workers have restarted onto it. See deployment.md: max_concurrent and max_pending for the live-vs-restart asymmetry and a rolling-restart recipe.
Exit codes:
| Code | Condition |
|---|---|
0 |
Row written and printed |
1 |
Both --max-concurrent and --clear given, neither given, or the queue name is not a valid queue name |
2 |
--max-concurrent below 1 (option range check, before any database access) |
taskq insights¶
Renders the taskq.insights SQL layer as operator tables: how long work waited, whether the fleet is imbalanced, how long a queue needs to drain, and whether a cron schedule is fanning out faster than it clears. The command is the terminal surface for that module — it writes no SQL of its own, every row comes from the module's statements (over BOTH retention tiers, live jobs + jobs_archive).
Read-only like taskq doctor: SELECT-only statements, exit 0 whether or not there is anything to report. Safe to run against production mid-incident.
Options: --window selects the trailing window for the windowed reads (wait, drain, cron); one of 1h, 6h, 24h, 7d (the module's INSIGHTS_WINDOWS). On hypertables a window older than the archive retention answers over the surviving chunks — the retention floor is the analytics floor. --actor is a wait-only grouping switch (see below). --queue keeps only that queue's rows on wait, balance, and drain; it is refused on cron, whose ledger is per schedule and has no queue dimension.
taskq insights wait¶
Per queue, how long terminalised attempts waited before their claim. Wait per attempt is started_at - scheduled_at (the dispatch claim latency); the rows are SEGMENTED clean vs deferred:
count— terminalised attempts in the window.p50/p95/max— the wait distribution's percentiles.segment—clean= first-delivery attempts (snooze_count = 0andrate_limit_blocked_count = 0), the subset a queue-latency SLO is written against.deferred= rows whosescheduled_ata snooze or a rate limit moved forward: reschedules are excluded from the clean percentiles, and a deferred row's wait measures only the FINAL leg (the deferred time is by construction not in the number).started_at - created_atis NOT their wait — it would fold the operator's own snooze choice into the queue's latency.
--actor ACTOR switches the statement to the module's per-(actor, queue) grouping and keeps one actor's rows — the plain per-queue read folds every actor's attempts together.
taskq insights balance¶
Per queue, is the fleet imbalanced:
queue depth live effective_capacity utilization oldest_due flag
email 122 1 4 30.50 15m2s !! over threshold
idle 1 0 0 - 10m2s !! no capacity
depth— pending rows due now (scheduled_at <= now): the work a worker could claim this instant.live— workers subscribed to the queue and heartbeating in the last 30s.effective_capacity— the routed actors' summedmax_concurrentx live workers: the queue's in-flight ceiling under the current fleet.utilization— depth ÷ effective capacity;-renders the NULL case.oldest_due— age of the oldest due job: a large depth with a tiny age is a burst, a small depth with a large age is a strand.flag— over-threshold rows are MARKED:!! over thresholdwhen utilization > 1 (more due work than one wave of capacity can absorb),!! no capacitywhen nothing can serve the queue (no live worker or no actor capacity — the starvation shape).
taskq insights drain¶
Per queue, seconds-to-drain: due depth ÷ completions per second over the window.
The eta is a THROUGHPUT extrapolation, not a promise: it assumes the next window looks like the last one, that the queue's workers stay up, and that nothing enqueues behind the current depth. The has_traffic caveat is rendered honestly: a queue whose window carried no completions has NO estimate — its eta renders no traffic in window, never 0, which would read as "already drained". Widen --window (up to the archive retention floor) before trusting anything else.
taskq insights cron¶
Per schedule, the fan-out ledger:
schedule— the schedule id, shortened to 8 characters.fires/cleared— jobs the schedule enqueued in the window (provenancemetadata cron_schedule_id) and how many of those are terminalised. Clearance lags fires at the window's right edge by construction; the ratio is a trend, not an instant.outstanding— the schedule's non-terminal backlog right now, windowless (future-armed fires included, by design).verdict—!! runaway trendingwhen fires > cleared in BOTH the current and the prior equal window (two consecutive windows is the runaway shape; one is a burst),okotherwise.
Two confounds the verdict's reader must hold: a DST allof schedule legitimately doubles a fire in the overlap hour (read the verdict against the schedule's dst_strategy), and a budget-deferred fire enqueues NO row, so a deferral reads as zero fires here — its record is the taskq.cron.budget_deferrals counter, not this ledger.
Exit codes:
| Code | Condition |
|---|---|
0 |
Tables printed (or a named nothing-to-report line per empty surface) |
1 |
Unknown surface, invalid --window, --actor outside wait, --queue on cron, or an invalid schema name |
taskq health live¶
Probes the worker's liveness endpoint.
Connects to the Unix socket at TASKQ_HEALTH_SOCKET_PATH and issues GET /live. There is no path auto-discovery: an unconfigured worker binds the per-process default /tmp/taskq_health_<pid>.sock (logged at boot as health-server-started's socket_path), so set TASKQ_HEALTH_SOCKET_PATH to probe it — the static fallback /tmp/taskq_health.sock applies only when the variable IS set, and a probe with no socket at the resolved path exits 1 with health socket unreachable (plus a hint naming the per-process default). The health server schedules a loop.call_later(0.01, ...) callback and waits up to 1.0s for it to fire, confirming the event loop is responsive.
The Unix socket is not reachable via Kubernetes httpGet probes. Use an exec probe:
Exit codes:
| Code | Condition |
|---|---|
0 |
HTTP 2xx: event loop is responsive ({"status":"ok"}) |
1 |
HTTP 5xx: event loop unresponsive or timeout exceeded |
1 |
Socket unreachable (worker not running or wrong socket path) |
Example:
taskq health ready¶
Probes the worker's readiness endpoint.
Connects to the Unix socket and issues GET /ready. The health server checks:
shutdown_phase == NONE(worker is not draining or shutting down)- PG ping succeeds within
TASKQ_HEALTH_PG_PING_TIMEOUT(default 0.2s) - when the worker runs a per-slot transaction pool, that pool is pinged too (overlapping probes share a single ping)
All conditions must pass for the response to be 200. During any shutdown phase, the response is 503 regardless of the PG ping result.
Response body (200):
{
"ready": true,
"redis_configured": false,
"active_jobs": 4,
"is_leader": false,
"shutdown_phase": null
}
Response body (503):
{
"ready": false,
"redis_configured": false,
"active_jobs": 2,
"is_leader": false,
"shutdown_phase": 1
}
shutdown_phase is null when NONE; otherwise the integer value (1=DRAINING, 2=CANCELLING, 3=FORCING, 4=RELEASING).
Exit codes:
| Code | Condition |
|---|---|
0 |
HTTP 200: worker is ready |
1 |
HTTP 503, not ready (shutting down or PG ping failed) |
1 |
Socket unreachable |
Example (Kubernetes readiness probe via exec):
readinessProbe:
exec:
command: ["taskq", "health", "ready"]
initialDelaySeconds: 5
periodSeconds: 10
taskq health metrics¶
Fetches the worker's Prometheus-format metrics.
Connects to the Unix socket and issues GET /metrics. Always returns 200. Prints the response body to stdout.
Example output:
# HELP taskq_active_jobs Currently in-flight jobs on this worker.
# TYPE taskq_active_jobs gauge
taskq_active_jobs 3
# HELP taskq_is_leader 1 if this worker holds the maintenance leader lock.
# TYPE taskq_is_leader gauge
taskq_is_leader 0
# HELP taskq_shutdown_phase Current shutdown phase enum value (0=NONE).
# TYPE taskq_shutdown_phase gauge
taskq_shutdown_phase 0
Exit codes:
| Code | Condition |
|---|---|
0 |
HTTP 200: metrics returned |
1 |
Socket unreachable or request timed out |
taskq ui serve¶
Starts the admin UI server (FastAPI + uvicorn).
Options:
| Option | Type | Env var fallback | Default | Description |
|---|---|---|---|---|
--pg-dsn |
str |
TASKQ_PG_DSN |
postgresql://taskq:taskq@localhost:5432/taskq |
Postgres DSN for the admin UI pool |
--schema |
str |
TASKQ_SCHEMA_NAME |
taskq |
Postgres schema |
--redis-url |
str |
TASKQ_REDIS_URL |
None |
Redis URL for real-time SSE progress |
--host |
str |
TASKQ_ADMIN_HOST |
0.0.0.0 |
Bind address |
--port |
int |
TASKQ_ADMIN_PORT |
8080 |
Bind port |
--migrate |
bool |
TASKQ_MIGRATE_ON_START |
false |
Apply pending migrations before starting. Aborts startup if migrations fail. |
--pg-credential-provider |
str |
None |
TASKQ_PG_CREDENTIAL_PROVIDER |
module:attr reference to a PgCredentialProvider for the admin UI's Postgres pool. See Managed identities. |
--redis-credential-provider |
str |
None |
TASKQ_REDIS_CREDENTIAL_PROVIDER |
module:attr reference to a RedisCredentialProvider. Requires TASKQ_REDIS_URL. |
Relevant environment variables:
| Variable | Default | Description |
|---|---|---|
TASKQ_ADMIN_MAX_SSE_CONNECTIONS |
50 |
Maximum concurrent SSE connections the admin UI will serve. Controls the connection-limit semaphore. |
All options fall back to the corresponding TASKQ_* environment variable when not supplied on the command line. The admin UI requires the fastapi optional dependency group:
The server creates a small asyncpg pool (min_size=1, max_size=4) and optionally a Redis client. The admin router is mounted at /admin. See admin-ui.md for the full UI reference.
Example:
The process blocks until killed. There is no graceful-shutdown option; use a process manager or container lifecycle hook.
The standalone server configures no Backend, so the backend-mediated admin mutations (job cancel, job retry, schedule run-now) answer 503 from taskq ui serve even with TASKQ_ADMIN_ACTIONS_ENABLED=true; use this CLI's taskq job cancel / taskq job retry for those, or embed the admin router with an explicit backend= (see admin-ui.md).
taskq workgroup validate¶
Validates a workgroup TOML configuration file without starting any workers.
Arguments:
| Argument | Type | Description |
|---|---|---|
CONFIG |
PATH |
Path to the workgroup TOML configuration file. |
Prints a summary of each worker's configuration. Exits 1 if the config is missing, malformed, contains invalid values (e.g. negative poll interval, misconfigured health check thresholds), or names an actors reference that cannot be resolved: an unimportable module, a module that raises on import, or a missing attribute.
Validation imports the actors module, so it must run where the application is importable: the same interpreter and PYTHONPATH the workgroup would start under. Resolving the reference here is what makes the check worth having: every child imports it the moment it is spawned, so an unresolvable reference crashes each one at import and the supervisor sees only a run of child exits, restarting them on backoff until the burst budget is spent. A CI or lint invocation that runs without the application on sys.path will report a valid config as invalid; run it from the deployment image instead.
Resolving the reference is also what lets validate warn about an actor whose queue no [[workers]] entry consumes. That is a warning, never an exit-1 condition: another workgroup or deployment may consume the queue, and no single supervisor can know the whole fleet.
Example:
taskq workgroup validate workgroup.toml
# config OK, 2 worker(s), actors='billing.actors:registry'
# api: queues=['default'] poll=0.5s concurrency=8 health=off
# batch: queues=['email', 'report'] poll=5.0s concurrency=2 health=on
taskq workgroup start¶
Starts a workgroup supervisor that manages multiple worker subprocesses from a TOML configuration file.
Arguments:
| Argument | Type | Description |
|---|---|---|
CONFIG |
PATH |
Path to the workgroup TOML configuration file. |
The supervisor spawns one taskq worker subprocess per [[workers]] entry, manages their lifecycle, and cleanly shuts them down on SIGTERM/SIGINT. See workgroups.md for the full configuration reference and operational guidance.
Example config:
actors = "myapp.actors:registry"
[[workers]]
name = "api"
queues = ["default"]
max_concurrency = 8
[[workers]]
name = "batch"
queues = ["email", "report"]
poll_interval = 5.0
max_concurrency = 2
Example:
Exit codes:
| Code | Condition |
|---|---|
0 |
Clean shutdown after SIGTERM or SIGINT |
1 |
Config file not found, invalid config, or health-check PG pool initialization failure |
Exit codes¶
| Code | Meaning |
|---|---|
0 |
Success |
1 |
Any failure: bad arguments, import errors, config drift, PG connection failures, health probe negative result |
2 |
Watchdog force-exit (wedged worker), not returned by taskq worker itself; emitted by os._exit(EXIT_WATCHDOG) |
3 |
taskq worker --until-idle: some jobs failed during the drain |
4 |
taskq worker --until-idle: idle-max-runtime exceeded before drain completed |
The taskq worker command exits with the code returned by worker_main(). On clean SIGTERM, worker_main() returns 0. With --until-idle, worker_main() returns 0 (all jobs succeeded), 3 (some jobs failed), or 4 (idle-max-runtime exceeded). A third SIGTERM calls sys.exit(1).
The taskq workgroup start command exits 0 on clean shutdown. It exits 1 if the config file is missing, invalid, or if the optional health-check PG pool fails to initialise.