Production Deployment¶

TaskQ is an async-native, Postgres-backed background job library for Python 3.12+. This guide covers running TaskQ workers, the admin UI, and supporting infrastructure in production: container orchestration, database and Redis configuration, observability, scaling, and security hardening.

Pre-1.0 stability

TaskQ is pre-1.0. Breaking changes (including schema changes) may land in minor version bumps (0.x.0), not only majors. Pin your taskq-py version and review the Changelog before every upgrade. See Upgrading for the forward-only migration policy and backup requirements.

Production Checklist¶

Worker Deployment¶

The worker is a single asyncio process running a TaskGroup of sibling coroutines: heartbeat, NOTIFY listener, maintenance leader, producer, and max_concurrency consumer loops. It blocks until SIGTERM/SIGINT and exits 0 on clean shutdown.

Container¶

The repo carries a production image definition at Dockerfile at the repository root; CI builds it on every release tag (build only, no push). It is a multi-stage build: a uv builder stage resolves the locked dependency set with uv sync --frozen --no-dev (dev group excluded, otel and redis extras installed), and a slim runtime stage ships only the resulting venv, run as a non-root taskq user. The image sets TASKQ_HEALTH_PORT=8600 and EXPOSE 8600 so orchestrators without exec probes can route HTTP probes at the TCP health listener, and its HEALTHCHECK runs taskq health ready against the in-container Unix socket, the same exec probe the Kubernetes and Compose recipes below use.

Applications that use TaskQ extend the image rather than duplicating it; PATH already points at the installed venv:

FROM taskq-worker        # this repo's Dockerfile, as built by CI
COPY myapp/ myapp/
CMD ["taskq", "worker", "--actors", "myapp.actors:registry"]

TaskQ's image surfaces are also expressible as containerspec ImageSpec chains — fluent, content-hashed specs whose hash is the build's cache key. The examples Compose stack uses this opt-in today: benchmarks/example_image_spec.py expresses examples/Dockerfile as a spec (byte-equal output, verified) and builds it under a content-hashed tag; examples/docker-compose.yml picks that tag up from TASKQ_EXAMPLE_IMAGE, so repeated compose runs skip the docker build entirely — including after a BuildKit cache prune, where the warm run drops from tens of seconds to about one. Set it before docker compose up (no --build):

export TASKQ_EXAMPLE_IMAGE="$(uv run python benchmarks/example_image_spec.py)"
docker compose -f examples/docker-compose.yml up -d

The production Dockerfile is deliberately NOT spec-expressed: containerspec cannot render its HEALTHCHECK, and its release-tag-only build cadence has no repeated-run cost for a content hash to amortize.

systemd¶

[Unit]
Description=TaskQ Worker
After=network-online.target postgresql.service

[Service]
Type=simple
User=taskq
WorkingDirectory=/opt/myapp
EnvironmentFile=/etc/taskq/worker.env
ExecStart=/opt/myapp/.venv/bin/taskq worker --actors myapp.actors:registry
Restart=on-failure
RestartSec=5
KillSignal=SIGTERM
TimeoutStopSec=120

TimeoutStopSec must exceed TASKQ_TERMINATION_GRACE_PERIOD so systemd does not SIGKILL the worker before it finishes its drain/cancel/abandon sequence. For custom bootstrap (DI providers, ErrorReporter), use worker_main programmatically; see workers.md.

Concurrency tuning¶

TASKQ_MAX_CONCURRENCY (default 8) bounds simultaneously executing jobs. The derived worker_pool_size is int(max_concurrency * 1.5).

Workload type	Recommended `max_concurrency`	Rationale
I/O-bound (HTTP, DB queries)	16-64	asyncio multiplexes I/O cheaply
Mixed I/O + CPU	8-16	Offload CPU work to `run_in_executor`
CPU-bound (image, ML)	2-4 per core	CPU work blocks the event loop

CPU-bound actors

asyncio consumers are cooperatively concurrent, not threaded. CPU-bound work blocks the event loop and starves heartbeats. Offload it with await loop.run_in_executor(None, blocking_fn, ...) or assign CPU-bound actors to a dedicated worker with low max_concurrency.

See workers.md for the full concurrency model and pool sizing, and ops.md for the worked fleet-sizing worksheet (throughput math and the Postgres connection budget).

Database Configuration¶

Direct connection requirement¶

TaskQ relies on session-scoped Postgres features (LISTEN/NOTIFY and pg_try_advisory_lock) that break under transaction-mode PgBouncer. The worker opens five connection paths:

Connection	DSN used	Why
`dispatcher_pool`	`pg_dsn_direct`	Dispatch SQL uses `FOR UPDATE SKIP LOCKED`
`heartbeat_pool`	`pg_dsn_direct`	Heartbeat extends job locks
`notify_conn`	`pg_dsn_direct`	`LISTEN` state is session-scoped
`leader_conn`	`pg_dsn_direct`	Advisory lock is session-scoped
`worker_pool`	`pg_dsn_pooled`	Short transactions; safe through PgBouncer

TASKQ_PG_DSN_DIRECT=postgresql://taskq:secret@postgres.internal:5432/taskq
TASKQ_PG_DSN_POOLED=postgresql://taskq:secret@pgbouncer.internal:6432/taskq

Without PgBouncer, set only TASKQ_PG_DSN: both split DSNs fall back to it. Never point TASKQ_PG_DSN at a transaction-mode PgBouncer. See workers.md: PgBouncer compatibility.

Enterprise DSN surfaces: multi-host, TLS, IPv6, Unix sockets¶

Every DSN form below rides the same pipeline: an env var parses into pydantic's PostgresDsn, post_load applies the direct/pooled fallback, str() serializes it, and the string goes to asyncpg's own connection resolver (the pool factories add no rewriting of their own — only the credential-injecting paths in taskq.auth touch the string, via ensure_sslmode_require / enrich_pg_dsn). Each surface's behaviour here is pinned by tests/test_enterprise_dsn_surfaces.py, and the latency numbers ops quotes ride benchmarks/latency_ladder.py.

Multi-host DSNs work end to end. The libpq multi-host form is accepted natively by asyncpg and passes through TaskQ's whole pipeline un-mangled — settings roundtrip, the pg_dsn_direct/pg_dsn_pooled fallback, the rewriters, and the pool factories:

# Failover pair; both hosts are tried in order until one accepts the session.
TASKQ_PG_DSN=postgresql://taskq:secret@pg1.internal:5432,pg2.internal:5432/taskq?target_session_attrs=read-write

target_session_attrs=read-write rides the query string untouched and asyncpg enforces it per host: a pool built from this DSN lands on a read-write session or fails to connect. The splitting of direct and pooled DSNs is per-DSN, not global — one side can name a multi-host failover pair while the other points at PgBouncer. Two caveats, both pinned as deliberate:

The pool-release-failed / connection log lines name the DSN's first host only (taskq._dsn.dsn_host is a logging helper; the query-carried Unix-socket form logs unknown). No routing decision reads it.
AWS IAM auth (taskq.aws.RdsIamProvider) parses multi-host DSNs. The token is signed for the FIRST hostspec (the one libpq tries first) and rides the DSN's shared userinfo password slot — i.e. every hostspec's — through enrich_pg_dsn, with hosts, ports and sslmode intact (pinned, including a live connect whose failure is the server's own auth error, proving the rewritten DSN reaches Postgres well-formed). A failover to a later host re-signs on the next pool rebuild (SIGHUP / TASKQ_RELOAD_INTERVAL).

TLS is opt-in per DSN, and the failure is honest. The plain DSN path adds no sslmode — asyncpg's own prefer default governs, so a DSN without one connects to a non-TLS dev container happily. sslmode=require (or stronger) in the DSN is honored by every pool TaskQ builds, because the DSN string reaches asyncpg.create_pool verbatim:

TASKQ_PG_DSN=postgresql://taskq:secret@pg1.internal:5432/taskq?sslmode=require
# Verifying modes need the CA bundle carried in the DSN (or PGSSLROOTCERT):
TASKQ_PG_DSN=postgresql://taskq:secret@pg1.internal:5432/taskq?sslmode=verify-full&sslrootcert=/etc/ssl/taskq-root.pem

What to expect when the wire disagrees with the DSN, both pinned against real servers:

sslmode=require against a server with SSL off fails loudly with ConnectionError: … rejected SSL upgrade — never a silent plaintext fallback.
sslmode=verify-full without a resolvable CA bundle fails at parameter-parse time (ClientConfigurationError naming ~/.postgresql/root.crt), before any socket is opened: asyncpg and libpq do not fall back to the system trust store, so a verifying DSN must carry sslrootcert=.
With the bundle present, verify-full connects and the session reports ssl=on in pg_stat_ssl — the one-line check to add to a deploy smoke test: select ssl from pg_stat_ssl where pid = pg_backend_pid();.

The credential-injecting paths (taskq.auth's factory builders) add sslmode=require for you when the DSN has none, and never downgrade an explicit verify-ca/verify-full; see managed-identities.md for the token-path rules.

IPv6 and Unix sockets parse with the same parity. The bracketed form (postgresql://taskq:secret@[::1]:5432/taskq) round-trips through settings and both rewriters byte-identically and connects through the pool factories. The socket form (postgresql://taskq:secret@/taskq?host=/var/run/postgresql) also round-trips; the rewriters percent-encode the host= value (host=%2Fvar%2Frun%2Fpostgresql), and asyncpg decodes query parameters, so the rewritten DSN still connects — verified against a live socket server, not assumed. Socket DSNs are the right choice for sidecar (same-pod) Postgres: no TCP attack surface, and the sslmode=disable an explicit socket deployment usually wants is honored (an explicit sslmode is never overridden).

The pooled boundary's per-statement budget, DSN side. The pooled DSN passes through settings byte-identical, and the worker pool built from it deliberately carries no client-side command_timeout — the server-side statement_timeout/lock_timeout budgets are the ruler there, so a stalled statement is bounded by what the server enforces, not by a client timer a pooler remap would orphan. When the pooler will not forward startup parameters, carry the budget in the DSN: asyncpg maps unknown query parameters to server_settings, and ?statement_timeout=2500ms lands on every session the pool hands out. Declare TASKQ_PG_IS_POOLED=true alongside — the statement-cache hygiene (0/0) must hold regardless of what the DSN carries.

Connection-ceiling honesty: the budget is pinned arithmetic, not prose. The worked profiles the sizing docs publish are computed by taskq.worker.budget.compute_connection_budget (the same function the worker's startup log reads); its outputs for the documented shapes, at the shipped pool-size defaults, one leader, no web pods:

Profile	Workers	`TASKQ_MAX_CONCURRENCY`	`total_direct`	`total_pooled`	`total_pg`	`pgbouncer_recommended`
Small	2	4	22	12	34	`False`
Medium	5	8	52	60	112	`True`
Large	10	16	102	240	342	`True`

These are the numbers to hold against the operator's real max_connections — with the application's own pools and the rolling-deploy doubling (roughly 2× steady state) included on your side; the function counts TaskQ's connections only, and the derivation lives in ops.md §4 and scaling.md.

Schema isolation and multi-tenancy¶

TASKQ_SCHEMA_NAME (default taskq) isolates all TaskQ tables into a dedicated Postgres schema. Multiple clusters can share one database:

TASKQ_SCHEMA_NAME=taskq_billing    TASKQ_PG_DSN=postgresql://app:secret@postgres:5432/appdb
TASKQ_SCHEMA_NAME=taskq_notifications TASKQ_PG_DSN=postgresql://app:secret@postgres:5432/appdb

Each schema gets its own migration set, NOTIFY channels (derived from a hash of the schema name; see architecture.md), and advisory-lock keyspace. Must match ^[A-Za-z_][A-Za-z0-9_]*$ and be at most 63 characters.

Storage engine: plain Postgres or TimescaleDB?¶

TaskQ runs on plain Postgres by default, and that is the right default for most fleets. On a server with the TimescaleDB extension available, the three retention tables (jobs_archive, job_attempts_archive, job_events) can convert to hypertables with Timescale-managed retention policies and a columnstore on the two archive tables, so aged data leaves the database as whole-chunk drops instead of long-window DELETEs, and the admin archive's newest-first reads get several times faster. Whether that trade suits your fleet is a decision with its own measured decision table and guarantee gradient — see ops.md §13: plain Postgres or hypertables?; the conversion mechanics, the DDL, and the retention-interplay rules live in timescaledb.md. This section is the deployment story: what to set, what the deploy does, what to verify afterwards, and how to go back.

The flag is TASKQ_TIMESCALEDB_HYPERTABLES (default false), and the taskq migrate up deploy step is the only thing that reads it: workers and clients never consult it, and no runtime code path branches on it. The deploy step loads the worker settings model, so the deploy environment and the workers' environment cannot disagree about the retention settings the chunk intervals and policies derive from (TASKQ_ARCHIVE_RETENTION_PERIOD, TASKQ_EVENT_RETENTION_PERIOD).

The mode is detected, not declared. Which of the three first-class storage modes a database actually presents — vanilla, timescale-apache, or timescale-tsl — is detected from the server (the extension catalog, then the extension's timescaledb.license setting), never inferred from the flag, and taskq doctor leads every report with the detected mode and its capability consequences. The matrix, cited from the detection code (taskq.timescale.detect_storage_mode / the branches enable_hypertables takes on it):

Capability	`timescale-tsl`	`timescale-apache`	`vanilla`
Hypertables	yes	yes	no — plain tables
Compression (columnstore)	yes, on the two archive tables	no — Timescale-license feature, refused under the license	no
Policy-driven chunk-drop retention	yes — registered per deploy	no — also a Timescale-license feature, refused under the license	no
What retention does instead	row-level sweeps still run inside young chunks; `expire_at` exact only there	row-level sweeps own all of retention (bounded batch deletes, `expire_at` exact everywhere)	row-level sweeps own all of retention (bounded batch deletes, `expire_at` exact everywhere)

What the deploy does with it:

Flag off (the default): zero new statements — no probe, no extension query, no DDL. Behavior is identical to plain Postgres.
Flag on: the conversion runs inside the migration advisory lock, after pending migrations apply, and refuses loudly with TimescaleDBUnavailableError when the server cannot honor the feature (no extension offered, no privilege to create it, or the extension installed but absent from shared_preload_libraries). It is idempotent converging DDL, re-run on every deploy, not a migration in the checksummed ledger — already-converted tables are skipped, changed retention settings are honored on the next deploy. The detected mode decides what "honored" means: on a timescale-tsl server the deploy registers the retention and compression policies; on a timescale-apache server (the license set to apache, whether by build or by configuration) it converts the hypertables and registers nothing — the policies and the columnstore are Timescale-license features the server refuses — and reports the skip in HypertableReport.policies_skipped while logging hypertable-policy-registration-skipped.

Mid-life license flips (both measured on 2.30.1). Upgrading apache → timescale needs nothing special: the schema converted under apache is a plain rowstore hypertable set, and the next taskq migrate up under the restored license adopts the full policy set — the schema is not stuck bare. Downgrading timescale → apache is the hazardous direction: the conversion-era policies SURVIVE the downgrade (the license cannot remove them — every policy removal API refuses under apache too) and then fail on every background run, while the row-level sweeps defer the aged end to them — so rows older than the dead policies' horizon strand, silently. The doctor names that state as storage-mode drift with the license-restore remedy, and taskq migrate disable-hypertables refuses it loudly rather than swapping tables under a live policy: restore the timescale license (ALTER SYSTEM SET timescaledb.license = 'timescale' and reload), re-run taskq migrate up to converge, then flip and disable properly.

Fresh vs mid-life. A fresh or young schema converts in seconds. On a populated schema the same deploy rewrites what is already there, and the enabling deploy window has three parts to size (the full worksheet is ops.md §13, "Planning the enabling deploy"):

The constraint surgery. The foreign-key drop, the primary-key drops, and one ADD CONSTRAINT ... UNIQUE per table each take ACCESS EXCLUSIVE and full-scan their table to validate the uniqueness asserted. On a big archive those scans are part of the deploy window, not a footnote to it.
The migrate_data copy. The conversion rewrites existing rows into chunks under the same access-exclusive lock: size the window for the constraint scans and the copy together, against the archive's current row count.
The first policy run. Every chunk older than the retention interval is drop-eligible the moment the enabling deploy finishes, so the first background policy run drops the whole aged tail in one sweep — potentially gigabytes of IO where vanilla expires the same rows gradually, row by row. Size the post-deploy window for that run too.

Run the enabling deploy in a maintenance window when no worker archives: between a table's primary-key drop and its unique add, a duplicate insert lands unguarded and wedges every later deploy until the duplicates are removed by hand. The window is two adjacent statements — tiny and bounded, but not zero. And note bounded runs convert anyway: migrate up --phase pre, --target, and --max-steps all run the full hypertable work and re-register the policies when the flag is on; do not aim a bounded run at a fleet's schema casually.

Local dev: the Compose service

The suite pins timescale/timescaledb:2.30.1-pg18 (the newest tagged release matching the suite's Postgres 18 image). Swap it in for the dev Compose file's postgres service — the image preloads the extension (shared_preload_libraries=timescaledb), and the explicit command flag keeps that requirement visible in the file:

  postgres:
    image: timescale/timescaledb:2.30.1-pg18
    container_name: taskq-postgres
    restart: unless-stopped
    environment:
      POSTGRES_USER: taskq
      POSTGRES_PASSWORD: taskq
      POSTGRES_DB: taskq
      POSTGRES_INITDB_ARGS: "--encoding=UTF-8 --locale=C"
    # The image already sets shared_preload_libraries=timescaledb; the
    # flag below makes the requirement explicit (and copy-paste-safe).
    command: ["postgres", "-c", "shared_preload_libraries=timescaledb"]
    ports:
      - "5432:5432"
    volumes:
      - postgres-data:/var/lib/postgresql
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U taskq -d taskq"]
      interval: 2s
      timeout: 3s
      retries: 20
      start_period: 5s

Then opt in and deploy: TASKQ_TIMESCALEDB_HYPERTABLES=true taskq migrate up. See timescaledb.md: Local development.

On Azure, allow-list the extension before the deploy

On Azure Database for PostgreSQL Flexible Server the extension must be allow-listed on the server before a flag-on deploy can succeed: add timescaledb to the azure.extensions server parameter, create it once with an administrative role (CREATE EXTENSION IF NOT EXISTS timescaledb;), and let the restart apply the preload. Without the allow-list there is no timescaledb row in pg_available_extensions and the deploy refuses with TimescaleDBUnavailableError, naming the setting and the parameter.

The columnstore prerequisite: raise the decompression budget

The same deploy arms the columnstore (compression) on the two archive tables, and compression puts chunks in the archive-expiry sweep's DML path. timescaledb.max_tuples_decompressed_per_dml_transaction defaults to 100000, and at that default the sweep hard-errors on compressed chunks with ConfigurationLimitExceededError (measured: one 10k-row expiry batch decompressed 356633 tuples). The deploy probes the setting and logs hypertable-decompression-budget-at-default when it sees the default. Raise the budget server-wide before relying on the columnstore:

ALTER SYSTEM SET timescaledb.max_tuples_decompressed_per_dml_transaction = '0';  -- 0 = unlimited
SELECT pg_reload_conf();

After the first flag-on deploy, verify the conversion landed (substitute your TASKQ_SCHEMA_NAME; the default schema is taskq):

-- Three hypertables: job_events, job_attempts_archive, jobs_archive
SELECT hypertable_name FROM timescaledb_information.hypertables
WHERE hypertable_schema = 'taskq' ORDER BY hypertable_name;

-- Three retention policies (proc_name = 'policy_retention'), plus the two
-- archive tables' compression policies ('policy_compression', or
-- 'policy_columnstore' on some extension versions)
SELECT hypertable_name, proc_name, config FROM timescaledb_information.jobs
WHERE hypertable_schema = 'taskq' AND proc_name IN
  ('policy_retention', 'policy_compression', 'policy_columnstore')
ORDER BY hypertable_name, proc_name;

-- At least one chunk exists once events have landed
SELECT show_chunks('"taskq".job_events');

Expect exactly three rows from the first query, three policy_retention rows plus the archives' two compression policies from the second, and at least one chunk from the third once the worker has enqueued anything. A policy that stops firing later shows up as chunk counts that grow without bound on the aged end; the day-2 watch list is ops.md §13, "What to watch after enabling".

Turning it off is a two-step, forward-only story. Flipping TASKQ_TIMESCALEDB_HYPERTABLES=false is a no-op by design: the deploy issues zero statements again, but the converted schema keeps its hypertables and policies at the last-registered intervals — nothing degrades a live fleet between the flip and the disable. The way back to plain tables is the explicit disable path, taskq migrate disable-hypertables, run after flipping the flag off (with the flag still true the command refuses loudly, and the library-level gate is a zero-statement no-op): it removes every registered policy, restores each table to a vanilla shape cloned from the bundled migrations' own output, count-verifies every row at each hand-off, and restores the traded-away behaviors (bare primary keys, the attempts cascade). It is idempotent, safe mid-life, and a crashed run converges on the re-run; see timescaledb.md: Turning it back off.

Migration strategy¶

Migrations are forward-only and idempotent. Run before workers start:

taskq migrate up

Each migration is recorded in {schema}.schema_migrations with a SHA-256 checksum. Re-running is a no-op when all are applied.

No down migrations

There is no down migration. To revert, restore the database from a backup taken before the migration was applied. Always take a backup (or confirm a PITR window) before upgrading. See Upgrading.

Some migrations are split into a pre and a post phase. The pre phase adds the new structures while keeping every structure the old release still needs; the post phase removes those old structures. A bare taskq migrate up applies both phases, right for a fresh install or a stop-and-replace redeploy, but wrong anywhere a rollout can overlap: a post phase applied while old pods still serve breaks them mid-rollout (dropping the old single-column idempotency index, for example, turns every enqueue from a pre-upgrade worker into InvalidColumnReferenceError, SQLSTATE 42P10; see the phase-obligation notes in 01.00.03_01_pre_idempotency_scope.sql). On a rolling deploy the sequence is:

Apply taskq migrate up --phase pre as the pre-deploy job or init container, safe against old and new code, before or during the rollout.
Roll the fleet; wait until every pod runs the new release.
A human confirms the rollout completed, then applies taskq migrate up --phase post once, by hand (or from a manually triggered one-off job). Never wire the post phase to an init container, a pod lifecycle hook, or any step that fires unsequenced per pod, those run mid-rollout, while old pods still need the structures post removes.

A worker started against a schema with a pending pre-phase migration refuses to boot, naming the missing migrations; a pending post-phase migration never blocks boot; that middle state is the rollout window by design.

Deployment order: (1) apply migrations as a pre-deploy job or init container, with --phase pre on anything that rolls, (2) start workers: they call sync_actor_config at startup and fail with ActorConfigDriftList if a registered actor's structural config (metadata) differs from the stored row (a differing queue literal is the rolling-deploy window of taskq actor-config move-queue: it logs actor-config-queue-override and boots). That drift check does not detect a stale schema: it compares config rows, never a schema version, but a stale schema is caught anyway: a worker whose schema has a pending pre-phase migration refuses to boot, naming the missing migrations (see above), (3) start the admin UI (optionally with TASKQ_MIGRATE_ON_START=true), (4) once the rollout is confirmed everywhere, apply --phase post as described above.

For rolling deploys where actor config changes, deploy the first pod with TASKQ_FORCE_UPDATE_ACTOR_CONFIG=true to overwrite stored config, then deploy the rest without it. See workers.md: ActorConfig sync.

Migrations under a running fleet: what rides, what must wait¶

Ops sometimes runs DDL against the tables while workers and admins still hold pools of cached prepared statements (asyncpg caches prepared statements per connection). The behaviour under that, measured against PostgreSQL 18.6 with a warm worker pool:

ADD COLUMN ... NOT NULL DEFAULT is safe under a live fleet. Cached plans over the old column set stay valid (PG stores the default as a fast default: no table rewrite, no plan invalidation), and an old-shape INSERT that omits the new column reads the default back invisibly. No error reaches the worker.
A cached plan that a DDL invalidates inside an open transaction (a result-type change: ALTER COLUMN TYPE, a DROP COLUMN under SELECT *) surfaces as InvalidCachedStatementError, SQLSTATE 0A000. TaskQ classifies the 0A000 family as transient infrastructure: the loop retries next tick and the statement re-prepares against the new schema (asyncpg has already cleared its pool-wide statement cache by the time the error propagates). A log line, a tick, not a restart.
The migration's own lock queue is the freeze risk. ALTER TABLE queues ACCESS EXCLUSIVE behind any open job transaction, and once queued every later statement on that table queues behind IT: one 30s job can freeze the whole fleet's dispatch, enqueue, and heartbeat for 30s. The runner bounds the wait (ddl_lock_timeout, default 30s) and gives up with MigrationLockTimeoutError, nothing applied; find the holder in pg_stat_activity/pg_locks, end it or let it finish, then re-run taskq migrate up. Do not raise the bound past your longest job: the bound is the fleet's worst freeze window.
What must still wait for the contract: DROP/RENAME of anything the old code still references. Old workers' SQL then fails with UndefinedColumnError (42703) on every execution, permanently - the text itself no longer matches the schema. That failure stays loud and fatal by design (42703 is deliberately not transient; classifying it as such would livelock a fleet that can never succeed again). Drop old structures only in the post phase, after the rollout completed.

Redis (Optional)¶

Redis is optional. TaskQ degrades gracefully without it:

Feature	With Redis	Without Redis
Progress fanout	Redis pub/sub → SSE push	Postgres only; admin UI polls
Admin UI mode	Real-time (SSE)	Polling (2s interval)
Rate limiters	Redis backend (shared, low-latency)	Postgres fallback (default) or in-memory (per-process)
Reservation slots	Redis state in admin UI	Postgres state only

TASKQ_REDIS_URL=redis://redis.internal:6379/0

The admin portal's Redis client requirement¶

TASKQ_REDIS_URL alone does not give the portal live progress: the admin router must receive the Redis client itself, as create_router(redis_client=...), so setup_admin_state puts it on app.state for the progress stream to subscribe with. taskq ui serve wires both from the same setting; a host application mounting the router itself must pass both halves. The two failure shapes, and the log lines that name them:

Redis configured, client not wired (the misconfiguration): the portal boots with one admin-ui-no-redis-client warning; live SSE progress is unavailable, the dashboard falls back to polling, and the job pages show the polling mode badge. Fix: pass the client to create_router.
No TaskQ Redis configured at all (legitimate polling mode): no warning. The job pages show the polling mode badge, and the first refused progress stream logs progress-stream-polling-degraded once, naming the 503 redis_not_configured answer and the polling fallback. Polling pages are first-class: an unchanged poll tick downloads nothing (conditional If-None-Match on the progress sequence, answered 304) and writes nothing to the DOM; see the polling UX contract.

When to provision Redis

Provision Redis if you use Redis-backed rate limiters across multiple workers, or if operators need real-time admin UI updates. Without Redis, rate limiting falls back to Postgres (higher latency, more DB load).

See rate-limiting.md and progress.md for details.

Admin UI in Production¶

The admin UI (taskq ui serve) is a FastAPI + Jinja2 dashboard on TASKQ_ADMIN_HOST:TASKQ_ADMIN_PORT (defaults: 0.0.0.0:8080). It fails closed by default: in non-dev environments, create_router() raises RuntimeError if no auth_dependency and TASKQ_ADMIN_UI_REQUIRE_AUTH=true (the default).

Variable	Default	Description
`TASKQ_ADMIN_UI_REQUIRE_AUTH`	`true`	Raises `RuntimeError` at startup if no `auth_dependency` in non-dev
`TASKQ_PROGRESS_REQUIRE_AUTH`	`true`	Raises `RuntimeError` at startup if the progress router has no `auth_dependency` in non-dev (the admin UI forwards its own `auth_dependency`, so this only bites standalone mounts and fully unauthenticated `taskq ui serve`)
`TASKQ_ADMIN_ACTIONS_ENABLED`	`false`	When `false`, cancel/retry/run-now return `403`
`TASKQ_ADMIN_UI_ALLOW_RATE_LIMIT_RESET`	`false`	Gates the rate-limit reset endpoint
`TASKQ_HEALTH_TOKEN`	(none)	Bearer token for the health/metrics endpoints `taskq ui serve` exposes; the worker's TCP health listener and scrape endpoint never check it
`TASKQ_HEALTH_REQUIRE_TOKEN`	`true`	Fails closed if `TASKQ_HEALTH_TOKEN` empty in non-dev

Reverse proxy authentication¶

Run taskq ui serve behind an authenticating reverse proxy and set TASKQ_ADMIN_UI_REQUIRE_AUTH=false:

server {
    listen 443 ssl;
    server_name admin.example.com;
    location /admin/ {
        auth_basic "TaskQ Admin";
        auth_basic_user_file /etc/nginx/.htpasswd;
        proxy_pass http://127.0.0.1:8080;
    }
}

Set TASKQ_ADMIN_UI_REQUIRE_AUTH=false, TASKQ_PROGRESS_REQUIRE_AUTH=false and TASKQ_HEALTH_REQUIRE_TOKEN=false to suppress the fail-closed checks when relying on the proxy for auth.

Run the admin UI as a separate process from the worker: different scaling, exposure, and resource characteristics. See admin-ui.md for create_router() embedding and SSO configuration.

Kubernetes Deployment¶

apiVersion: apps/v1
kind: Deployment
metadata:
  name: taskq-worker
spec:
  replicas: 3
  selector:
    matchLabels:
      app: taskq-worker
  template:
    metadata:
      labels:
        app: taskq-worker
    spec:
      terminationGracePeriodSeconds: 120
      initContainers:
        # `--phase pre` is essential to the invariant: the init container runs on every new
        # pod DURING the rolling update, while old pods still serve. A bare
        # `migrate up` would also apply `post`-phase migrations, which remove
        # structures the old release still needs, breaking the old pods'
        # enqueue path mid-rollout. See "Migration strategy" above.
        - name: migrate
          image: myapp:latest
          command: ["taskq", "migrate", "up", "--phase", "pre"]
          env:
            - name: TASKQ_PG_DSN
              valueFrom:
                secretKeyRef:
                  name: taskq-db
                  key: dsn
      containers:
        - name: worker
          image: myapp:latest
          command: ["taskq", "worker", "--actors", "myapp.actors:registry"]
          env:
            - name: TASKQ_PG_DSN
              valueFrom:
                secretKeyRef:
                  name: taskq-db
                  key: dsn
            - name: TASKQ_REDIS_URL
              valueFrom:
                secretKeyRef:
                  name: taskq-redis
                  key: url
            - name: TASKQ_ENVIRONMENT
              value: production
            - name: TASKQ_MAX_CONCURRENCY
              value: "16"
            - name: TASKQ_QUEUES
              value: default,priority
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: http://otel-collector:4317
            - name: OTEL_SERVICE_NAME
              value: taskq-worker
          livenessProbe:
            exec:
              command: ["taskq", "health", "live"]
            initialDelaySeconds: 5
            periodSeconds: 10
          readinessProbe:
            exec:
              command: ["taskq", "health", "ready"]
            initialDelaySeconds: 5
            periodSeconds: 10
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "2000m"
              memory: "1Gi"

Once the rollout has completed (every pod confirmed on the new release), close out the phase split with the post phase, exactly once, as a deliberate manual step (never an init container or automated post-deploy hook: those fire unsequenced, mid-rollout):

kubectl exec deploy/taskq-worker -- taskq migrate up --phase post

A pending post phase is a safe steady state meanwhile: workers boot and run normally against it, so there is no urgency that would justify automating the step away. See Migration strategy for what each phase guarantees.

See configuration.md for the full set of TASKQ_* environment variables and cross-field validation constraints.

Health probes¶

The worker serves the same two endpoints (/live and /ready) over two transports:

Transport	Enabled by	Reachable from
Unix socket at `TASKQ_HEALTH_SOCKET_PATH` (the image pins `/tmp/taskq_health.sock`; an unconfigured worker binds the per-process default `/tmp/taskq_health_<pid>.sock`)	`TASKQ_HEALTH_ENABLED` (on by default)	inside the container: `taskq health live` / `taskq health ready`, so Kubernetes `exec` probes
TCP, `TASKQ_HEALTH_HOST`:`TASKQ_HEALTH_PORT`	setting `TASKQ_HEALTH_PORT` (unset by default)	anything that can reach the container port: `httpGet` and `tcpSocket` probes

Responses are what orchestrators expect: 200 when healthy, 503 when not, 404 for an unknown path. Kubernetes and Azure Container Apps both treat 200-399 as probe success, so a 503 reliably fails a probe on either.

/live answers whether the event loop is responsive; a failing liveness probe means restart me. /ready additionally pings Postgres, checks for stale worker loops, fails during shutdown, and runs any checks you registered (see below); a failing readiness probe means stop sending me work, not restart me.

What authenticates where: the token is a taskq ui serve feature, not a worker one

TASKQ_HEALTH_TOKEN protects only the health/metrics routes that taskq ui serve exposes; the worker's listeners below never check it. What guards each transport instead:

Surface	Endpoints	Auth
Unix socket	`/live`, `/ready`, `/metrics`, opt-in `/tasks`	filesystem permissions on the socket (`0600` when `TASKQ_HEALTH_TASKS_ENABLED=true`), so same-pod only
TCP listener (`TASKQ_HEALTH_PORT`)	`/live`, `/ready`	none: keep the port pod-network-only
Scrape listener (`TASKQ_METRICS_PORT`)	`/metrics`	none: keep it pod-network, security-group-scoped, or loopback

Both TCP listeners bind nothing until you set their port, so the port setting is the opt-in (see Listener deployment recipes for the full matrix and the fail-closed bind semantics). If your probes or scraper must traverse an untrusted network, put the auth in front (proxy, network policy); no header check exists at the endpoint.

A unix-socket collision does not stop the boot: the WARN does

If the Unix socket path cannot be bound, almost always a path a still-live peer (or a previous, still-draining replica) owns, the worker logs health-server-unavailable at WARN with the socket path and the errno, and keeps booting without that listener: it still registers, heartbeats, and claims work, because refusing to boot would crash-loop a healthy pair during a rolling restart. The failure is never silent: the WARN is the named, alertable record. What the configured address then answers depends on who holds it: a live peer's listener answers for the peer (its leadership, its shutdown phase, not this worker's), and an address nobody holds refuses connections, so treat the WARN as an action item: give each replica a unique socket path (or a per-replica directory), because until you do this replica's health is not observable at the address you configured.

When TASKQ_HEALTH_PORT is set, the collision costs the Unix surface alone: the TCP probe listener is bound anyway, so the port your manifest routes probes to keeps answering for this replica (the WARN carries tcp_listener=serving to say so). The Unix surface stays the action item above: unique path per replica, and with no port configured the answer is unchanged: warn, boot, no listener anywhere.

A TCP probe port that cannot bind refuses startup

The TCP listener is a different contract: the deployment manifest routed health probes for THIS replica to TASKQ_HEALTH_PORT, and a worker that boots without it answers nothing there, or worse, under a tcpSocket probe on a port some other process holds, the probes pass against the wrong process. When the port cannot be bound the worker logs health-http-bind-failed (ERROR) and exits with HealthTcpBindError instead of running with probes silently dead. Give each replica a unique port (a downward-API pod port or an orchestrator-managed value), not a shared one.

Azure Container Apps¶

ACA supports only httpGet and tcpSocket probes; there is no exec probe type (Health probes in Azure Container Apps, "Restrictions"). A Unix socket is therefore unprobeable there, and taskq health ready cannot be used as a probe. Set TASKQ_HEALTH_PORT and probe it over HTTP:

containers:
  - image: myregistry.azurecr.io/taskq-worker:1.0.0
    name: worker
    env:
      - name: TASKQ_HEALTH_PORT
        value: "8600"
    probes:
      - type: Liveness
        httpGet:
          path: /live
          port: 8600
        initialDelaySeconds: 5
        periodSeconds: 10
        failureThreshold: 3
      - type: Readiness
        httpGet:
          path: /ready
          port: 8600
        initialDelaySeconds: 5
        periodSeconds: 5
        timeoutSeconds: 5
        failureThreshold: 48
      - type: Startup
        httpGet:
          path: /live
          port: 8600
        initialDelaySeconds: 3
        periodSeconds: 3
        failureThreshold: 40

Replacing a TCP↔Unix bridge

Before TASKQ_HEALTH_PORT existed, the only way to probe a worker on ACA was to run a second TCP listener alongside it that forwarded each connection to the Unix socket. If you built one, you can now delete it and set TASKQ_HEALTH_PORT to the port it used to listen on: the probe config above is unchanged, and the endpoints, status codes and body shape are identical because the bridge was forwarding to this same handler. Drop the bridge module, its start/stop calls in your worker entrypoint, and its host/port settings; keep the port in your infrastructure template pointed at the same number.

Notes drawn from the ACA probe reference:

The probe port does not have to be the ingress target port: a worker with no ingress at all can still expose 8600 purely for probes. Port values must be integers; named ports are not supported.
Only one probe of each type per container.
If you enable ingress and define no probes, the portal adds default TCP probes against the ingress target port. A TCP probe only proves something accepted a connection, so prefer httpGet against /ready: that is the only variant that reflects Postgres reachability and shutdown state.
TASKQ_HEALTH_HOST defaults to 0.0.0.0, which is what the ACA runtime needs to reach the replica. Do not narrow it to 127.0.0.1 unless a sidecar in the same network namespace is the only prober.

Kubernetes¶

Both styles work. httpGet is the portable choice and the one to reach for if the same manifest must also run on ACA:

          ports:
            - name: health
              containerPort: 8600
          livenessProbe:
            httpGet:
              path: /live
              port: 8600
            initialDelaySeconds: 5
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /ready
              port: 8600
            initialDelaySeconds: 5
            periodSeconds: 10

with TASKQ_HEALTH_PORT=8600 in the container env. The exec probes shown in the Deployment above remain valid on Kubernetes and need no open port; they are the better choice when you would rather not expose one.

Azure App Service¶

App Service Health check pings a single path on the app's own HTTP port every minute and treats 200-299 as healthy (Monitor the health of App Service instances). It has no liveness/readiness split, so point it at /ready (the endpoint that reflects dependencies) and run the worker's health listener on the port App Service routes to.

Registering your own readiness checks¶

/ready covers TaskQ's own dependencies. To add your application's, register a check before starting the worker:

from taskq.worker.health import register_readiness_check


async def search_index_reachable() -> str | None:
    """Return None when healthy, or a reason string when not."""
    if not await index.ping():
        return "search index unreachable"
    return None


register_readiness_check("search_index", search_index_reachable)

Registered checks run on both transports and on taskq health ready. They fail closed: a check that raises, or that outruns TASKQ_HEALTH_READINESS_CHECK_TIMEOUT, makes the worker unready and its reason appears in the /ready body.

Add a PodDisruptionBudget (minAvailable: 1, selector matching app: taskq-worker) to prevent voluntary evictions from taking all workers offline during node drains.

terminationGracePeriodSeconds

Setting this just above TASKQ_TERMINATION_GRACE_PERIOD is not enough, and is the sizing mistake most likely to bite you.

The shutdown phases (DRAINING → CANCELLING → FORCING → RELEASING) are bounded by TASKQ_CANCELLATION_GRACE_PERIOD + TASKQ_CLEANUP_GRACE_PERIOD, but the bounded-close tail that unwinds after them is additive, and nothing enforces that it fits inside TASKQ_TERMINATION_GRACE_PERIOD, the shipped default does cover it, but a custom value below the worst case only warns at startup. Against a dead or hung Postgres/Redis: an Azure Cache failover, or a token expiry dropping every connection, i.e. exactly when you are being SIGTERMed; each of the 8 sequential bounded closes (four pools: dispatcher, heartbeat, worker, and the conditional per-slot transaction pool; plus notify_conn, the Redis client, and the two credential-provider closes a managed-identity deployment resolves) can take CLOSE_TIMEOUT_SECS (5s), plus a 2s progress-publish drain: 42s of tail.

Size the pod grace from the whole budget:

terminationGracePeriodSeconds
    >= TASKQ_CANCELLATION_GRACE_PERIOD
     + TASKQ_CLEANUP_GRACE_PERIOD
     + 42      # 8 bounded closes x 5s + 2s publish drain

At TaskQ's defaults (85 / 30 / 10) the modelled worst case is 82s, so terminationGracePeriodSeconds: 90 is a safe value at defaults, not the 60-ish the old advice implied. (The default grace covers the model with 3s to spare, but the ~87s sibling-crash path, nine sequential closes where the orchestrated leader-conn close never ran, including the conditional per-slot pool, now exceeds the default by 2s on per-slot workers. Deployments running the per-slot path with tight crash budgets should raise TASKQ_TERMINATION_GRACE_PERIOD accordingly.) The worker logs shutdown-budget-exceeds-termination-grace at startup, with the computed number, whenever the configured budget does not cover it, i.e. for custom grace combinations, not for the shipped defaults.

If the kubelet SIGKILLs mid-unwind, in-flight write_cancel_escalation / mark_interrupted release writes may not land; those jobs are left running and recovered later by the leader's crash-reclaim sweep after lock_lease expires, rather than being released cleanly.

Listener deployment recipes¶

The worker has up to three listeners, and every one of them is off unless you set it:

Listener	Enabled by	Serves	Auth
Unix health socket	`TASKQ_HEALTH_ENABLED` (default `true`)	`/live`, `/ready`, `/metrics`, opt-in `/tasks`	filesystem permissions on the socket (`0600` when `TASKQ_HEALTH_TASKS_ENABLED=true`)
TCP health listener	`TASKQ_HEALTH_PORT`	`/live`, `/ready`	none; `TASKQ_HEALTH_TOKEN` on `taskq ui serve` does not apply to it, so keep the port pod-network-only
Prometheus scrape listener	`TASKQ_METRICS_PORT` (needs `taskq[prometheus]` + `TASKQ_OTEL_AUTOCONFIGURE=true`)	`/metrics`	unauthenticated (see SECURITY.md); keep it on the pod network or loopback

Fail-closed semantics, identical across the platforms below:

Both TCP listeners bind nothing until you set a port. A worker that binds a port nobody asked for is a surprise network surface; setting the port is the opt-in.
A TCP health listener that cannot bind refuses startup. The worker logs health-http-bind-failed (ERROR) and exits with HealthTcpBindError rather than run with the manifest's probe port dead (a tcpSocket probe on a port another process holds would pass while checking nothing). The unix socket's collision is deliberately softer: a collision there means a live peer owns the path, so the boot warns (health-server-unavailable) and continues, and because the two transports bind independently, the TCP listener still comes up, so the port the manifest routes probes to keeps answering; only with no TASKQ_HEALTH_PORT configured does the collision leave the worker with no listener at all.
The scrape listener refuses startup. If [prometheus] or autoconfigure is missing the worker tells you and binds nothing; if the listener itself cannot be configured or bound, OtelExporterConfigurationError exits the worker with code 1 rather than run with its scrape silently dead.
The scrape endpoint answers without a token: keep it on interfaces only your scraper reaches.

Kubernetes¶

httpGet probes and annotations-driven scraping, both over the pod network (TASKQ_HEALTH_HOST defaults to 0.0.0.0, which pod-network probers need):

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9464"
    prometheus.io/path: /metrics
spec:
  template:
    spec:
      containers:
        - name: worker
          env:
            - name: TASKQ_HEALTH_PORT
              value: "8600"
            - name: TASKQ_METRICS_PORT
              value: "9464"
          ports:
            - name: health
              containerPort: 8600
            - name: metrics
              containerPort: 9464
          livenessProbe:
            httpGet: { path: /live, port: 8600 }
          readinessProbe:
            httpGet: { path: /ready, port: 8600 }

Loopback sidecar scraper next to pod-network probers: leave the probes on the pod network and pin the scrape listener to loopback with TASKQ_METRICS_HOST, so the unauthenticated endpoint is reachable only inside the pod:

            - name: TASKQ_METRICS_PORT
              value: "9464"
            - name: TASKQ_METRICS_HOST
              value: 127.0.0.1
          containers:
            - name: worker
              ...
            - name: prometheus-sidecar
              # scrapes http://127.0.0.1:9464/metrics and remote-writes

Azure Container Apps¶

ACA probes support only httpGet/tcpSocket, so TASKQ_HEALTH_PORT is required there: the Unix socket is unprobeable and the worker has no other surface a probe can reach.

env:
  - name: TASKQ_HEALTH_PORT
    value: "8600"
  - name: TASKQ_METRICS_PORT
    value: "9464"

Keep TASKQ_HEALTH_HOST at its 0.0.0.0 default: the ACA runtime reaches the replica over the pod network, and a loopback bind would fail every probe. The full probe definitions (liveness, readiness, startup) are in Health probes above.

AWS EC2¶

On a plain EC2 host (systemd unit as above) there is no orchestrator probe, so pick the shape that matches who reads the endpoints:

# /etc/taskq/worker.env
TASKQ_HEALTH_PORT=8600
TASKQ_METRICS_PORT=9464

Loopback bind + reverse proxy (the default posture): set TASKQ_HEALTH_HOST=127.0.0.1 and front the worker with nginx, which terminates the ALB target-group health check on http://127.0.0.1:8600/ready and can require-scrape /metrics. Nothing outside the host can reach either listener.
Security-group-scoped 0.0.0.0: leave TASKQ_HEALTH_HOST at its default and restrict the security group so only the scraper (or ALB) security group may reach 8600 and 9464. The scrape endpoint is unauthenticated, so the SG is the only thing standing between it and the VPC.

Either way the metrics listener still refuses startup on a bind failure (exit 1), and a health-port collision is a WARN plus failed probes, so give each instance on a shared host its own port.

docker / docker-compose¶

Publish the probe port, keep the health socket on a tmpfs, and let a sidecar scraper in the worker's network namespace see the metrics:

services:
  worker:
    image: myapp:latest
    environment:
      TASKQ_HEALTH_SOCKET_PATH: /run/taskq/health.sock   # tmpfs, not the image layer
      TASKQ_HEALTH_PORT: "8600"
      TASKQ_METRICS_PORT: "9464"
      TASKQ_METRICS_HOST: "127.0.0.1"   # scrape listener loopback-only
    tmpfs:
      - /run/taskq:size=1m,mode=0700
    ports:
      - "8600:8600"        # host probes / load balancer health check
    healthcheck:
      test: ["CMD", "taskq", "health", "ready"]
      interval: 10s
      timeout: 5s
      retries: 5
  prometheus:
    image: prom/prometheus
    network_mode: "service:worker"   # shares the worker loopback, scrapes 127.0.0.1:9464/metrics

The healthcheck uses the Unix socket and needs no published port; the 8600 mapping exists for host-level probes. If your scraper is a plain linked container on the compose bridge network instead of a loopback sidecar, drop TASKQ_METRICS_HOST so the scrape listener falls back to 0.0.0.0 and reach it as worker:9464, scoped by whatever firewall fronts the bridge.

Docker Compose for Production¶

A hardened Compose file with resource limits, healthchecks, restart policies, and a migration gate. Postgres and Redis are assumed to exist with healthchecks configured (see the dev docker-compose.yml for reference service definitions):

services:
  migrate:
    image: myapp:latest
    command: ["taskq", "migrate", "up"]
    environment:
      TASKQ_PG_DSN: postgresql://taskq:${POSTGRES_PASSWORD}@postgres:5432/taskq
    depends_on:
      postgres:
        condition: service_healthy
    restart: "no"

  worker:
    image: myapp:latest
    command: ["taskq", "worker", "--actors", "myapp.actors:registry"]
    restart: unless-stopped
    environment:
      TASKQ_PG_DSN: postgresql://taskq:${POSTGRES_PASSWORD}@postgres:5432/taskq
      TASKQ_REDIS_URL: redis://redis:6379/0
      TASKQ_ENVIRONMENT: production
      TASKQ_MAX_CONCURRENCY: "16"
      TASKQ_QUEUES: default,priority
      TASKQ_TERMINATION_GRACE_PERIOD: "120"
      TASKQ_CANCELLATION_GRACE_PERIOD: "60"
      TASKQ_CLEANUP_GRACE_PERIOD: "20"
      TASKQ_LOCK_LEASE: "90"
      OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4317
      OTEL_SERVICE_NAME: taskq-worker
    depends_on:
      migrate:
        condition: service_completed_successfully
      redis:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "taskq", "health", "ready"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 15s
    stop_grace_period: 130s
    deploy:
      resources:
        limits:
          cpus: "2.0"
          memory: 1G

The admin UI service follows the same pattern with command: ["taskq", "ui", "serve"], TASKQ_ADMIN_UI_REQUIRE_AUTH: "false", and TASKQ_HEALTH_REQUIRE_TOKEN: "false" (see Admin UI in Production). Add a migrate dependency to both worker and admin services so they wait for migrations to complete.

stop_grace_period must exceed termination_grace_period

Docker's stop_grace_period (default 10s) controls how long Compose waits between SIGTERM and SIGKILL. Set it above TASKQ_TERMINATION_GRACE_PERIOD plus the ~42s bounded-close tail (see the terminationGracePeriodSeconds warning above) so the worker can complete its shutdown sequence.

Workgroup Deployment¶

The workgroup supervisor (taskq workgroup start) manages multiple taskq worker subprocesses within a single container, each with independent queue subscriptions and concurrency caps.

Approach	Use when
Workgroup	Multiple queue groups in one container; single-pod simplicity
Separate Deployments	Independent scaling per queue; independent rolling deploys

actors = "myapp.actors:registry"

[defaults]
poll_interval = 1.0
max_concurrency = 4

[[workers]]
name = "api"
queues = ["default", "priority"]
max_concurrency = 16
poll_interval = 0.5

[workers.health]
enabled = true
check_interval = 15
stale_after = 60

[[workers]]
name = "batch"
queues = ["email", "report", "cleanup"]
max_concurrency = 2
poll_interval = 5.0

taskq workgroup start /etc/taskq/workgroup.toml

The supervisor assigns a workgroup_instance UUIDv7 for cross-process correlation in the workers table.

Supervisor is a single point of failure

If the supervisor crashes (e.g. OOM), managed workers become orphaned and are reclaimed by the leader's sweep after lock_lease expires. Always run it under a process manager (systemd, Docker, Kubernetes) for restart.

See workgroups.md for the full configuration reference and restart policy.

Observability Setup¶

TaskQ instruments itself with OpenTelemetry (vendor-neutral) and structlog. No vendor SDK is bundled; export to any OTLP-compatible backend via standard OTel environment variables. Install the OTel extra: uv add "taskq-py[otel]".

OTel exporter configuration¶

Variable	Example	Description
`OTEL_EXPORTER_OTLP_ENDPOINT`	`http://otel-collector:4317`	OTLP gRPC endpoint (`:4318` with `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf`)
`OTEL_TRACES_EXPORTER` / `OTEL_METRICS_EXPORTER`	`otlp`	Exporter per signal; defaults to `otlp` once an endpoint is set
`OTEL_SERVICE_NAME`	`taskq-worker`	Service name for all spans/metrics
`OTEL_RESOURCE_ATTRIBUTES`	`deployment.environment=production,k8s.pod.name=worker-0`	Resource attributes on all telemetry

With the [otel] extra installed, taskq worker installs SDK tracer and meter providers from these variables at startup (through the SDK's own configurator, the same as opentelemetry-instrument taskq worker) and logs otel-exporter-configured traces=... metrics=... source=env. Verify that line in the worker's startup log: otel-exporter-unavailable means the variables are set but the extra is missing, and nothing is exported. TASKQ_OTEL_AUTOCONFIGURE=false opts out when the embedding application configures the SDK itself. TaskQ does not override standard OTel variables. All spans and metrics use instrumentation name "taskq" and OTel messaging semconv attributes for dashboard compatibility.

Prometheus scrape¶

The worker series (leader-sampled gauges, dispatch/consume counters, attempt failures, the watchdog family) exist only in the worker processes; the admin UI's /jobs/health/metrics serves the admin process's own activity and none of them. Scrape every worker pod: install taskq[prometheus], set TASKQ_METRICS_PORT=9464, and point a scrape job at each worker on that port with metrics_path: /metrics (a pod-role discovery with a taskq-worker selector, or a headless Service). The listener binds TASKQ_METRICS_HOST when set, falling back to TASKQ_HEALTH_HOST (0.0.0.0); set TASKQ_METRICS_HOST=127.0.0.1 for a sidecar scraper that shares the pod's loopback while the health probes stay on the pod network. The port and host writes are scoped to the SDK call and rolled back, so neither leaks to child processes. A worker missing the [prometheus] extra binds no listener and says so; one that cannot configure or bind the listener exits 1 rather than run with its scrape silently dead. The endpoint is unauthenticated (see SECURITY.md), so keep it on the pod network or loopback. The worker's health socket additionally serves three process gauges at GET /metrics without any extra. See observability.md: Serving the metrics.

Structured logging¶

Use TASKQ_LOG_FORMAT=json (the default) in production. Every log line includes worker_id, timestamp (ISO 8601 UTC), level, and trace_id/span_id from the active OTel span. See observability.md for the full span hierarchy and metrics reference.

Scaling Considerations¶

Horizontal scaling¶

Adding worker processes is the primary scaling lever. Multiple workers against the same database are fully supported: dispatch uses FOR UPDATE SKIP LOCKED, so concurrent workers never pick up the same job. Only one worker per schema holds the maintenance leader advisory lock (the lock name is schema-qualified, so each schema in a shared database elects its own leader); if the leader dies, another worker wins the next election. Scale by increasing replica count (kubectl scale deployment taskq-worker --replicas=10).

Queue partitioning¶

Partition work across named queues and assign workers to specific subsets to prevent a deep backlog on one queue from starving others:

TASKQ_QUEUES=default,priority taskq worker --actors myapp.actors:registry
TASKQ_QUEUES=media taskq worker --actors myapp.actors:registry

For multi-tenant queues, set round_robin mode to interleave by fairness_key cohort:

taskq queues set-mode multi round_robin

See workers.md: Queue dispatch modes.

max_concurrent and max_pending¶

max_concurrent (per-actor via @actor(max_concurrent=N)) is a best-effort fleet-wide damper on how many jobs for an actor run simultaneously, distinct from TASKQ_MAX_CONCURRENCY (total jobs per process).

max_concurrent is not a hard cap

Dispatch reads its running count once per round, before taking row locks, and never rechecks it. Two worker replicas dispatching concurrently each see the same count, each admit up to the cap, and lock disjoint rows, so both succeed. The over-dispatch bound is (num_producers - 1) * max_concurrent per round, and those jobs genuinely run; reclaiming stale locks does not undo an over-dispatch.

With max_concurrent=2 and 3 replicas you can see 6 concurrent executions. For a memory- or GPU-bound actor that is an OOMKill, a restart, and a re-dispatch.

If you need a strict cap, use the per-queue leased-slot reservation instead: taskq queues set-max-concurrent <queue> --max-concurrent N. Slots are physical rows and each acquire is a single read-and-write statement on one row, so there is no read-then-decide window. See rate-limiting.md. Note the asymmetry with the per-actor damper: taskq queues set-max-concurrent writes the queues row immediately, but each running worker read the cap into memory at startup, so every worker keeps enforcing the OLD value until it restarts; a cap change lands fleet-wide only after the workers serving that queue have been restarted (a rolling restart is fine, one replica at a time). A per-actor change via taskq actor-config set needs no restart: dispatch re-reads actor_config every round and picks the new value up on the next cycle. taskq queues get shows the database value; it cannot tell you which workers have restarted onto it, so treat the change as live only after the fleet has rolled. max_pending (per-actor via @actor(max_pending=N)) caps queued pending jobs; when exceeded, enqueue is rejected and taskq.backpressure.errors is incremented with kind="max_pending". Monitor taskq.queue.depth (leader samples every 15s) for backlog and taskq.backpressure.errors for sustained producer pressure; filter on the kind label to the capacity kinds (max_pending, max_pending_lock_timeout), because the same counter also carries identity-serialization refusals (unique_for_lock_timeout, idempotency_lock_timeout) that page on contention unrelated to capacity.

Connection pool sizing¶

Pool	Default	Scales with
`dispatcher_pool`	4	`TASKQ_DISPATCHER_POOL_SIZE`
`heartbeat_pool`	4	`TASKQ_HEARTBEAT_POOL_SIZE`
`worker_pool`	`int(max_concurrency * 1.5)`	`TASKQ_MAX_CONCURRENCY`
slot pool (conditional)	`max_concurrency + 1` (direct DSN)	`TASKQ_MAX_CONCURRENCY`
`notify_conn` + `leader_conn`	2 (dedicated)	Fixed

The slot pool exists only when a LOOP-scope asyncpg.Connection is registered and max_concurrency > 1, the per-slot transaction pool that carries every per-job transaction (the actor's own writes via its injected slot connection, the terminal write, transactional sub-enqueues). Total per worker ≈ dispatcher + heartbeat + worker_pool + 2, plus max_concurrency + 1 direct connections on the per-slot path. For 10 workers at max_concurrency=16: ~10 × 34 = 340 connections, or ~10 × 51 = 510 on the per-slot path (each worker adds 17 direct). On that path TASKQ_MAX_CONCURRENCY is boot-blocking: the worker opens max_concurrency + 1 direct connections at startup and fails to boot if it cannot, and a credential-rotation window peaks at 2 × (max_concurrency + 1) slot connections per worker (old pool draining + new pool warm, bounded by the reload drain timeout), so plan max_connections against that peak whenever rotations can coincide across the fleet (a scheduled TASKQ_RELOAD_INTERVAL is exactly that), against the steady state only when rotations are staggered. Ensure Postgres max_connections accommodates this plus your application's connections. The full budget formula (idle floors, leader extras, client pods, PgBouncer compression) is in ops.md: Sizing.

Security Hardening¶

Admin UI authentication¶

Embed in your FastAPI app with an auth_dependency callable (HTTPBearer, OIDC, session middleware). See admin-ui.md.
Or run behind a reverse proxy with auth (nginx basic auth, OAuth2 proxy, mTLS) and set TASKQ_ADMIN_UI_REQUIRE_AUTH=false.
Keep TASKQ_ADMIN_ACTIONS_ENABLED=false unless operators need cancel/retry/run-now: these are write operations that modify job state.
Set TASKQ_HEALTH_TOKEN for machine-to-machine auth on the health/metrics endpoints taskq ui serve exposes (the worker's own TCP health and scrape listeners do not check it), or explicitly set TASKQ_HEALTH_REQUIRE_TOKEN=false if relying on network policy.

Network policies¶

Port	Service	Exposed to
5432	Postgres	Workers, admin UI, migrate jobs only
6379	Redis	Workers, admin UI only
8080	Admin UI	Internal operators only; never public
8600 (`TASKQ_HEALTH_PORT`)	Worker TCP health listener (`/live`, `/ready`)	Orchestrator probes / load balancer health checks only (pod network, SG or loopback plus proxy)
9464 (`TASKQ_METRICS_PORT`)	Worker Prometheus scrape listener (`/metrics`, unauthenticated)	The scraper only: pod network, security-group-scoped, or loopback sidecar
Unix socket	Worker health	Same pod only (exec probes)

The admin UI should never be exposed to the public internet without an authentication layer. Use Kubernetes NetworkPolicy resources to restrict pod-to-pod communication.

Database credentials¶

Store TASKQ_PG_DSN and TASKQ_REDIS_URL in a secret manager (Kubernetes Secret + secretKeyRef as shown in the Deployment manifest), not in image layers or git. Use a dedicated Postgres role for TaskQ with least-privilege permissions: CREATE on the schema for migrations, SELECT, INSERT, UPDATE, DELETE on all TaskQ tables for runtime.

Redis ACLs¶

If Redis is shared, restrict the TaskQ user to its keyspace:

redis-cli ACL SETUSER taskq on >${REDIS_PASSWORD} ~taskq:* +@all -@dangerous

Redis TLS

For managed Redis (ElastiCache, MemoryDB, Azure Cache), use rediss:// to enable TLS: TASKQ_REDIS_URL=rediss://redis.internal:6379/0.