Skip to content

Installation

This guide covers prerequisites, installation, and Docker setup for local development and integration testing.

Prerequisites

  • Python 3.12+ (3.13 and 3.14 are also supported — the CI matrix runs all three majors)
  • Postgres: 15 is the minimum supported major; 15–18 are covered by CI — the system-e2e tier runs its full multi-process lifecycle against every major via a testcontainers version matrix. docker-compose.yml pins the postgres:18 image for local development (PG 18 is also what the fast test suite uses). No PG18-specific SQL exists in the bundled migrations; the floor is 15 because that is the oldest major the CI matrix exercises — older majors may work but are untested and unsupported. Production requires a direct connection (not PgBouncer in transaction-pooling mode) for advisory locks and LISTEN/NOTIFY.
  • Redis (optional): required only for real-time progress fanout and admin UI live updates.
  • uv or pip for package management.

Install

Name check: the package is taskq-py. There is an unrelated taskq distribution on PyPI — pip install "taskq[redis]" installs that package, not TaskQ. Always use the taskq-py name.

pip install taskq-py                      # core only
pip install "taskq-py[redis]"             # + real-time SSE fanout
pip install "taskq-py[fastapi]"           # + admin UI
pip install "taskq-py[redis,fastapi]"     # full

Or with uv:

uv add taskq-py                           # core only
uv add "taskq-py[redis,fastapi]"          # full

Extras:

Extra Installs Features enabled
taskq-py[redis] redis>=8.0.1 Real-time progress fanout via Redis pub/sub, Redis-backed rate limiters (TokenBucket, SlidingWindow)
taskq-py[otel] opentelemetry-sdk, opentelemetry-exporter-otlp, opentelemetry-instrumentation Configuring OTel providers programmatically; in-process test utilities in taskq.testing.otel
taskq-py[fastapi] fastapi, jinja2, sse-starlette, uvicorn, humanize, python-multipart Admin UI (taskq ui serve), SSE progress bridge
taskq-py[prometheus] opentelemetry-exporter-prometheus, prometheus-client Prometheus metric scrapes via taskq.contrib.prometheus
taskq-py[oidc] authlib>=1.8, httpx2, itsdangerous OIDC/SSO auth backend for the admin UI; see SSO / SAML. The OIDC path uses httpx2 as its HTTP client (authlib ≥ 1.8 is httpx2-first); httpx is not installed by this extra
taskq-py[saml] python3-saml, itsdangerous SAML/SSO auth backend for the admin UI; see SSO / SAML
taskq-py[aad] azure-identity, aiohttp Azure Entra ID managed-identity DB auth; see Managed Identities
taskq-py[aws] boto3 AWS IAM RDS auth for Postgres; see Managed Identities
taskq-py[vault] hvac HashiCorp Vault dynamic credentials; see Managed Identities
taskq-py[reload] watchfiles Autoreload of workers and the admin UI during local development
taskq-py[bench] py-spy, pyinstrument Profilers for the benchmarks/ toolkit (spot-check profiling; nothing under src/taskq imports them)

Without an extra installed, the corresponding feature degrades gracefully or raises a clear ImportError with install instructions:

Feature Without extra Behavior
Progress fanout No [redis] ctx.progress() still coalesces and flushes to Postgres. JobHandle.progress_stream() falls back to 500 ms PG polling. HTTP SSE endpoint returns 503.
Redis rate limiters No [redis] TokenBucket(backend="redis") / SlidingWindow(backend="redis") raise ImportError at acquire time. Use backend="postgres" or backend="memory" instead.
Admin UI No [fastapi] taskq ui serve raises ImportError. The admin UI requires FastAPI.
OTel SDK providers No [otel] taskq.testing.otel raises ImportError. The library emits OTel API spans/metrics regardless (they are no-ops without a configured provider).
Prometheus bridge No [prometheus] taskq.contrib.prometheus raises ImportError.

Docker Compose for integration tests

The bundled docker-compose.yml starts Postgres, Redis, and the admin UI in one command. This is the fastest path to a running local environment.

docker compose up -d

Services started:

Service Port Notes
postgres 5432 Postgres with max_connections=200, shared_buffers=256MB
redis 6379 Redis without persistence (appendonly no)
admin 8080 TaskQ admin UI (runs taskq ui serve --migrate on startup) on startup

The admin service runs taskq ui serve --migrate which applies pending migrations before starting the UI. When using the full compose stack you do not need to run migrations manually.

To start Postgres and Redis only (for running the worker locally outside Docker):

docker compose up -d postgres redis

Environment setup

Copy the example env file and adjust as needed:

cp .env.example .env

No env var is strictly required: TASKQ_PG_DSN defaults to postgresql://taskq:taskq@localhost:5432/taskq. For any real deployment, set it to your actual database.

# Direct PG DSN: sessions, LISTEN/NOTIFY, and advisory locks require this.
TASKQ_PG_DSN=postgresql://taskq:taskq@localhost:5432/taskq

# Schema name for all TaskQ tables. Override if multi-tenanting.
TASKQ_SCHEMA_NAME=taskq

# Optional. Enables real-time progress fanout and admin UI live updates.
TASKQ_REDIS_URL=redis://localhost:6379/0

TASKQ_SCHEMA_NAME is the one schema truth for every entry point: the worker, the CLI, and the TaskQ client all resolve it (the client consults it whenever TaskQ(schema=...) is omitted; an explicit schema= always wins). Keep it identical across every process that enqueues or consumes: an enqueuer on a different schema than its workers lands jobs that are never picked up, with no error raised.

PgBouncer

Advisory locks and LISTEN/NOTIFY require a direct Postgres connection. Do not point TASKQ_PG_DSN at a PgBouncer endpoint in transaction-pooling mode. Use TASKQ_PG_DSN_DIRECT and TASKQ_PG_DSN_POOLED to split traffic; see Configuration.

TaskQ loads configuration through dotenvmodel with cascading .env discovery: .env → .env.local → .env.{env} → .env.{env}.local, where {env} comes from the ENV variable (default dev). Real environment variables take precedence over .env files; see Configuration for the full resolution rules.

Run migrations

Apply all pending migrations before starting a worker:

taskq migrate up

The command is idempotent: re-running against an up-to-date schema is a no-op. To inspect applied and pending migrations without making changes:

taskq migrate status

Alternatively, set TASKQ_MIGRATE_ON_START=true to have the admin UI apply migrations automatically at startup. Production workers should still run taskq migrate up manually before the worker process starts.

Next steps