Client¶
TaskQ, JobsClient, JobHandle, CancelResult, and ActorsClient.
_taskq ¶
Top-level entry point for non-worker applications.
Provides :class:TaskQ — a Postgres-backed client that manages its own
connection pool and exposes job operations (enqueue, get, list, cancel)
directly.
Two lifecycle patterns are supported:
Async context manager (scripts, tests)::
async with TaskQ(dsn="postgresql://user:pw@host/db") as tq:
handle = await tq.enqueue(my_actor, MyPayload(...))
result = await handle.wait()
Explicit open/close (FastAPI lifespan, long-lived processes)::
tq = TaskQ(dsn=settings.pg_dsn)
@asynccontextmanager
async def lifespan(app: FastAPI):
await tq.open()
yield
await tq.close()
@app.post("/tasks")
async def create_task(payload: MyPayload):
handle = await tq.enqueue(my_actor, payload)
return {"job_id": str(handle.job_id)}
Passing an existing pool (e.g. shared with the rest of the application)::
async with TaskQ(pool=app.state.pool) as tq:
await tq.cancel(job_id)
EventRow
dataclass
¶
EventRow(
event_id: int,
job_id: JobId,
occurred_at: datetime,
kind: Literal["state_change", "cancel_request"],
detail: dict[str, object],
)
Read-model of a taskq.job_events row.
Mirrors the job_events table shape: monotonic event_id,
the owning job, timestamp, event kind, and a detail payload.
ActorsClient ¶
Pool-wrapping facade for actor configuration operations.
Acquires a connection from the injected pool for each call, delegates
to taskq.actor_config_ops, and returns the result. The
caller must have opened the pool; this class does not manage its
lifecycle.
.. note::
This client is Postgres-only — it delegates to
:mod:taskq.actor_config_ops, which executes raw SQL against
the actor_config table. The :class:~taskq.backend._protocol.Backend
protocol does not include actor config operations, so
:class:~taskq.testing.InMemoryBackend does not support
ActorsClient.
Parameters¶
pool:
An open asyncpg.Pool. The caller retains ownership.
schema:
TaskQ schema name. Defaults to "taskq".
Source code in src/taskq/client/_actors.py
list
async
¶
List all stored actor_config rows, ordered by actor name.
get
async
¶
Get one actor_config row, or None if not found.
set_capacity
async
¶
set_capacity(
actor: str,
*,
max_concurrent: int | Unset | None = UNSET,
max_pending: int | Unset | None = UNSET,
result_ttl: float | Unset | None = UNSET,
) -> ActorConfigRow | None
Update capacity fields on an existing actor_config row.
Source code in src/taskq/client/_actors.py
deregister
async
¶
Deregister an actor with safety checks.
See :func:taskq.actor_config_ops.deregister_actor for
the full semantics.
Source code in src/taskq/client/_actors.py
JobEvent ¶
Bases: BaseModel
A single event yielded by :meth:TaskQ.stream.
Represents a point-in-time snapshot of a job's observable state.
Yielded on every status transition or progress update; the final
event always has terminal=True.
The progress_state and progress_seq fields reflect the last
values written by the worker. They are None / 0 until the
worker emits a progress update.
Serialises cleanly to JSON via model_dump() for SSE or WebSocket
fanout — fields are deliberately flat so the caller can forward the
event without transformation::
async for event in tq.stream(job_id):
await websocket.send_json(event.model_dump())
TaskQ ¶
TaskQ(
*,
dsn: str | None = None,
pool: Pool | None = None,
pool_factory: PoolFactory | None = None,
pg_provider: PgCredentialProvider | None = None,
schema: str = "taskq",
min_pool_size: int = 1,
max_pool_size: int = 5,
redis_url: str | None = None,
redis_client: Any | None = None,
pg_conn_factory: ConnFactory | None = None,
listen_conn: Connection | None = None,
poll_timeout: float = 30.0,
reclaim_event_visibility_delay: timedelta | None = None,
)
Postgres-backed TaskQ client.
Manages a connection pool and exposes job operations directly. Supports
both the async context manager pattern and explicit open() / close()
for frameworks like FastAPI that manage their own lifecycle.
Parameters¶
dsn:
Postgres DSN string. Mutually exclusive with pool.
pool:
An already-open asyncpg.Pool. The caller retains ownership;
close() will not close it.
pool_factory:
A zero-arg async factory returning an asyncpg.Pool - the same
:data:~taskq.connections.PoolFactory the worker takes via
:class:~taskq.connections.WorkerConnections. TaskQ invokes it at
open() and owns the result (close() closes it), and
:meth:reload_credentials re-invokes it to rotate the pool in place.
This is the client-side equivalent of a <role>_pool_factory:
pair it with :func:taskq.auth.make_pg_pool_factory for a
rotating-credential deployment. Mutually exclusive with dsn and
pool.
pg_provider:
A :class:~taskq.auth.PgCredentialProvider. Sugar for
pool_factory=make_pg_pool_factory(dsn, pg_provider,
min_size=min_pool_size, max_size=max_pool_size) - it is exactly
that call and nothing more, so the two paths share one mechanism.
Requires dsn; use pool_factory directly when you need the
factory's other hooks (init, server_settings,
command_timeout).
schema:
TaskQ schema name. Defaults to "taskq".
min_pool_size:
Minimum pool connections. Only used when dsn is provided.
max_pool_size:
Maximum pool connections. Only used when dsn is provided.
redis_url:
Redis URL string. Mutually exclusive with redis_client.
The library creates and owns the Redis client; close() will
close it.
redis_client:
An already-open redis.asyncio.Redis client. The caller retains
ownership; close() will not close it. Mutually exclusive with
redis_url.
pg_conn_factory:
A zero-arg async factory returning an asyncpg.Connection for the
LISTEN/NOTIFY transport used by :meth:stream. Mutually exclusive
with listen_conn. Takes precedence over dsn when set. Use
this when you have no DSN (e.g. AAD-managed-identity auth) but still
want streaming. TaskQ owns and closes the connection produced by
the factory per stream() call.
listen_conn:
A pre-constructed asyncpg.Connection for the LISTEN transport.
Caller-owned; TaskQ does not close it. Mutually exclusive with
pg_conn_factory. Takes precedence over dsn when set. Use
this to share a dedicated LISTEN conn across callers.
poll_timeout:
Maximum seconds to wait between transport wakeups before re-fetching
job state. Defaults to 30.0.
Source code in src/taskq/client/_taskq.py
218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 | |
actors
property
¶
Actor configuration client — list, get, set capacity, deregister.
Raises RuntimeError if called before open() or outside an
async with block.
open
async
¶
Open the connection pool and prepare the client.
Called automatically by __aenter__. Safe to call explicitly
for frameworks that manage lifecycle outside an async with block.
Raises :class:RuntimeError if already open.
Source code in src/taskq/client/_taskq.py
close
async
¶
Close the client and release the pool if owned.
Called automatically by __aexit__. Safe to call explicitly.
No-op if already closed.
Source code in src/taskq/client/_taskq.py
reload_credentials
async
¶
Rebuild the pool from pool_factory and swap it in, live.
The client-side counterpart of
:func:taskq.worker.deps.reload_credentials, and the supported
replacement for reaching into _pool / _client: it invokes the
factory (which fetches a fresh credential), atomically points every
subsystem that holds a pool - the backend deps behind
enqueue/get/list/cancel, the :attr:actors client, and
the stream() LISTEN fallback - at the new pool, then closes the old
one with the same bounded drain used at close().
Call this on a schedule shorter than your credential's lifetime (an
Entra access token is typically ~60 min), or on any signal your issuer
gives you. It is not needed for the ordinary token refresh that
:func:taskq.auth.make_pg_pool_factory already does per physical
connection; it is how you drop sessions opened under a revoked
credential, and the only way to pick up a changed username, which
asyncpg resolves once per pool.
Raises :class:RuntimeError if the client is not open, or if the pool
is caller-owned (pool=) - TaskQ must never close a pool it does not
own, so the caller rotates that one itself.
If the factory fails, the exception propagates and the current pool is left untouched and serving: a transient token-endpoint outage must not turn into a client outage.
Source code in src/taskq/client/_taskq.py
__aenter__
async
¶
__aexit__
async
¶
enqueue
async
¶
enqueue(
ref: ActorRef[P, R],
payload: P,
*,
queue: QueueName | None = None,
scheduled_at: datetime | None = None,
priority: int | None = None,
schedule_to_close: datetime | None = None,
start_to_close: timedelta | None = None,
heartbeat_timeout: timedelta | None = None,
identity_key: IdentityKey | None = None,
fairness_key: str | None = None,
idempotency_key: IdempotencyKey | None = None,
idempotency_scope: str | None = None,
trace_id: str | None = None,
span_id: str | None = None,
metadata: dict[str, object] | None = None,
tags: list[str] | None = None,
) -> JobHandle[R]
Enqueue a job and return a typed handle.
schedule_to_close (absolute datetime) is deprecated — it crosses
clock domains (the app clock that produced it vs the database clock
that evaluates it). Declare retry.time_budget on the actor
instead; the interval form is anchored to the database clock.
Source code in src/taskq/client/_taskq.py
enqueue_batch
async
¶
enqueue_batch(
items: list[EnqueueItem],
*,
batch_id: UUID | None = None,
connection: Connection | None = None,
failure_policy: BatchFailurePolicy | None = None,
finalizer: EnqueueItem | None = None,
) -> BatchHandle
Enqueue multiple jobs in a single batched INSERT.
Delegates to :meth:JobsClient.enqueue_batch; see its docstring
for validation rules and idempotency-key collision semantics.
Source code in src/taskq/client/_taskq.py
enqueue_batch_streaming
async
¶
enqueue_batch_streaming(
items: Iterable[EnqueueItem],
*,
batch_id: UUID | None = None,
connection: Connection | None = None,
failure_policy: BatchFailurePolicy | None = None,
finalizer: EnqueueItem | None = None,
chunk_size: int = DEFAULT_CHUNK_SIZE,
) -> BatchHandle
Enqueue jobs from a lazy iterable in chunks.
Delegates to :meth:JobsClient.enqueue_batch_streaming; see its
docstring for chunk_size validation and streaming semantics.
Source code in src/taskq/client/_taskq.py
get_batch
async
¶
Fetch a single batch row by ID.
Delegates to :meth:JobsClient.get_batch. Returns None when
the batch does not exist.
Source code in src/taskq/client/_taskq.py
list_batches
async
¶
List batches matching filter, returning :class:BatchSummary objects.
Delegates to :meth:JobsClient.list_batches.
Source code in src/taskq/client/_taskq.py
enqueue_batch_fast
async
¶
enqueue_batch_fast(
items: list[EnqueueItem],
*,
batch_id: UUID | None = None,
connection: Connection | None = None,
) -> int
Enqueue jobs via COPY FROM protocol for maximum throughput.
Delegates to :meth:JobsClient.enqueue_batch_fast; see its
docstring for tradeoffs vs the regular :meth:enqueue_batch.
Source code in src/taskq/client/_taskq.py
get
async
¶
Look up a job by id. Returns None when the job does not exist.
Source code in src/taskq/client/_taskq.py
get_row
async
¶
Look up a job by id and return the raw JobRow — no handle.
Delegates to :meth:JobsClient.get_row: one backend read,
None when the job does not exist, mirroring :meth:get's
contract without the handle machinery.
Source code in src/taskq/client/_taskq.py
list
async
¶
List jobs matching filter, returning a :class:JobPage.
Delegates to :meth:JobsClient.list — note filter.active
is not Celery's 'active' ('currently executing'); it selects by
terminality ('not yet finished'). See :class:JobFilter.
Source code in src/taskq/client/_taskq.py
cancel
async
¶
Request cancellation of a job. Raises :class:KeyError if not found.
cancel_where
async
¶
cancel_where(
filter: JobFilter,
reason: str | None = None,
*,
allow_empty_filter: bool = False,
) -> BulkCancelResult
Cancel all jobs matching filter. See :meth:JobsClient.cancel_where.
Source code in src/taskq/client/_taskq.py
create_schedule
async
¶
create_schedule(
actor: str | ActorRef[P, R],
cron_expr: str,
*,
timezone: str = "UTC",
dst_strategy: DstStrategy = "skip",
payload_factory: str | None = None,
static_payload: dict[str, object] | None = None,
name: str = "",
identity_key: IdentityKey | None = None,
enabled: bool = True,
) -> ScheduleHandle
Create a cron schedule. Delegates to :meth:JobsClient.create_schedule.
dst_strategy controls how DST gaps/overlaps are handled; see
:meth:JobsClient.create_schedule for the full semantics.
Source code in src/taskq/client/_taskq.py
list_schedules
async
¶
List cron schedules. Delegates to :meth:JobsClient.list_schedules.
Source code in src/taskq/client/_taskq.py
update_schedule
async
¶
update_schedule(
schedule_id: UUID,
*,
cron_expr: str | None = None,
enabled: bool | None = None,
payload_factory: str | None = None,
static_payload: dict[str, object] | None = None,
clear_payload_factory: bool = False,
) -> ScheduleRecord
Update a cron schedule. Delegates to :meth:JobsClient.update_schedule.
Source code in src/taskq/client/_taskq.py
delete_schedule
async
¶
Delete a cron schedule. Delegates to :meth:JobsClient.delete_schedule.
stream
async
¶
Stream live state changes for a job as :class:JobEvent objects.
Yields one event per observable state transition (status change or
progress update), terminating automatically when the job reaches a
terminal state. The final event always has ``terminal=True``.
Usage::
async for event in tq.stream(job_id):
print(event.status, event.progress_state)
# loop exits automatically when event.terminal is True
# Or wire directly into a FastAPI SSE response:
async def event_generator():
async for event in tq.stream(job_id):
yield f"data: {event.model_dump_json()}
"
Raises
------
RuntimeError
Called before ``tq.open()`` or outside an ``async with`` block.
KeyError
The job does not exist.
RuntimeError
PG LISTEN transport requested but ``dsn`` was not provided at
construction (pool-only mode).
Source code in src/taskq/client/_taskq.py
watch_reclaims
async
¶
watch_reclaims(
after_id: int = 0, *, poll_timeout: float | None = None
) -> AsyncIterator[EventRow]
Stream fleet-wide crash-reclaim events as :class:EventRow objects.
Delivery guarantee. At-least-once, and gap-free only under a
bounded writer-transaction assumption: an event is silently and
permanently missed if a job_events writer transaction stays
open longer than reclaim_event_visibility_delay (default 2s —
see :data:taskq.constants.RECLAIM_EVENT_VISIBILITY_DELAY)
between its INSERT and its COMMIT. Ids are allocated at INSERT
time but transactions commit out of order, so a late-committing
lower-id row can land behind the cursor after the cursor has
already advanced past its position; no error is raised anywhere.
Sweep and terminal-write transactions are a handful of
single-round-trip statements, so 2s is a generous bound under
normal operation — but it is an assumption enforced by nothing in
the SQL, not a property the query guarantees. So this watcher
does not leave detection to chance: on a slow cadence
(_VISIBILITY_RISK_CHECK_INTERVAL, 60s) it runs the backend's
check_reclaim_visibility_delay_risk diagnostic (when the
backend implements it — PostgresBackend does) and logs a loud
watch-reclaims-visibility-delay-at-risk warning for every
long-open job_events writer it finds. That is a proxy
warning, not proof of an actual miss.
Yields job_events rows with kind='state_change' and
detail['reason']='lock_expired', ordered by the monotonic
event_id cursor ascending. to_state is 'pending' for
a retried reclaim, 'crashed' or 'cancelled' (cancel was
in-flight when the worker died) for a terminal one.
Cursor and duplicate semantics¶
The caller persists the last-seen event_id and passes it back
as after_id on resumption. Persist the cursor after
processing each event: a crash between processing and persisting
re-delivers that event on the next run — delivery is
at-least-once, so consumers must dedupe on event_id. The
cursor is a watermark, not a reference: pruning job_events
rows at or below it is always safe, but rows pruned before the
consumer reads them are gone for good — only prune rows older
than your slowest consumer's cursor. A cursor far behind after
a long outage drains at query speed (full batches are re-polled
immediately, not one batch per poll_timeout).
Shutdown and backpressure¶
This is a pull-based async generator: events are fetched only as
fast as the consumer iterates, so a slow consumer simply polls
slower — no internal buffer grows. To stop, break out of the
async for (or cancel the consuming task); generator cleanup
removes the LISTEN registration and closes any owned connection.
Transport¶
On Postgres with a LISTEN transport source (dsn,
pg_conn_factory, or listen_conn), the method LISTENs on
wake_channel(schema) purely as a low-latency wakeup, but
always polls backend.poll_reclaim_events(after_id) as the
durable source of truth — NOTIFY is an optimisation, never the
only path, and a dropped LISTEN connection degrades to (and
recovers from) polling automatically. Without a LISTEN
transport or on Redis-configured backends, a plain poll loop
against poll_reclaim_events runs on poll_timeout.
Usage::
cursor = await load_cursor() # your own durable store
async for evt in tq.watch_reclaims(after_id=cursor):
outstanding -= 1 # fan-out completion tracking
cursor = evt.event_id
await save_cursor(cursor) # persist AFTER processing
Raises¶
RuntimeError
Called before tq.open() or outside an async with block.
Source code in src/taskq/client/_taskq.py
777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 | |
_jobs ¶
JobsClient — the primary entry point for enqueuing, querying, and cancelling jobs, and managing cron schedules.
Wraps a :class:~taskq.backend._protocol.Backend instance and adds the
client-layer behaviours the protocol intentionally omits: payload
serialization through the actor's payload_type, CancelResult
construction in :meth:cancel, typed :class:JobHandle[R]
wrapping in :meth:enqueue / :meth:get, and cron schedule management
via :meth:create_schedule, :meth:list_schedules,
:meth:update_schedule, :meth:delete_schedule.
The backend is injected at construction so the same client can target
either an :class:~taskq.testing.in_memory.InMemoryBackend (tests) or a
:class:taskq.backend.postgres.PostgresBackend (production).
JobsClient ¶
JobsClient(
backend: Backend,
*,
clock: Clock | None = None,
settings: TaskQSettings | None = None,
capacity_cache_ttl: float = DEFAULT_CAPACITY_CACHE_TTL,
)
Public API for job operations.
Delegates to the injected :class:Backend and wraps results in
typed :class:JobHandle[R] instances. The client owns the
payload-serialization step that turns a typed P into the
dict[str, object] carried by :class:EnqueueArgs; the backend
sees only erased payloads.
Source code in src/taskq/client/_jobs.py
backend
property
¶
The underlying :class:Backend this client delegates to.
Exposed so :class:JobHandle can read the backend through the
client without accessing the private _backend attribute.
close
async
¶
invalidate_actor_capacity_cache ¶
Drop the cached actor_config.max_pending snapshot.
The next enqueue refreshes from the backend instead of waiting
out the TTL. Not needed in normal operation (staleness is
bounded by capacity_cache_ttl, default 5s); intended for
tests and for tooling that knows it just changed the table and
cannot wait out the TTL.
Source code in src/taskq/client/_jobs.py
enqueue
async
¶
enqueue(
ref: ActorRef[P, R],
payload: P,
*,
queue: QueueName | None = None,
scheduled_at: datetime | None = None,
priority: int | None = None,
schedule_to_close: datetime | None = None,
start_to_close: timedelta | None = None,
heartbeat_timeout: timedelta | None = None,
identity_key: IdentityKey | None = None,
fairness_key: str | None = None,
idempotency_key: IdempotencyKey | None = None,
idempotency_scope: str | None = None,
trace_id: str | None = None,
span_id: str | None = None,
metadata: dict[str, object] | None = None,
tags: list[str] | None = None,
) -> JobHandle[R]
Enqueue a job for the given actor and return a typed handle.
The payload is serialized through ref.payload_type so the
EnqueueArgs.payload carried over the backend boundary is a
plain dict[str, object] ready for the JSONB column. The
returned :class:JobHandle[R] carries ref.result_adapter so
:meth:JobHandle.wait can validate the stored result back to
R.
The metadata.singleton key is reserved by the library for
singleton enforcement. When ref.singleton is True the
library unconditionally writes metadata.singleton = True,
overriding any caller-supplied value. Callers MUST NOT set
metadata.singleton manually.
max_pending:
-
When the actor's effective
max_pendingis set, a pre-flight count ofpending+scheduledjobs for the actor is compared to the limit. Ifcount >= max_pending, :class:MaxPendingExceededErroris raised synchronously — the caller decides whether to retry, fail, or wait; the library does not block on capacity. -
The effective limit is operator-owned: a non-NULL stored
actor_config.max_pending(set viataskq actor-config set --max-pending) wins over the@actor(max_pending=...)literal; a cleared or absent stored value falls back to the literal. The client reads the stored value through a TTL-bounded cache (default 5s staleness; see :class:taskq.client._capacity.ActorCapacityCache), so an operator change takes effect fleet-wide within seconds without any redeploy or restart. -
Evaluation order at enqueue:
unique_fordedup → singleton pre-flight →max_pendingcount check →idempotency_keyINSERT → job INSERT. Aunique_forhit bypasses all remaining checks; a singleton collision fires beforemax_pendingto give the caller the more specificSingletonCollisionError. -
idempotency_keydoes not bypassmax_pending— the idempotency ON CONFLICT fires at step 5, after the max_pending check at step 3. Re-enqueuing with a duplicateidempotency_keywhen the queue is full raisesMaxPendingExceededError, not the deduplicated handle. Onlyunique_for(step 1) bypasses max_pending.
idempotency_key:
-
idempotency_keyis unique within itsidempotency_scope(composite(idempotency_scope, idempotency_key)uniqueness). The default scope (idempotency_scope=Noneor"") preserves the prior global-until-prune behavior exactly, so existing callers see zero behavior change. Passing an explicit scope (e.g. a run/batch/epoch id) lets two enqueues with the same business key in different scopes both succeed, decoupling the dedupe horizon fromprune_retention_*. -
Key length is bounded at
idempotency_key_max_bytes(TASKQ_IDEMPOTENCY_KEY_MAX_BYTES, default 1024 UTF-8 bytes) — the bound is the composite unique index's btree entry size, not a round number. Empty and whitespace-only keys raise :class:ValueErrorat the client boundary before any backend call. The same bound applies toidempotency_scope; an empty scope ("") is valid and equivalent toNone(the default/global scope). -
No time-based (TTL) dedupe window.
idempotency_scopedecouples the dedupe horizon fromprune_retention_*by namespace, not by time — there is noidempotency_ttlor equivalent "dedupe for the next N seconds" parameter. A key within a given scope still dedupes until pruned, exactly like the pre-scope global behavior, just scoped to that namespace. This is a deliberate scope decision, not an oversight: a real sliding-window TTL cannot be expressed as a single static unique index the way scope can — every mature job queue that offers one (Oban, River) either gives up the atomicINSERT ... ON CONFLICTfor a check-then-insert lock (weaker concurrency guarantee) or buckets time into the key itself (coarser, edge-artifact-prone semantics). If your use case genuinely needs "dedupe for the next hour, not forever," encode the window into the scope yourself (e.g. a time-bucketed scope string) until/unless a TTL parameter ships as a separate feature. -
Rolling-deploy note: if this schema is mid-upgrade (the
01.00.03_01_pre_idempotency_scope.sqlmigration applied but01.00.03_01_post_idempotency_scope_drop_old_index.sqlnot yet applied), reusing the sameidempotency_keyunder two differentidempotency_scopevalues raises :class:~taskq.exceptions.ScopedIdempotencyMigrationPendingErrorrather than silently dedupe against the wrong scope's job. The trigger is a key existing under a different scope, in either direction — an unscoped call reusing a key first written under a non-default scope raises it too. Only brand-new keys and same-scope repeats are unaffected. See that exception's docstring and the migration file's header comment for the full rationale.
unique_for:
-
unique_fordeduplication is best-effort. Concurrent enqueues for the same(actor, identity_key)may both insert; the dispatch CTE'srunning_identitiesfilter ensures only one runs. -
When either dedup mechanism matches an existing job,
JobHandle.was_existingisTrue. This field replaces the need for callers to inspect the row'screated_atto detect a dedup return.
Source code in src/taskq/client/_jobs.py
227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 | |
enqueue_batch
async
¶
enqueue_batch(
items: list[EnqueueItem],
*,
batch_id: UUID | None = None,
connection: Connection | None = None,
failure_policy: BatchFailurePolicy | None = None,
finalizer: EnqueueItem | None = None,
) -> BatchHandle
Enqueue multiple jobs in a single batched INSERT and return a
:class:~taskq.batch.BatchHandle.
All items share a single batch_id UUID written into each
job's metadata.batch_id field (as a string). When
batch_id is not supplied it is auto-generated as a UUIDv7 via
:func:~taskq._ids.new_job_id.
failure_policy:
When failure_policy is set (e.g.
:class:~taskq.batch_policy.AbortBatchAfter), a batches row
is created with the policy's failure threshold. After each child
job reaches a terminal state, the
:func:~taskq.batch.apply_batch_terminal_outcome hook inspects
the outcome: succeeded resets the consecutive-failure
counter; failed increments it and aborts the batch if the
threshold is reached. Aborting cancels all pending and scheduled
child jobs and sets the batch row to aborted.
finalizer:
When finalizer is set, a finalizer job is enqueued alongside
the batch. The finalizer is NOT stamped with batch_id
metadata (deadlock prevention — if it were, wait_for_batch
would count it as a child and the finalizer would wait for
itself). The batch row's finalizer_job_id column records the
link, and wait_for_batch automatically excludes that job
from counts. The finalizer is dispatched immediately; the
in-actor wait_for_batch snooze pattern gates on child-job
completion.
Transactional enqueue:
When failure_policy or finalizer is set and
connection is None, the entire operation (batch row +
all child jobs + finalizer) is inserted in a single transaction
via :meth:Backend.enqueue_batch_atomic. If any insert fails,
no rows are committed. When a connection is provided, the
caller controls the transaction boundary; the batch row and
finalizer are created as the last statements on that connection.
Validation rules:
len(items) == 0raises :class:ValueError.len(items) > MAX_BATCH_SIZEraises :class:ValueError.- ALL payloads are validated before any INSERT. A single failure
raises :class:
~taskq.exceptions.PayloadValidationErrorand leaves no rows inserted.
max_pending:
One aggregated SELECT actor, count(*) … WHERE actor = ANY($1)
GROUP BY actor is issued for the entire batch. Per-actor
effective limits (operator-owned stored value when set, else the
@actor(...) literal — same resolution as :meth:enqueue)
are checked before the INSERT; any violation raises
:class:~taskq.exceptions.MaxPendingExceededError.
idempotency_key collisions:
Items whose idempotency_key collides with an existing row
return the existing :class:~taskq.client.JobHandle (same
semantics as single-item :meth:enqueue).
Source code in src/taskq/client/_jobs.py
421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 | |
enqueue_batch_streaming
async
¶
enqueue_batch_streaming(
items: Iterable[EnqueueItem],
*,
batch_id: UUID | None = None,
connection: Connection | None = None,
failure_policy: BatchFailurePolicy | None = None,
finalizer: EnqueueItem | None = None,
chunk_size: int = DEFAULT_CHUNK_SIZE,
) -> BatchHandle
Enqueue jobs from a lazy iterable in chunks, returning a single
:class:~taskq.batch.BatchHandle.
Unlike :meth:enqueue_batch, this method accepts an
:class:~collections.abc.Iterable (including generators) and
inserts in chunks of chunk_size (1-MAX_BATCH_SIZE). All items share
the same batch_id. Payloads are validated on the fly as
each chunk is built.
When failure_policy or finalizer is set and
connection is None, the entire operation is delegated to
:meth:Backend.enqueue_batch_atomic for single-transaction
atomicity. Otherwise, chunks are inserted via
:meth:Backend.enqueue_batch on the caller-owned connection,
and the batch row + finalizer are created as the last
statements.
Failure-policy counting limitation (caller-connection path):
the batch row is created AFTER all chunk inserts — it must carry
the final expected_size and create_batch is INSERT, not
upsert — so a child job enqueued on that connection that
reaches a terminal state BEFORE the row exists is not counted
toward failure_policy: increment_batch_failures finds
no row and returns (0, None, 0). The atomic
(no-connection) path is unaffected — its single transaction
makes the batch row and the child jobs visible together.
max_pending: NOT enforced on this path — unlike
:meth:enqueue_batch, which runs one aggregated per-actor
check before the INSERT, neither the chunked
:meth:Backend.enqueue_batch inserts nor the atomic delegation
consult max_pending. The caller is responsible for ensuring
the stream will not exceed actor limits (the same bulk-import
semantics :meth:enqueue_batch_fast discloses).
Source code in src/taskq/client/_jobs.py
675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 | |
get_batch
async
¶
Fetch a single batch row by ID.
Delegates to :meth:Backend.get_batch. Returns None when the
batch does not exist.
Source code in src/taskq/client/_jobs.py
list_batches
async
¶
List batches matching filter, returning :class:BatchSummary objects.
Delegates to :meth:Backend.list_batches and maps each
(BatchRow, BatchCounts) pair to a :class:BatchSummary
with a :class:BatchCompletionStatus derived from the live counts.
Source code in src/taskq/client/_jobs.py
enqueue_batch_fast
async
¶
enqueue_batch_fast(
items: list[EnqueueItem],
*,
batch_id: UUID | None = None,
connection: Connection | None = None,
) -> int
Enqueue jobs via COPY FROM protocol for maximum throughput.
WARNING — bulk-import semantics, not general-purpose enqueue:
this method does NOT enforce max_pending, does NOT detect or
reject idempotency-key collisions (a duplicate key aborts the
whole batch instead of being treated as "already enqueued"), and
returns a bare row count, not per-job handles — there is no
way to await, cancel, or otherwise reference an individual job
from the return value. Use :meth:enqueue_batch unless you
specifically need COPY-level throughput for a one-shot bulk
import/backfill and have already accounted for these gaps.
Returns the count of inserted rows — no :class:~taskq.batch.BatchHandle,
no per-job :class:~taskq.client.JobHandle instances.
Validation rules:
len(items) == 0raises :class:ValueError.len(items) > 50_000raises :class:ValueError.- ALL payloads are validated before any INSERT — a single failure
raises :class:
~taskq.exceptions.PayloadValidationError.
Tradeoffs vs enqueue_batch:
- No idempotency-key collision handling. A duplicate key
aborts the entire batch with
asyncpg.UniqueViolationError. Callers must pre-deduplicate. One carve-out: during the01.00.03pre→post migration window, a key reused across different scopes raises :class:~taskq.exceptions.ScopedIdempotencyMigrationPendingErrorinstead, matching the other enqueue paths. - No max_pending check. The caller is responsible for ensuring the batch won't exceed actor limits.
- No JobHandle instances. Only the inserted row count is
returned. Use
batch_idto query rows post-insert. - All-or-nothing atomicity. No partial success — the entire COPY fails on any constraint violation.
Use for bulk import / backfill with 1K-50K rows where throughput matters more than idempotency guarantees.
Source code in src/taskq/client/_jobs.py
1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 | |
get
async
¶
Look up a job by id.
Returns None when the job does not exist; otherwise wraps
the row in a :class:JobHandle[R]. The caller may supply
result_adapter because lookups by id do not carry actor
identity — typical sources are
my_actor.result_adapter (when reuniting with an actor) or
TypeAdapter(type(None)) (when only row metadata is needed).
When result_adapter is None it defaults to
TypeAdapter(type(None)), which is suitable for status-only
lookups.
Source code in src/taskq/client/_jobs.py
get_row
async
¶
Look up a job by id and return the raw :class:JobRow.
Mirrors :meth:get's contract — one backend.get, None
when the job does not exist — without the handle machinery or
result adapter. For callers that never need a
:class:JobHandle, this is the direct form; for the fresh-read
case that does want a handle, prefer get plus the handle's
row property (still a single round trip).
Source code in src/taskq/client/_jobs.py
list
async
¶
List jobs matching filter, returning a :class:JobPage.
filter.status accepts a single :data:JobStatus or a
sequence of statuses (e.g. JobFilter(status=["pending",
"running"])).
filter.active is a meta-filter — not Celery's 'active':
active=True selects non-terminal statuses (pending,
scheduled, running — 'not yet finished', not 'currently
executing') and active=False selects terminal ones. See
:class:JobFilter for full semantics.
next_cursor is returned for every ordering, encoded from the
columns that ordering actually sorts by, and is only None on
the last page.
Source code in src/taskq/client/_jobs.py
cancel
async
¶
Request cancellation of a job and return a :class:CancelResult.
Reads the row first via :meth:Backend.get. If the job does not
exist, raises :class:KeyError — matching Python's stdlib
idiom for "asked for an entry by id; it isn't there".
Then calls :meth:Backend.write_cancel_request and reads the
row again to capture the new status. The previous_status
reflects the row at the first read, not atomically at
write-time (TOCTOU per ).
: increments taskq.cancellation.requested exactly once
per call, regardless of cancellation_initiated outcome.
Source code in src/taskq/client/_jobs.py
cancel_where
async
¶
cancel_where(
filter: JobFilter,
reason: str | None = None,
*,
allow_empty_filter: bool = False,
) -> BulkCancelResult
Cancel all jobs matching filter in a single set-based operation.
Pending/scheduled jobs are moved straight to terminal 'cancelled'
(no running actor to cooperate with). Running jobs get
cancel_phase=1 set (cooperative cancel) — the worker's
heartbeat-driven cancel controller observes the phase change and
sets the in-process cancel_event.
Guardrail: a filter with no predicates (no queue, status,
actor, identity_key, batch_id, tags, or active) is rejected with
:class:EmptyFilterError unless allow_empty_filter=True is
passed.
Filter fields used: queue, status, actor,
identity_key, batch_id, tags, active. The
limit, cursor, and order_by fields are ignored.
Returns a :class:BulkCancelResult with counts and affected IDs.
Source code in src/taskq/client/_jobs.py
create_schedule
async
¶
create_schedule(
actor: str | ActorRef[P, R],
cron_expr: str,
*,
timezone: str = "UTC",
dst_strategy: DstStrategy = "skip",
payload_factory: str | None = None,
static_payload: dict[str, object] | None = None,
name: str = "",
identity_key: IdentityKey | None = None,
enabled: bool = True,
) -> ScheduleHandle
Create a cron schedule. Raises :class:ValueError if both
payload_factory and static_payload are provided, or if
cron_expr is invalid.
The (actor, name) UNIQUE constraint means each (actor, name)
pair may have at most one schedule; a second create_schedule for
the same pair raises asyncpg.UniqueViolationError (PG) or
:class:ValueError (in-memory). Pass distinct name values to run
several cron schedules per actor (e.g. a per-property sync).
When identity_key is set, the cron loop propagates it to cron-fired jobs so they dedup against on-demand jobs for the same business key.
Does NOT validate actor existence at creation time — any string actor name is accepted (validation is deferred to fire time).
The first next_fire_at is seeded from the clock that
arbitrates its due-check: on a Postgres-backed client the PG
server clock is read first (one-row SELECT clock_timestamp()
via the backend's pool, mirroring the worker bootstrap), so
app↔DB clock skew cannot shift the fire chain; on a pool-less
(in-memory) client the seed comes from the client's injected
Clock. This matters permanently: the cron loop's normal path
recomputes every subsequent fire from the STORED fire time
(only a miss beyond cron_catch_up_window re-anchors on the
server clock), so the seed — not any per-tick correction —
fixes the chain's phase for the schedule's life.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dst_strategy
|
DstStrategy
|
How to handle DST gaps and overlaps.
|
'skip'
|
Source code in src/taskq/client/_jobs.py
1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 | |
list_schedules
async
¶
List cron schedules, optionally filtered by actor or enabled status.
Source code in src/taskq/client/_jobs.py
update_schedule
async
¶
update_schedule(
schedule_id: UUID,
*,
cron_expr: str | None = None,
enabled: bool | None = None,
payload_factory: str | None = None,
static_payload: dict[str, object] | None = None,
clear_payload_factory: bool = False,
) -> ScheduleRecord
Update a cron schedule. Setting enabled=True clears
last_fire_error and resets consecutive_failures to 0.
Raises :class:ValueError if both payload_factory and
static_payload are provided, or if cron_expr is invalid.
To explicitly clear payload_factory (set the column to NULL),
pass clear_payload_factory=True — None for payload_factory
means "don't change this field."
When cron_expr changes, the recomputed next_fire_at is
seeded from the same clock as create_schedule (the PG
server clock on Postgres-backed clients; the client's injected
Clock in-memory) — the stored chain keeps its server-anchored
phase.
Source code in src/taskq/client/_jobs.py
delete_schedule
async
¶
_handle ¶
Generic :class:JobHandle — typed handle to an enqueued job.
Carries a :class:pydantic.TypeAdapter for the actor's return type
R, which is the mechanism that prevents R from being a phantom
type parameter. The single blocking accessor :meth:wait returns R
(never R | None); it raises on missing / failed / timeout.
The handle reads the backend through self._backend for
:meth:wait; read-back operations (:meth:status, :meth:refresh,
:meth:attempts, :meth:cancel) require a :class:JobsClient and
raise :class:RuntimeError when the handle was constructed without
one.
JobHandle ¶
JobHandle(
*,
row: JobRow,
result_adapter: TypeAdapter[R],
was_existing: bool,
client: JobsClient | None = None,
backend: Backend | None = None,
_redis_client: Redis | None = None,
_settings: TaskQSettings | None = None,
)
Typed handle to a single enqueued job.
Created by :class:JobsClient methods (:meth:~JobsClient.enqueue,
:meth:~JobsClient.get) or by :class:SubJobEnqueuer (with
backend= only). The type parameter R flows from the
actor's declared return type through :class:ActorRef into this
handle: JobHandle[OrderResult] for an actor returning
OrderResult, JobHandle[None] for fire-and-forget actors.
At least one of client or backend must be supplied. When
client is provided, _backend is filled from
client.backend. When only backend is provided, the four
read-back methods (:meth:status, :meth:refresh,
:meth:attempts, :meth:cancel) raise :class:RuntimeError
because they require the client's higher-level coordination.
:meth:wait always works (it reads through _backend directly).
Why result_adapter: TypeAdapter[R] is a constructor arg: pyright
only infers R for a generic class when the type parameter
appears in at least one field or method signature. The adapter is
that field — without it R would be phantom and inference would
silently fall back to Unknown.
Source code in src/taskq/client/_handle.py
row
property
¶
The last :class:JobRow this handle observed.
Seeded at construction with the row the creating call fetched,
and advanced by every successful row fetch through the handle —
:meth:refresh, :meth:status, and :meth:wait's polling loop
each record the row they just read. :meth:progress_stream
does not advance it — its Redis path fetches no rows, and
advancing only on the PG fallback would make the semantics
backend-dependent. Reading the property costs no backend round
trip: handle = await tq.get(id) followed by handle.row
is the single-read pattern for full row state, and a
long-lived handle's row stays current as its owner
refreshes or waits.
The row is returned by reference (it is frozen and backends hand the handle an isolated row); repeated reads between fetches return the same object.
status
async
¶
Return the current status of this job (live read).
Cheap, non-blocking: a single backend.get and a status
projection. No polling. Use this when you want to know the
state without waiting for a terminal transition. Advances
:attr:row to the fetched row.
Raises:
| Type | Description |
|---|---|
RuntimeError
|
this handle was constructed without a
:class: |
Source code in src/taskq/client/_handle.py
refresh
async
¶
Re-read the row from the backend and return the raw
:class:JobRow.
Useful for callers that want full row state (timestamps,
attempt counts, error metadata) without going through
:meth:wait. Does not block on terminal state — returns the
current row whatever its status. Advances :attr:row to the
fetched row, so after a refresh handle.row and the return
value are the same row.
Raises:
| Type | Description |
|---|---|
RuntimeError
|
this handle was constructed without a
:class: |
Source code in src/taskq/client/_handle.py
attempts
async
¶
Return the attempt rows for this job, ordered by attempt number.
Raises:
| Type | Description |
|---|---|
RuntimeError
|
this handle was constructed without a
:class: |
Source code in src/taskq/client/_handle.py
cancel
async
¶
Delegate to :meth:JobsClient.cancel.
Raises:
| Type | Description |
|---|---|
RuntimeError
|
this handle was constructed without a
:class: |
Source code in src/taskq/client/_handle.py
wait
async
¶
Block until the job reaches a terminal status, then return R.
Returns the actor's return value, validated through
:attr:result_adapter. The result type is R exactly —
never R | None. Missing or failed results raise. Advances
:attr:row to each row the polling loop fetches — on return,
the terminal row the result was extracted from.
Raises:
| Type | Description |
|---|---|
ResultUnavailable
|
terminal state reached but no result was
stored (result TTL expired, actor returned |
JobFailed
|
the job ended in a non-success terminal state
( |
TimeoutError
|
|
Source code in src/taskq/client/_handle.py
progress_stream
async
¶
Stream live progress events for this job.
When Redis is configured, subscribes to the per-job Redis pub/sub
channel and yields :class:~taskq.progress.ProgressEvent objects in
real time. When Redis is not available, falls back to polling Postgres
at 500 ms intervals and synthesising events from row diffs.
Raises :class:NotImplementedError when the in-memory backend is
detected — the in-memory backend does not support pub/sub.
Does not advance :attr:row — the Redis path fetches no rows,
and advancing only on the PG fallback would make the semantics
backend-dependent.
Yields events until a terminal=True event is produced.
Source code in src/taskq/client/_handle.py
ActorsClient ¶
Pool-wrapping facade for actor configuration operations.
Acquires a connection from the injected pool for each call, delegates
to taskq.actor_config_ops, and returns the result. The
caller must have opened the pool; this class does not manage its
lifecycle.
.. note::
This client is Postgres-only — it delegates to
:mod:taskq.actor_config_ops, which executes raw SQL against
the actor_config table. The :class:~taskq.backend._protocol.Backend
protocol does not include actor config operations, so
:class:~taskq.testing.InMemoryBackend does not support
ActorsClient.
Parameters¶
pool:
An open asyncpg.Pool. The caller retains ownership.
schema:
TaskQ schema name. Defaults to "taskq".
Source code in src/taskq/client/_actors.py
list
async
¶
List all stored actor_config rows, ordered by actor name.
get
async
¶
Get one actor_config row, or None if not found.
set_capacity
async
¶
set_capacity(
actor: str,
*,
max_concurrent: int | Unset | None = UNSET,
max_pending: int | Unset | None = UNSET,
result_ttl: float | Unset | None = UNSET,
) -> ActorConfigRow | None
Update capacity fields on an existing actor_config row.
Source code in src/taskq/client/_actors.py
deregister
async
¶
Deregister an actor with safety checks.
See :func:taskq.actor_config_ops.deregister_actor for
the full semantics.