Upgrading

TaskQ's schema migrations are forward-only by design. There is no down migration mechanism and none is planned, so this section explains why, and what to do if you need to undo a change.

---

Forward-only migration policy

The migration runner (taskq.migrate) discovers *.sql files bundled under taskq.migrations in lexicographic order ({ver}_{nn}_{pre|post}_{description}.sql), applies any not already recorded in {schema}.schema_migrations, and records a SHA-256 checksum of the rendered SQL after each successful apply.

There is no down operation. To revert a migration, restore the database from a backup taken before it was applied.

This is a deliberate tradeoff, not a missing feature:

• Down migrations are rarely exercised in practice and rot quietly until the one time they're needed, at which point they often don't work.
• A schema rollback that isn't paired with a data rollback (e.g., a dropped column that already lost data) is not actually safe to run automatically.
• Point-in-time recovery / backup restore is the operation you actually want for "undo a bad deploy" in a durable job queue, since job state itself needs to roll back together with the schema.

Before upgrading TaskQ

1. Take a backup. Since there is no automated rollback, a recent backup (or PITR window) is your only revert path.
2. Check the Changelog for the target version: TaskQ is pre-1.0 (see the Stability note in the README), so breaking changes, including schema changes, may land in minor version bumps (0.x.0), not only majors.
3. Review pending migrations before applying them:

taskq migrate status

This lists every discovered migration and whether it has already been applied, without changing anything.

1. Apply migrations explicitly, from a pre-deploy job or init container, before any worker starts:

taskq migrate up

Workers never self-migrate. TASKQ_MIGRATE_ON_START=true is honoured only by taskq ui serve; a worker warns and ignores it, and refuses to boot while a pre-phase migration is pending, so relying on it there produces a crash-loop rather than a migrated schema. N replicas racing to migrate is the concurrent-migration hazard migrations exist to avoid.

The command is idempotent: migrations already recorded in {schema}.schema_migrations are skipped. See cli.md for the full option reference (--phase, --target, --max-steps, --ddl-lock-timeout).

TASKQ_MIGRATE_ON_START=true is not a substitute here: it is honoured only by taskq ui serve, which runs as a single process. The worker ignores it (and warns when it is set): N worker replicas racing to migrate is the hazard the migration advisory lock exists to prevent, and a worker started against a schema still missing a pre-phase migration refuses to boot rather than running against a schema behind its code.

Non-transactional migrations

By default every migration file runs inside its own transaction, so a failure rolls the whole file back. PostgreSQL forbids some statements inside a transaction block, notably CREATE INDEX CONCURRENTLY and DROP INDEX CONCURRENTLY, the only forms that build or drop an index without blocking writes on the table. On hot tables (jobs, job_events) a plain CREATE INDEX takes a SHARE lock that blocks INSERT/UPDATE/DELETE for the duration of a full-table scan and stalls the worker fleet, so index migrations on those tables should use the concurrent forms.

A migration opts out of the transaction wrapper with a header directive in its leading comment block (-- line comments only, before the first SQL token):

-- taskq:no-transaction
-- NOT redundant with IF NOT EXISTS below: an interrupted CREATE INDEX
-- CONCURRENTLY leaves an INVALID index that IF NOT EXISTS alone would
-- silently skip rebuilding, so drop the debris first.
DROP INDEX CONCURRENTLY IF EXISTS "{schema}".jobs_queue_idx;
CREATE INDEX CONCURRENTLY IF NOT EXISTS jobs_queue_idx ON "{schema}".jobs (queue);

The runner then executes the file statement by statement, each in its own implicit transaction (the same semantics as Alembic's autocommit_block or Rails' disable_ddl_transaction!). This changes the failure contract, so three rules apply:

• The migration must be idempotent and re-runnable. Nothing rolls back: if the third statement fails, the first two stay applied. The ledger records the migration only after every statement succeeds, so the next migrate up re-executes the whole file: every statement must tolerate being re-run (IF NOT EXISTS, guarded inserts, etc.).
• An interrupted CREATE INDEX CONCURRENTLY leaves an INVALID index behind. The standard remedy is drop-and-rebuild, written into the migration itself as shown above: the DROP INDEX CONCURRENTLY IF EXISTS line removes debris from an interrupted attempt before rebuilding. A plain CREATE INDEX CONCURRENTLY IF NOT EXISTS alone would silently skip the rebuild while the invalid index keeps its name. You never have to find these by hand: when a run fails, taskq migrate up lists any INVALID indexes in its failure report.
• No transaction-control statements. BEGIN/COMMIT/ROLLBACK (and aliases) are rejected before anything executes: they would silently re-open a transaction, defeating the directive. The statement splitter assumes the server default standard_conforming_strings=on.

Operators can see the distinction two ways: taskq migrate status annotates non-transactional migrations with (no transaction), and the {schema}.schema_migrations ledger records how each migration ran in its use_transaction column (false = ran outside a transaction). The runner adds that column when recording the next migration, so deployments upgraded from older TaskQ versions need no manual step; rows applied before the column existed read true.

If a migration goes wrong

taskq migrate up diagnoses its own failures: it tells you which migration failed, what state the schema is in, and the one action to take. You never need to inspect catalog state by hand.

Transactional migration (the default)

The whole file rolled back automatically, so the schema is exactly as it was before the attempt. Fix the cause of the error, then re-run taskq migrate up.

The migration gave up waiting for a table lock

A transactional migration's DDL (ALTER TABLE jobs, a plain CREATE INDEX) needs a lock that any session holding even a read on the table blocks; an actor's open transaction connection mid-job, an admin snapshot, pg_dump. Postgres queues lock requests first-come-first-served, so once the DDL is queued every later statement on that table (dispatch, enqueue, heartbeat) queues behind it; left unbounded, the wait outlives the workers' heartbeat budget and the fleet self-terminates while the migration is still waiting. Each transactional migration therefore waits at most DEFAULT_MIGRATION_DDL_LOCK_TIMEOUT (30 s, SET LOCAL lock_timeout inside its own transaction) and then fails with MigrationLockTimeoutError; the report names the migration, the bound, and this remedy. The migration rolled back and nothing was applied: find the holder in pg_stat_activity / pg_locks, end it or wait for it, then re-run taskq migrate up. To wait longer, pass taskq migrate up --ddl-lock-timeout SECS, or ddl_lock_timeout= to apply_pending / apply_pending_locked from your own deploy tooling (0 waits indefinitely, at the cost of parking every statement on the table behind the queued DDL). The bound governs the wait only: a statement that already holds its lock, such as an index build, is never interrupted by it. The runner's own upgrade of the schema_migrations ledger (an ALTER TABLE before the first migration of a run) waits under the same bound and reports the same error, naming the ledger instead of a migration. -- taskq:no-transaction migrations are not bounded: their CONCURRENTLY phases wait on heavyweight locks by design.

Non-transactional migration (-- taskq:no-transaction)

Nothing rolls back: statements before the failure remain applied, and the migration is not recorded. Re-run taskq migrate up; the migration is idempotent, and the command's failure report lists any INVALID indexes the interrupted attempt left behind; the drop-and-rebuild already written into the migration cleans them up on the re-run. Only pin taskq-py back to the previous version if the migration SQL itself is wrong and you need time to ship a correction.

A migration applied successfully but broke older workers

Restoring from backup is for this scenario: the migration itself succeeded, but not-yet-upgraded workers cannot run against the new schema. Stop the workers pointed at the affected schema, restore the pre-migration backup, and pin taskq-py back until every worker is upgraded.

• Stop workers pointed at the affected schema to avoid further writes.
• Restore the database from the pre-migration backup.
• Pin taskq-py back to the previous version until the issue is resolved, since the previous version's code may not be compatible with the new schema.

---

Breaking import path changes

taskq.worker.actor_config → taskq.actor_config

Released in v0.2.0-v0.2.2, moved in unreleased. This is a breaking change for anyone importing ActorConfig from the old path.

The ActorConfig dataclass has moved from taskq.worker.actor_config to the top-level taskq.actor_config module. It is a shared carrier used by the client, CLI, and admin UI, not worker-internal.

Old (v0.2.0-v0.2.2):

from taskq.worker.actor_config import ActorConfig

New:

from taskq.actor_config import ActorConfig

The old import path raises ImportError; update your imports.

taskq.worker.actor_config_ops → taskq.actor_config_ops

The actor_config_ops module (listing, inspecting, tuning, and deregistering actors on a live deployment) has moved from taskq.worker.actor_config_ops to the top-level taskq.actor_config_ops. This module was introduced on the unreleased branch; if you were importing it from the worker.* path during development, update to the top-level path.

Old (unreleased branch only):

from taskq.worker.actor_config_ops import (
    list_actor_configs,
    get_actor_config,
    set_actor_config_capacity,
    deregister_actor,
)

New:

from taskq.actor_config_ops import (
    list_actor_configs,
    get_actor_config,
    set_actor_config_capacity,
    deregister_actor,
)

State-machine constants: taskq.backend.statemachine → taskq.backend

Unreleased. Not a breaking change: the old path still imports. This is guidance for embedders reviewing their imports against the public API surface.

TERMINAL_STATUSES (and ACTIVE_STATUSES, VALID_TRANSITIONS, assert_valid_transition) are re-exported from the public taskq.backend package. The defining module, taskq.backend.statemachine, is an internal layout detail: taskq.backend is the covered surface: tests/test_backend_init_coverage.py pins that every taskq.backend.__all__ entry resolves and that each re-export is identical to its submodule source, so that is the path that stays stable.

Old (internal defining module; works today, not the covered surface):

from taskq.backend.statemachine import TERMINAL_STATUSES

New:

from taskq.backend import TERMINAL_STATUSES

The same one-line switch applies to ACTIVE_STATUSES, VALID_TRANSITIONS, and assert_valid_transition.

TERMINAL_STATUSES and the JobStatus literal alias are additionally re-exported from the top-level taskq package, the shortest spelling for "is this job done" checks and JobStatus annotations:

from taskq import TERMINAL_STATUSES, JobStatus

Both paths are the same objects: tests/test_public_api_exports.py pins that the top-level TERMINAL_STATUSES is (identical to, not merely equal to) the taskq.backend one, so the two import paths can never diverge.

---

Breaking API changes

Queue and actor names are bounded at 255 characters

Unreleased. Breaking for anyone enqueueing to a queue name longer than 255 characters or declaring an actor name longer than 255 characters: on previous releases both were accepted and worked; now each raises ValueError at its chokepoint (the queue at enqueue, the actor at @actor(...) registration).

jobs.queue and jobs.actor are carried in btree indexes (the (queue, priority DESC, scheduled_at) dispatch order and the composite dispatch probes), and a btree index item caps at 2704 bytes: an incompressible name past that bound used to fail the jobs INSERT with an opaque ProgramLimitExceededError ("index row size ... exceeds btree version 4 maximum") at enqueue, far from the declaration that caused it. Both names are now refused with a plain ValueError naming the bound and echoing at most the first 64 characters of the rejected name.

The bound is deliberately a character count, stricter than the ~2700 bytes the storage mechanism alone forces, and that is the point: the composite dispatch index item carries the actor name and the queue name side by side, so a per-name bound has to compose with a second variable-length name in the same item. 255 characters is at most 1020 bytes at UTF-8's worst-case 4 bytes/char, and two of those plus the fixed-width tail columns fit inside one index item. A byte-measured single-name bound (~2500 bytes) would admit a queue name that only fails when paired with a legal actor name, resurfacing the same opaque storage error the cap exists to remove.

To upgrade: if you enqueue to queue names over 255 characters or declare actors with names over 255 characters, shorten them before deploying this release. A name past the bound now raises ValueError at the call (enqueue or registration) instead of failing deep in the storage engine.

/logout is now a POST with a session-bound CSRF token

Unreleased. Breaking for anything that ended an SSO session by fetching GET /admin/logout.

Both SSO backends accepted GET /logout, so a forced top-level navigation to it, a link, an image, a redirect from any page, cleared the admin session. Logout is now POST-only and requires a CSRF token derived from the live session cookie; GET /logout answers 405 and a POST without the token answers 403. The admin UI's Sign out control posts the token. Switch scripts that logged out via GET /admin/logout to POST (see sso.md).

The same change scopes the OIDC state cookie to the callback route (matching the SAML correlation cookie) and caches the OIDC discovery document and JWKS per issuer with a 300 s TTL, so an unauthenticated /login flood no longer translates into one outbound IdP fetch per request.

validate_actor_payload: actor_name= → actor=

Unreleased. Breaking for anyone calling taskq.validate_actor_payload with the third argument passed by keyword. Passing it positionally is unaffected.

The public taskq.validate_actor_payload export previously resolved to a duplicate implementation in taskq.exceptions that embedded the raw payload in its exception message and attached pydantic errors with include_input. That message is persisted to the job row's error_message and rendered in the web admin, so attacker-controlled payload values could leak into both. The export now resolves to the single sanitized implementation in taskq._validation, whose third parameter is named actor.

Old:

from taskq import validate_actor_payload

model = validate_actor_payload(MyPayload, raw_payload, actor_name="send_email")

New:

from taskq import validate_actor_payload

model = validate_actor_payload(MyPayload, raw_payload, actor="send_email")

The positional form needs no change and works on both versions:

model = validate_actor_payload(MyPayload, raw_payload, "send_email")

Three further changes to be aware of, none of which fail loudly:

• actor is now optional (defaults to None), so omitting it no longer raises TypeError.
• The exception message is shorter and no longer contains payload values. It is now Payload validation failed for actor '<name>': <model title>: the pydantic detail and the Raw payload: {...} dump are gone. Anything that parses error_message (log pipelines, alert rules, admin tooling) must be updated.
• PayloadValidationError.validation_errors entries no longer carry input or url keys, because the errors are collected with include_url=False, include_input=False. Code reading err["input"] will now KeyError.

raw_payload also now accepts an existing BaseModel in addition to a dict, a widening, so no action is required.

consume_one_job: payload validation moved before rate-limit acquisition

Unreleased. No action is required for workers. This affects only code that calls taskq.worker._consumer.consume_one_job directly with validated_payload=None.

On the normal dispatch path nothing changes: dispatch_one_job already validates the payload and passes validated_payload, so consume_one_job never runs its own fallback validation there.

For a direct caller, the fallback validate_actor_payload call now runs before the rate-limit acquire block, and acquire_for_actor receives the validated model rather than the raw row dict. Two consequences:

1. PayloadValidationError now escapes before any token is consumed. Previously the consumer acquired first, so an invalid payload burned a non-refunded rate-limit token (release_for_actor sets refund_on_release=False) for an actor body that could never run. If your caller was relying on acquire/release being invoked on the invalid-payload path, it no longer is.
2. Your caller owns the terminal write for the escaping error. This was already true: the outer try has no except clauses, so the error propagated from the in-try fallback as well, but it now propagates earlier. dispatch_one_job's outer handler and the in-memory test runner both already do this.

This also fixes a defect you may have been working around: the registry previously re-validated the raw dict against a keyed ref's own payload_type, discarding the actor model's defaults. An actor model defaulting tenant_id="unattributed" against a ref model requiring tenant_id failed the job non-retryably on a payload that was valid for the actor. If you added a redundant default to a ref model to work around this, you can now remove it.

Keyed refs: payload_type is required, key_fn receives the model

Unreleased. Breaking for every KeyedRateLimitRef / KeyedReservationRef declaration.

Old (broken):

KeyedRateLimitRef(
    base_name="api-per-tenant", key_fn=lambda p: p["tenant_id"], capacity=10, refill_per_second=1.0
)

New:

KeyedRateLimitRef.typed(
    MyPayload,
    base_name="api-per-tenant",
    key_fn=lambda p: p.tenant_id,
    capacity=10,
    refill_per_second=1.0,
)

payload_type is now required, and key_fn receives the validated Pydantic model, not the raw dict. Use .typed() for compile-time type checking of key_fn against the payload model.

Deploy hazard: if defaults or validators change a key-deriving field's value vs the raw row, new concrete names materialize fresh full-capacity buckets alongside old ones (a temporary over-admission window). Drain affected queues before deploying payload model changes that affect key derivation.

Dispatch-path malformed payloads fail immediately

Unreleased. Breaking for callers that relied on invalid payloads being retried.

Dispatch-path malformed payloads now fail immediately as PayloadValidationError (non-retryable) instead of being retried as a generic ValidationError. In-flight legacy rows with invalid payloads will fail on first dispatch instead of exhausting the retry budget.

wait_for_batch defaults to on_empty="error"

Unreleased. Breaking for callers that relied on the silent empty return.

Previously, calling wait_for_batch on a batch_id with zero jobs and no batches row returned an empty BatchCompletionStatus silently. The default is now on_empty="error", which raises EmptyBatchError. Pass on_empty="ok" to preserve the old silent-return behaviour.

Sub-enqueue failure events carry error_class/error_message

Unreleased. Breaking for log pipelines.

sub_enqueue_re_enqueue_error and sub_enqueue_flush_error now carry error_class + error_message instead of the single message field, matching the error_class/error_message convention used by every other error event (job_timeout, job_exception, job_failed, rate_limit_release_failed, savepoint_rollback_failed, stranded_jobs_query_failed, and the failed_details payload of sub_enqueue_flush_failed). Log pipelines querying fields.message on these two events must switch to error_message.

dotenvmodel 1.x: environment-variable precedence flips

Unreleased. Breaking for deployments that rely on .env files beating the process environment.

dotenvmodel is bumped to 1.x (>=1.1.0,<2), adopting its 1.0 defaults. Environment-variable precedence flips: the process environment now beats .env files by default (previously .env values overwrote os.environ); restore the files-beat-env-vars behaviour with DOTENV_OVERRIDE=true or TaskQSettings.load(override=True). load() no longer mutates os.environ: read TASKQ_* values from the settings instance, not the process environment, after a load. TaskQSettings.load() now forwards dotenvmodel's full parameter surface (env, override, env_dir, read_dotfiles, read_environ, load_local). Subclass string-field defaults containing ${VAR} references are interpolated at load time (unset references resolve to "").

Time is unified on the database clock

Unreleased. Breaking for Backend implementors and for callers passing absolute schedule_to_close datetimes.

EnqueueArgs.scheduled_at is now nullable: "immediate" enqueue passes None and the server stamps it (no more client-side now() default), and Backend implementations that require a non-None datetime fail loudly. The raw schedule_to_close datetime form is deprecated in favour of schedule_to_close_interval (or declaring retry.time_budget on the actor, because absolute datetimes cross clock domains and can misbehave under skew); every enqueue arm writes the deadline from one domain (server clock + interval). The rate-limit Redis Lua scripts derive now from redis.call('TIME'); the caller-supplied now ARGV is removed.

JobFilter.active renamed to unfinished

Unreleased. Not a breaking change: the old name still constructs. Positional callers are unaffected: unfinished was appended after created_before rather than slotted into active's old position, so an 11-argument positional call binds its arguments exactly as before the rename.

JobFilter.active read as 'currently executing' but the predicate is terminality: True selects every non-terminal status (pending, scheduled, running), False the terminal ones. The field is now unfinished, which states the predicate it applies. The old name stays accepted as a deprecated constructor kwarg: it raises a DeprecationWarning once, at construction, and feeds the same predicate. The warning fires once per filter even across the dataclasses.replace copies the client's list probe and the bulk-cancel sanitizer make. Update call sites to unfinished=; JobFilter(active=True) becomes JobFilter(unfinished=True).

Terminal writes raise on drifted outcome branches

Unreleased. Not a breaking change: a drifted row previously fell through the SQL arm silently, and nothing could observe that path without a schema or template drift the library itself introduces. The raise is the loud surface for a library bug, not a state your application code can produce or handle.

mark_snoozed, mark_failed_or_retry and mark_abandoned (and the mark_retry arms behind them) branch on a typed outcome instead of free-form SQL arms, so a terminal write whose row matches no branch — a status the statements cannot emit — raises (AssertionError naming the branch) instead of falling through an unhandled SQL arm and returning a row that was never written. Nothing migrates: correct deployments never see it, and the in-memory twins share the same typed branches so the differential corpus holds. Related, and visible on the happy path: the snoozed deadline exit's state-change event persists denial_reason, so a denial-keyed row's terminal event names the denial class the caller reported and an operator can see which starvation killed the job without joining the counter columns.

firstof/allof DST strategies become live

Unreleased. Breaking for schedules that already exist and declare them.

The cron tick's SELECT never listed the dst_strategy column and fire_schedule read it as row.get("dst_strategy", "skip"), so the stored value was always skip in production whatever the schedule said; branch that enqueues the second job for a DST fall-back overlap was unreachable. The column is now selected and read, so a schedule configured allof or firstof years ago changes behaviour on upgrade, with no configuration change and nothing raised: on the autumn fall-back night an allof schedule enqueues two jobs for the repeated local hour where it used to enqueue one. Audit for non-skip schedules before rolling out, and make sure their actors are idempotent.

worker_id is no longer a metric dimension

Unreleased. Breaking for dashboards, queries and alert rules that group or filter by worker_id.

taskq.lock.expires_in_seconds, taskq.heartbeat.misses, taskq.leader.election_attempts, taskq.leader.election_failures, taskq.cron.lock_contention and the taskq.heartbeat.consecutive_failures gauge now emit a single undimensioned series each: they return one series where they used to return one per worker. worker_id is a fresh UUID per worker process, so every deploy, restart and autoscale event minted new time series without bound; Azure Monitor counts each unique (metric, dimension key, dimension value) seen in 12 hours as an active series, caps a subscription at 50,000 per region, and throttles ingestion for every custom metric once the cap is passed, with no backfill of what was dropped. Per-worker attribution is unchanged on the channels where cardinality is free: worker_id is bound onto every log line via contextvars, and taskq.worker_id is a cron-fire span attribute. The record_* helpers still accept their worker_id argument; only the dimension is gone. taskq.cron.consecutive_failures is labeled by actor, not schedule_id: schedule rows accept any actor string at creation time, so the label is capped at the emitter like queue. A dashboard grouped by schedule_id loses its series; group by actor and read per-schedule attribution from the cron-fire span and the log lines, where cardinality is free. cron_auto_disable_threshold is still evaluated per schedule in the database.

taskq._json.dumps() requires str dict keys

Unreleased. Breaking for raw (unvalidated) dicts with non-str keys.

taskq._json.dumps() no longer passes OPT_NON_STR_KEYS to orjson: dict keys that are not str (e.g. int keys in job metadata, actor results, or progress data) now raise TypeError instead of being silently coerced to string keys. Callers must stringify keys before enqueue.

Pydantic-validated payloads are unaffected: dict[str, ...] model fields reject non-str keys at validation (they do not coerce), so a payload that reaches EnqueueArgs through jobs.enqueue already has string keys. Non-str keys were always lossy on the wire: JSON objects and PG jsonb can only carry string keys, so failing fast surfaces at the boundary what used to surface as a silently rewritten key on read-back. Dropping the flag is also 1.29-1.73x faster on str-keyed input (its only effect there). See the taskq._json.dumps docstring for the full reasoning.

heartbeat_timeout is enforced by the reclaim sweep

Unreleased. New enforcement for a parameter that was previously accepted and silently ignored.

heartbeat_timeout, accepted by every enqueue API (JobsClient.enqueue, the TaskQ facade, SubJobEnqueuer.enqueue() / enqueue_batch()), is now enforced: the leader's reclaim sweep reclaims a running job whose holder has been silent past the job's heartbeat_timeout, exactly as an expired lock is reclaimed, through the same crash-recovery transitions and the same reason='lock_expired' outbox channel (the event carries cause='heartbeat_timeout'), even while the global TASKQ_LOCK_LEASE lease is still valid. Previously the value was stored and read by nothing. A non-positive value now raises ValueError at the enqueue boundary (mirroring start_to_close's rule: a zero-or-negative timeout anchors the deadline in the past and would reclaim a healthy job on the first sweep tick). Size it >= 2x the fleet's TASKQ_HEARTBEAT_INTERVAL. The supporting partial index ships as migration 01.00.10_01; fleets that set no heartbeat_timeout keep an empty index and unchanged sweep cost.

Reservation and rate-limit denials are counters, not event rows

Unreleased. Breaking for anything that consumed per-denial event rows.

Reservation/rate-limit denials no longer persist per-occurrence rows: a denial increments the bounded snooze_count/rate_limit_blocked_count counter columns on the job row and emits an OTEL counter; it writes no job_attempts/job_events row and no longer raises max_attempts. Anything that consumed per-denial event rows (e.g. dashboards over job_events) must read the counters or OTEL instead.

Snoozing and denials no longer raise max_attempts; both refund the claim's attempt

Unreleased. Breaking for anything that read max_attempts as a counter that deferrals inflate.

max_attempts is now immutable: no automatic code path raises it (the ceiling is a bound, not a counter; nothing a deferral, denial or retry loop does moves it). The one deliberate exception is the explicit operator re-run, retry_job, which raises the ceiling to GREATEST(max_attempts, attempt + 1) (capped at the smallint bound) so a re-pended job can always run at least once more; the ceiling rises only by explicit admin action. Two behaviours follow from the immutability rule:

• Actor-requested deferrals are unbounded, and never spend budget. A Snooze, or a RetryAfter(consume_budget=False) honouring a server 429, refunds the dispatch claim's attempt increment (attempt - 1, floored at 0), when work is deferred without executing, the attempt reservation is released so the job can wait out an unready downstream indefinitely: attempt oscillates and never walks toward the smallint ceiling, and max_attempts never moves. Backoff keys off real executions only.
• Admission denials have HTTP-429 semantics. A reservation or rate-limit denial means "come back later": the actor body never ran, so the denial refunds the claim's attempt increment, spends no retry budget, never writes a job_events or job_attempts row, and never by itself terminally fails the job. A denied job is rescheduled with backoff for as long as the bucket stays saturated. The single bound on a job that is never admitted is its schedule_to_close deadline, which fails it through the ordinary deadline path: a queue or limiter misconfiguration cannot kill a job whose only offence is that the fleet was busy more times than its max_attempts.

Sustained contention stays visible on the aggregated counters the job row already carries (rate_limit_blocked_count and snooze_count) and on the denial metrics, rather than as one durable row per denial.

A consuming RetryAfter (the default, consume_budget=True) is unaffected: it is a real execution asking for a known delay, so its budget exhaustion still terminally fails the job, deadline or no deadline.

Operator re-runs (retry_job) keep the attempt and raise the ceiling

Unreleased. Behaviour change for operator tooling that reads the job row after an admin retry; nothing breaks at a call site.

retry_job no longer resets the attempt counter. In v0.2.2 a re-run set attempt = 0, handing the job a full fresh budget; the row now keeps its attempt (an idempotent admin operation must not restart the counter, and a re-pended job climbs to fresh attempt numbers so no job_attempts write can collide on a spent epoch's primary key) and raises max_attempts to GREATEST(max_attempts, attempt + 1), capped at the smallint bound (32767). An exhausted 3/3 job comes back as 3/4: one fresh execution, not three; a mid-budget 1/5 job stays 1/5 and keeps its remaining budget. When the attempt already sits at the cap the retry is refused and the row stays terminal rather than re-pending a job whose next claim would overflow the attempt column. The retryable source statuses also widened: every terminal status is now valid (succeeded: the replay path after a bad deploy, and abandoned included), while running stays excluded because re-pending a live row races its own terminal write.

Graceful shutdown interrupts: it no longer terminalises in-flight work

Unreleased. Behaviour change with one additive pre migration (01.00.12_02_pre_job_interrupt_count.sql; apply it before rolling the code, as with every pre file).

When a deploy's grace windows expire with a job still running, the worker no longer writes a terminal state for it. The job is interrupted: released back to the fleet as pending (the actor provably exited; an async actor unwound on the cancel, a sync actor's thread finished, the transactional unwind completed; the consumer parks on the tracked exit handles, bounded by the remaining termination budget, before writing) or scheduled behind the remaining TASKQ_TERMINATION_GRACE_PERIOD budget plus the watchdog's exit tail (the dump-interval lag before the deadline trip is observed, and the ~2s bounded flush before os._exit), when the actor never provably exits (the row stays unclaimable until the exiting process is provably gone; with TASKQ_WATCHDOG_ENABLED=false the hold is TASKQ_LOCK_LEASE). The no-concurrent-run promise is two-legged: it holds when the watchdog is enabled, or when the platform SIGKILL lands at or before TASKQ_TERMINATION_GRACE_PERIOD plus the exit tail. The claim's attempt increment is NOT refunded: the attempt did start executing, and a refund would re-create the attempt epoch the interrupted handler still holds, letting its late terminal write land on the re-dispatched attempt. An interrupted claim therefore spends the attempt; a job interrupted on every deploy climbs its retry budget until it finishes or its schedule_to_close fails it with DeadlineExceeded. The release writes one job_events transition with reason = 'interrupted' and bumps the new interrupt_count column on the row; taskq.jobs.interrupted{actor,hold} and taskq.jobs.interrupted_noop are the OTEL counters.

What to audit:

• Sync actors are no longer assumed gone at the cancel. A plain def actor's thread cannot be interrupted: before this change its row went straight back to pending while the body was still executing, and a second worker could claim it: two live runners for one row across every deploy that caught a sync actor mid-body. The release now waits (bounded by the remaining termination budget, capped by the lease) for the thread and holds the row behind the process's exit window when the thread outlives the wait. A fleet of sync actors should teach long loops to poll ctx.should_abort() so they exit inside the cancellation grace and get the immediate pending release instead of the held one.
• An actor that outlives its budget is now ended at the deadline trip. The shutdown watchdog stays armed until every tracked actor handle is reaped, and the trip carries a dedicated tracked-actor-outlived-teardown reason. Previously the clean path disarmed the watchdog and then joined the detached thread for up to THREAD_JOIN_TIMEOUT (300s): a released row could become claimable while its actor still ran. The thread's late completion accomplished nothing (the row was already released-with-hold and the fleet re-attempts the work), so ending it at the deadline loses nothing, but an embedder's own cleanup that expected the process to outlive a stuck actor no longer does.
• Two new startup warnings, no new hard validation. The release park is lease-capped (lock_lease - heartbeat - write budget), which makes the mid-park lease expiry structurally impossible for every loadable config; release-park-lease-capped fires when that cap binds before the budget bound (interrupted sync actors release earlier with longer holds safe but slower). lock-lease-below-disown-exit-floor fires at the shipped defaults (60 vs 63): both release writers failing their writes leaves a ~3s window in which the reclaim sweep can take the disowned row before the trip ends its actor. Raising TASKQ_LOCK_LEASE to 63+ closes the residue; whether to spend those 3 seconds of lease on a double-write-failure shape is an operator call this release surfaces rather than makes.
• abandoned now means "operator cancel". A graceful shutdown never produces cancelled or abandoned for infrastructure reasons; any alert or dashboard that reads those statuses as deploy noise should now treat them as operator intent. (The abandoned-jobs alert is purely an operator-cancel signal now.)
• Actors that return early on cancel persist that result. A cancel request, operator or deploy, no longer overrides an actor that returns a value: returning records succeeded and keeps the result. Actors that must not keep a partial result on a deploy should read ctx.cancel_origin and re-raise on SHUTDOWN (see cancellation.md).
• The phase-4 name changed. ShutdownPhase.ABANDONING is now ShutdownPhase.RELEASING; the integer value 4 is unchanged, so /health JSON and the CLI keep their numbers, but the phase="RELEASING" log string and any code importing the old enum member must move.
• Long actors re-run from scratch on every deploy. Anything longer than cancellation_grace_period + cleanup_grace_period is interrupted and re-claimed repeatedly; bound such actors with schedule_to_close or checkpoint via progress state (the released row carries the last checkpoint).

Maintenance leadership is now a bounded lease

Unreleased. Requires one new pre-phase migration.

maintenance_leader gains an expires_at column (01.00.12_01_pre_leader_lease.sql). Leadership is held by the row while its lease is unexpired and renewed on the heartbeat cadence, so a leader that dies without closing its connection is replaced within TASKQ_LEADER_LEASE plus one heartbeat instead of waiting on the server's connection-reaping horizon. The migration is pre-phase and rolling-safe: it only adds a nullable column that pre-lease pods never read, and rolling back to the previous release ignores it. Recovering a dead leader now needs no privilege beyond table access, so the pg_terminate_backend runbook steps are gone. The advisory lock a new leader also takes is a transition courtesy for mixed-version fleets and is never waited on; it is removed entirely in the release after every pod in your fleet runs this one. Size TASKQ_LEADER_LEASE at four or more heartbeat intervals (the default, 40 s, already is).

The SAML cookie-less ACS fallback is opt-in and defaults off

Unreleased. Breaking for SAML deployments whose users complete login with the correlation cookie withheld by the browser.

The SAML ACS callback previously accepted a response with no usable taskq_saml_request correlation cookie whenever its validated InResponseTo named a pending AuthnRequest the process had issued, a fallback for browsers that block the cross-site cookie. Two problems came with it: nothing tied that response to the browser posting it, so a party who starts a login and captures the signed response for their own account could plant it on a cookie-less victim (login CSRF); and the same process-local pending-set requirement rejected every SAML login whose callback landed on a different admin replica or uvicorn --workers N process than the one that issued it (#239).

One policy fixes both: the signed correlation cookie is the binding; verifiable by every replica sharing session_secret, so cross-replica callbacks now succeed with no sticky sessions, and the cookie-less fallback moved behind TASKQ_SAML_ALLOW_COOKIELESS_FALLBACK, default false. What to do on upgrade:

• Default deployments (flag unset): cookie-less callbacks are now refused with the standard login error. A user who hits it should retry from the same browser, which re-issues a fresh correlation cookie. If it recurs for users on third-party-cookie-blocking browsers, decide whether to opt in.
• Deployments that must serve cookie-blocking browsers: set TASKQ_SAML_ALLOW_COOKIELESS_FALLBACK=true and accept the login-CSRF tradeoff; it is stated directly in sso.md. With the flag on, put the SSO routes behind sticky sessions: the fallback's pending-request record is per process, so a cookie-less callback must land on the process that issued the login.
• Sticky sessions are no longer required for the default (cookie) path; multi-replica and uvicorn --workers N deployments can drop them for SAML logins.

---

Silent behaviour changes

These change what your code does without changing what it accepts. Nothing raises, so nothing points you at the call site; audit for them explicitly.

• State-change events now consume the progress seq (it is the event stream's total order). A state-change event (the dispatch running transition, retries, snoozes, and every terminal or interrupted exit) used to carry the last progress event's seq unchanged; it now carries one past it, and the durable jobs.progress_seq ends at the terminal event's consumed value. Consumers deduping or resuming by seq alone were the reason: a guard keyed on seq treated every state-change event as a duplicate of the progress event sharing its number and dropped it, so a terminal event could never arrive and an SSE stream never closed. Existing consumers that dedupe progress events see extra, correctly-ordered values and need no change; consumers that special-cased the old repeats (or snapshotted from Postgres on every reconnect because seq-cursor delivery of state was unreliable) can drop the special cases. Scope the guarantee to events whose durable write landed: across worker death it does not hold, because the reclaim sweep does not touch progress_seq, so the reclaimed job's next attempt re-publishes running at a seq the wire already carried (a terminal publish whose mark_* write failed repeats its seq on retry the same way), and a consumer can see the same seq twice with different payloads; the Postgres snapshot stays authoritative across the gap. During a rolling deploy, workers still on the previous release emit state-change events that repeat the head instead of consuming a new seq until they are upgraded; both windows produce duplicates a seq-cursor guard already drops. There is no migration: the counter keeps its column, type, and monotonicity, only the state-change increments are new.

• The wake payload names the inserted row's queue, and queue-scoped subscribers filter on it. The insert trigger's NOTIFY carried an empty payload and woke every worker's producer; it now names the row's queue (migration 01.00.16_01, applied automatically), and a worker's listener wakes its producer only when the payload names a queue the worker serves. An EMPTY payload wakes every subscriber unfiltered (the COPY fixup's bulk wake, or an older trigger still live in a rolling deploy). A filtered wake never drops work: the worker's jittered fallback poll stays authoritative, so the worst case for a dropped wake is one poll interval of added latency. External code that LISTENs on the wake channel and parses payloads (nothing in TaskQ does) sees the new payload shape.

• ctx.progress() coalesces Redis publishes: at most one publish per job is in flight at a time. Every call used to fire its own background publish; a call racing a running publish now latches onto the buffer and the running task re-publishes the latch when its round trip lands, so the final publish always lands and the latest state is what subscribers see. Intermediate events under publish pressure are skipped, which the consumer-side seq-discard discipline already treats as replaceable. Postgres-side coalescing is unchanged.

• Cron payload-factory budget exhaustion now defers instead of striking, and micro-grants no longer exist. A factory-backed cron schedule the tick's funded budget could not pay for used to take a failure strike (consecutive_failures +1, last_fire_error set to a "tick budget exhausted" reason), marching healthy schedules behind one hung factory to auto-disable in lockstep. It now advances next_fire_at one leader tick and retries: no strike, no error text. A leftover too small to honour was also previously granted as a micro-grant, cutting slower factories into manufactured "timed-out-after-0.08s" strikes behind a slow-but-successful monopolizer that never drains; a factory is now called only when the leftover can fund at least TASKQ_CRON_PAYLOAD_FACTORY_TIMEOUT or a quarter of the tick's funded budget, whichever is smaller; smaller leftovers defer. Operators keying on the old last_fire_error text should watch the cron-fire-budget-deferred log event and the new taskq.cron.budget_deferrals counter instead (see the cron guide's tick-budget section and the TaskQCronBudgetDeferrals runbook; genuinely slow/hung factories still strike exactly as before).

• taskq_leader_lock_contention_total stops rising in a healthy fleet. It previously incremented on every follower's every heartbeat, so the sustained-rate alert fired permanently wherever more than one pod ran. It now records one event per distinct holder a pod finds in its way. Dashboards that graphed the old always-rising counter will go flat.

• dispatch_scope_by_home_queue is a deprecated no-op. The setting (removed from the docs of the old probe-narrowing shape it applied to) is accepted again so configurations that set it keep loading, but dispatch is assignment-routed now and the flag has nothing left to apply. The worker logs a deprecated-setting-ignored WARNING at startup when it is set; delete TASKQ_DISPATCH_SCOPE_BY_HOME_QUEUE from the environment.

• Scrubbed exception messages on the job_exception / job_timeout log lines are bounded at 2000 characters. The durable ErrorInfo row keeps the full text (the admin UI reads it there); only the log line truncates, and it reports the dropped remainder count. The bound follows TASKQ_EXCEPTION_MESSAGE_MAX_CHARS when raised.

Terminal outcome writes raise on drift instead of falling through silently

Unreleased. Backend implementors and test-twin drivers see a new exception in the drift window; application code using the client API through a worker is unchanged.

Terminal writes used to resolve through a final catch-all branch: when a job row no longer matched the branch's preconditions (an operator cancel landed between the read and the write, a claim epoch moved, a sibling attempt terminalised first), the write quietly reported success while writing nothing, and the drift was invisible. The outcome branches are now typed and exhaustive: a row outside every branch's preconditions raises, so the drift surfaces at the write instead of disappearing into a no-op success. denial_reason also persists on the snoozed-deadline exit, where it used to be dropped.

Code driving terminal writes directly (a custom Backend implementation, or tests driving the taskq.testing in-memory twin) must handle the raised error. Application code enqueues and awaits through JobsClient/JobHandle and never touches a terminal write; the worker's own drift handling is unchanged. The Postgres backend and the in-memory twin enforce the same contract, pinned by the terminal-drift guard tests.

A crash reclaim honours an in-flight cancel: the row ends cancelled, not pending

Unreleased.

If a job's holder died while a cancel request was in flight and the lock stayed expired past cancel_grace + cleanup_grace + 60s, the crash-reclaim sweep used to put the row back pending with cancel_phase and cancel_requested_at wiped whenever retries remained, so the retried attempt ran to completion even though a cancel had been requested. The reclaim now evaluates operator intent first: a row carrying cancel_phase != 0 past that horizon terminalises cancelled whatever its retry budget, keeping cancel_phase/cancel_requested_at as the audit trail of the honored request and stamping finished_at. The attempt row still records outcome='crashed'/WorkerCrashed; the job row left by a worker that isolated itself on heartbeat loss now also carries error_class='HeartbeatLost' where it used to be NULL. Automation that expected a cancel-addressed row to re-run after its holder crashed now sees a terminal cancelled row instead (see Cancellation and the Crash-reclaim interaction section of Architecture).

unique_for's default unique_states now includes succeeded

Unreleased. Breaking for actors using unique_for without an explicit unique_states.

The default unique_states was ("pending", "scheduled", "running"); it is now ("pending", "scheduled", "running", "succeeded"). unique_for reads as "at most one job for this identity in this period"; leaving succeeded out freed the identity the instant the first job completed, so a re-delivered webhook or a double-clicked button inside a still-open window could run the work a second time. The failure states (failed, cancelled, crashed, abandoned) remain excluded: they mean the work did not happen, so matching them would let one transient failure suppress every later attempt for the rest of the window. This matches the default every comparable job queue ships (the completed state is included in the uniqueness check by default).

To keep the old "block only concurrent execution" behaviour, pass unique_states=("pending", "scheduled", "running") explicitly on the actor.

A dedup onto a job that already finished is now surfaced: the enqueue logs a WARN-level enqueue_deduplicated line naming the matched status, and JobHandle.deduplicated_onto_terminal is True.

migrate.apply_pending_locked defaults to the pre phase only

apply_pending_locked(...) previously applied every pending migration when called without a phase argument; it now applies only pre-phase migrations. The entry point exists to fire on process lifecycle events: a pod restart, a rollout, an autoscale event, that nobody sequences, and a post-phase migration exists precisely to be withheld until the whole fleet is confirmed upgraded. Letting a restart apply one would close a rolling-deploy overlap window mid-rollout (for example, dropping the old single-column idempotency index while half the fleet still issues ON CONFLICT (idempotency_key) takes that half's entire enqueue path down).

If you call apply_pending_locked from your own deploy tooling and relied on the old all-phases default, pass phase=None explicitly to restore it; from a context that knows the fleet is fully upgraded. The operator-sequenced path is unchanged: taskq migrate up --phase post remains the way to close out a phased migration.

Rate-limit refunds now credit the store that paid

If you run backend="redis" rate limits with rate_limit_pg_fallback_enabled (the default), refunds were previously credited to Redis even when a Redis outage had caused the acquire to fall through to Postgres and spend the token there. Postgres was never repaid; permanently, for a fixed-quota bucket with refill_per_second == 0, and Redis gained a token it never spent.

Refunds now dispatch on the store the acquire actually used. Expect your effective quotas to shift after upgrading: Postgres-side buckets that had silently drained will recover, and Redis-side buckets that had been inflated will return to their configured capacity. If you had raised a capacity to compensate for the drift, re-check it against the corrected behaviour rather than leaving the compensation in place.

Two further refund defects in the same area are fixed: the in-memory and Postgres log-style sliding-window refund() was a silent no-op (it now properly frees slots), and the Postgres token-bucket refund() was likewise a no-op (it now properly refunds tokens, capped at capacity, via FOR UPDATE on rate_limit_buckets). Both were released behaviour: release-and-retry cycle never gave the slot back, so fixed-quota buckets may again admit work that had been permanently locked out.

@actor(...) capacity literals no longer win over the stored row

max_concurrent, max_pending and result_ttl are now operator-owned. sync_actor_config seeds them on an actor's first registration and never writes them again: the UPSERT omits them from its SET clause, and a difference between the code literal and the stored row is logged at INFO as actor-config-capacity-override instead of raising ActorConfigDriftList.

This is deliberate: it is what lets an operator retune a live fleet through taskq actor set-capacity (or taskq.actor_config_ops) without a redeploy, and all three fields take effect without a worker restart.

The upgrade hazard is on schemas that have already run a worker. There, a row already exists, so changing an @actor(max_concurrent=...) literal and redeploying now has no effect: the stored value continues to win, silently. On 0.2.2 that same mismatch aborted worker startup, so the failure was loud and you could not miss it.

Two consequences worth auditing before you upgrade:

• If you tune capacity in code and rely on redeploys to apply it, that workflow no longer works. Move the value to the operator surface, or clear the override to fall back to the literal.
• If an environment variable feeds an @actor(...) capacity argument, it stops being the effective value on any existing schema.

To see where code and stored rows disagree, check for actor-config-capacity-override in your worker logs; it names every field whose literal is being ignored. To hand a field back to the code literal, clear the override: --clear-max-pending and --clear-result-ttl write NULL, which their enforcement paths read as use the @actor(...) value. However, --clear-max-concurrent does not do this: the dispatch SQL reads NULL as unlimited, because it cannot see the code literal once the row exists.

Schema-qualified advisory locks: adopt by restart

Unreleased. Silent for correctly-deployed fleets; a rolling deploy across the rename has a split-leader window.

The advisory-lock names are now schema-qualified: taskq:maintenance_leader:<schema>, taskq:cron:<schema>, taskq:prune:<schema>, taskq:archive_expiry:<schema>, taskq:migrate:<schema>, replacing the unqualified (taskq:maintenance_leader, …) forms outright. Advisory locks live in a per-database namespace, so the unqualified names serialized every schema in the database against each other: two schemas sharing one PG instance meant one schema's leader could silently starve the other's. The qualified names give each schema its own locks.

The upgrade hazard is a rolling deploy. Old and new workers hold different lock names, so during the roll both an old and a new worker can act as leader of the same schema at once. The maintenance sweeps stay row-safe in that window (every snap uses FOR UPDATE SKIP LOCKED), but cron gains a double-fire window; its advisory lock is what serialises ticks. Nothing errors anywhere; the fleet-level signal is sum(taskq_maintenance_leader_is_leader) != 1.

Adopt by restarting the fleet onto the new release, not by rolling it. Stop the old workers, start the new ones. The exposure window is the deploy itself, not the steady state. See maintenance-sweeps.md §5 for the full reasoning and runbooks.md for the alert whose remediation carries this note.

A cross-actor idempotency_key hit now raises

Unreleased. Breaking only for callers that share one idempotency key across actors and relied on the second enqueue returning the first actor's job.

enqueue / enqueue_batch with an idempotency_key that matches an existing job of a different actor used to return that job's handle with was_existing=True: a handle whose .result() belongs to another actor, logged at INFO as an ordinary dedup. It now raises IdempotencyKeyActorMismatchError (naming both actors, the key, the scope and the existing job id) with nothing enqueued; in enqueue_batch the whole batch is withdrawn. enqueue_batch_fast aborts the whole COPY either way (all-or-nothing bulk-import semantics are unchanged) and now classifies the abort the same way: a cross-actor pair raises IdempotencyKeyActorMismatchError, a same-actor pair keeps DuplicateIdempotencyKeyError. Same-actor hits on the single and batch tiers are unchanged. Keys were always documented as unique per scope across actors; namespace them per actor ("send_receipt:order_123") or use per-actor idempotency_scope values. See jobs-clients.md.

NOTIFY channels embed a hash of the schema: adopt by restart

Unreleased. Silent for correctly-deployed fleets; a rolling deploy across the rename leaves old workers polling instead of woken. Fixes a bug for schema names of 14 characters or more.

Every NOTIFY channel (the wake channel, the fleet events channel, the per-worker cancel channel, the progress channels and the cron commit gate) now embeds schema_channel_tag(schema) (the first 10 hex digits of sha224(schema)) in place of the schema name: taskq_wake_taskq becomes taskq_wake_124a200651. Channels are Postgres identifiers bounded by 63 bytes, LISTEN silently truncates longer ones and pg_notify rejects them, and the schema name alone may be 63 characters. With the name interpolated, the per-worker cancel channel (13 + schema + 37 bytes) broke from a 14-character schema on: every cancel's NOTIFY errored, was logged as cancel-request-notify-failed, and cancellation fell back to the heartbeat poll, and the wake channel broke every enqueue from a 53-character schema. The tag makes every channel's length independent of the schema; settings loading now refuses any schema whose channels would not fit.

Migration 01.00.14_01_pre_wake_channel_schema_tag.sql redefines the wake trigger to compute the same tag in SQL. Once it applies, workers of the previous release stop receiving wakes (they LISTEN on the old name) and claim on their poll interval until restarted; new workers are woken normally. Nothing errors. Adopt by restarting the fleet onto the new release after migrating, rather than rolling it; the same discipline as the schema-qualified advisory locks above.

Anything outside TaskQ that issued SELECT pg_notify('taskq_wake_<schema>', '') must switch to the derived name; see workers.md for the SQL expression.

Retries at the backoff cap spread over [cap × (1 − jitter), cap]

Unreleased. Silent; only the distribution of capped retry delays changes. The documented bounds are unchanged and jitter=0 remains the identity.

Jitter was applied to the saturated curve value and the draw clipped at the effective cap, so the upper half of the band collapsed onto the cap exactly and about half of a cohort retrying at the cap came due at the same instant; the herd jitter exists to spread, on the retries most likely to follow a fleet-wide event. The band is now fitted under the cap before the draw: a capped retry (the default exponential policy from attempt 11, any fixed/linear policy whose base reaches the cap, any curve above max_retry_backoff, every reclaimed cohort at the ceiling) is drawn uniformly from [cap × (1 − jitter), cap] (e.g. [48 min, 60 min] for the default policy). The RNG path and the row-derived reclaim path (Python and SQL) evaluate the same expressions, so they remain bit-for-bit equal. See retries.md.

Transactional migrations give up waiting for a table lock after 30 s

Unreleased. A migration that previously parked indefinitely behind a lock holder now fails with a typed error; nothing else changes.

Each transactional migration runs under SET LOCAL lock_timeout of DEFAULT_MIGRATION_DDL_LOCK_TIMEOUT (30 s) and raises MigrationLockTimeoutError, naming the migration (or the runner's own ledger upgrade), the bound and the remedy, with the driver's LockNotAvailableError as its cause, instead of waiting forever with every jobs statement queued behind it until the fleet's heartbeats gave out. Nothing is applied on a timeout; re-run after ending the holder. The bound is settable with taskq migrate up --ddl-lock-timeout or ddl_lock_timeout= on apply_pending / apply_pending_locked (0 restores the unbounded wait). -- taskq:no-transaction migrations are unaffected. See the migration gave up waiting for a table lock.

Migration 01.00.06_01 takes write-blocking index locks

Unreleased. Operational note for the jobs / job_attempts index migration; nothing breaks, but the apply can stall fleet writes.

Migration 01.00.06_01_pre_cancel_and_cascade_indexes.sql adds the indexes that serve the bounded bulk-cancel/deregistration drains and the stale-worker cleanup fan-out. It uses plain transactional CREATE INDEX: each build takes a write-blocking lock on its table for the duration, and jobs is the hottest table in the system (enqueue, dispatch and heartbeat all write it). Build time scales with the current row count: on a large, busy production jobs table the apply can stall the worker fleet's writes for a noticeable window.

• Apply during a maintenance window, or when jobs is small/quiescent (e.g. right after a prune sweep), on any deployment where jobs is large. Most deployments see momentary builds, the bounded maintenance sweeps keep steady-state jobs small.
• The wait for the lock is bounded (ddl_lock_timeout, 30 s by default, see the migration gave up waiting for a table lock): a session holding jobs open fails the migration cleanly instead of parking the fleet behind it. The bound never interrupts a build that already holds its lock.
• It is transactional deliberately: the CREATE INDEX CONCURRENTLY form deadlocks under the migration runner's own serialized-migrator advisory lock (the concurrent build waits on every transaction that started before it, including a second replica's blocking lock wait: a cycle the deadlock detector breaks by failing the apply). This follows the 01.00.02_01 precedent; the migration file's header carries the full derivation.

Rolling deploys from 0.2.2: old pods' prune sweep fails after 01.00.03

Known, bounded, self-healing. Documented in the 01.00.03_01_pre_idempotency_scope.sql header (RESIDUAL RISK, reviewed twice); repeated here because it is the one rolling-deploy cell the migration chain cannot protect.

A fleet rolling from 0.2.2 (whose schema predates 01.00.03) to this release: 0.2.2's prune/archive sweep moves terminal rows with a positional INSERT INTO jobs_archive SELECT j.*, ..., which relies on jobs and jobs_archive sharing physical column order. 01.00.03 appends idempotency_scope to both tables, and ADD COLUMN always appends at the end of each table's OWN order, so on jobs_archive the new column lands after archived_at/expire_at while on jobs it lands after tags: the orders diverge, and every old-pod prune attempt fails with a type error (the text lands in the archived_at timestamptz position) until the last old pod exits. Non-destructive: the sweep's CTE rolls back cleanly, nothing is lost or corrupted, and dispatch/enqueue/dequeue are unaffected. The cost is retention lag while old pods lead: terminal rows accumulate for the window and the old leader logs a prune failure per sweep tick. This release's sweep names every column explicitly, so pruning drains as soon as a new pod holds leadership. Fleets that need a bounded window: apply the migrations and roll the pods back to back, or promote a new pod to the maintenance leadership before the retention cutoff matters.

Migration 01.00.13_03 adds the batch open-members index

Unreleased. Operational note; nothing breaks. Same lock caveat as 01.00.06_01 above.

01.00.13_03_pre_jobs_batch_open_members_index.sql adds jobs_batch_open_members_idx, a partial index over the non-terminal members of every batch keyed by batch_id. It serves the member probe every batched job's terminal write runs (complete_batch's guard and the remaining count the failure-counter writes return, the same probe as count_batch_non_terminal) as one index range over the members still open, where the metadata @> containment form walked the batch's whole membership on every write. Pre-phase and rolling-safe: the previous release's statements never reference the index and this release's run without it (only the cost bound is lost). Like every sibling index migration it is a plain CREATE INDEX; the build takes a write-blocking lock on jobs, so on a large jobs table build it CONCURRENTLY by hand first (the statement is in the migration file) and let the migration no-op. As with 01.00.06_01, the wait for the lock is bounded by ddl_lock_timeout and a held jobs fails the migration cleanly rather than stalling the fleet; the build itself, once it holds the lock, is not.

The assignment-routed marker round applies in bounded lock windows

Unreleased. Operational note for the 01.00.12_05 / 01.00.12_07 / 01.00.12_08 / 01.00.12_09 round; the upgrade's lock profile changes, the end state does not (same columns, same backfill, same indexes plus the two new probe indexes below).

The marker round originally shipped as one migration whose single transaction held ACCESS EXCLUSIVE on jobs (taken by the ADD COLUMN) across the re-pend backfill and two full-table index builds: every read, heartbeat and claim on the table queued behind it for the whole run, long enough to burn a live fleet's heartbeat budget during the apply (the monolithic-migration lock-hold fix). It is now four single-purpose migrations, each holding the narrowest lock its work allows:

• 01.00.12_05: only the two metadata-only ALTER TABLE ... ADD COLUMN statements. The ACCESS EXCLUSIVE window is the catalog writes plus the commit: milliseconds, independent of table size.
• 01.00.12_07: only the backfill UPDATE. An UPDATE holds ROW EXCLUSIVE, which blocks neither readers nor other writers, so a deep re-pend backlog backfills for as long as it needs without parking the fleet.
• 01.00.12_08 / 01.00.12_09: only the CREATE INDEX builds. Each build takes a SHARE lock (writes queue, reads keep flowing). The runner wraps each file in one transaction, not each statement; so a file's builds share ONE write-block window whose duration is the sum of that file's builds (01.00.12_08's two builds run back-to-back with no drain between them; measured, a writer INSERT blocked 1.04 s behind both vs 0.49 s behind a single-build file). Postgres' FIFO lock queue grants the writes that queued behind a file when it commits, before the next file asks for the table: the fleet sees one write-block window per file, never one continuous window across the whole round, and reads never block.

Same caveat as 01.00.06_01 above on the builds themselves: build time is proportional to the jobs row count, and on a deployment where a single build would outrun the workers' heartbeat budget the statements to pre-build CONCURRENTLY by hand are in each migration file's OPS NOTE (the migration then no-ops via IF NOT EXISTS). The wait for each lock is bounded by ddl_lock_timeout as everywhere else.

Two more changes land with the same round (the probe-index removal and the checksum-ledger cleanup):

• The dead probe index pair is gone. The unreleased stack built jobs_repended_probe_idx (01.00.11_01) and dropped it again two files later (01.00.12_05:post); no shipped code ever read its predicate, so every upgrade paid one write-blocking full-table build for nothing. Neither file ships now. A database that ran a dev checkout of the unreleased stack carries the index only if its 01.00.12_05:post never ran: a pre-phase-only or intermediate-era database (one that ran the stack's full pre AND post phases already had the index dropped by the post file). Such a database also keeps the deleted files' ledger rows; if it carries the index, drop it by hand (DROP INDEX IF EXISTS "{schema}".jobs_repended_probe_idx;), nothing reads it. Released deployments never had it.
• A database that ran a dev checkout of the unreleased stack will also log a migration-checksum-drift warning on every future migrate up, for 01.00.12_05:pre (the file was restructured into the columns/backfill/index files above) and 01.00.13_03:pre (a comment-only header fix changed its rendered checksum). The warning means exactly what it says: the ledger's recorded checksum for that key no longer matches the bundled file: the runner compares them, warns, and skips the file because the ledger key is already recorded. It is permanent for as long as that ledger lives (released checksums are frozen and the files will not be reverted), and it is harmless-but-expected for pre-release dev checkouts: nothing fails, nothing re-applies, and every new migration applies normally. Released deployments never see it: their ledgers never held the pre-restructure checksums.
• 01.00.12_09 adds the producer-placed population's two probe indexes (jobs_unrouted_actor_dispatch_idx, jobs_unrouted_round_robin_probe_idx), partial on status = 'pending' AND NOT assignment_routed. The label-routed dispatch arms filter NOT assignment_routed; on the pending-only twins that conjunct is a post-scan Filter, so every claim probe walked the re-pended rows ahead of the rows it could admit, so claim cost grew linearly in re-pend depth, exactly on the crash-reclaim tails a fleet-wide restart produces. The probes now ride the marker-partial indexes and visit zero re-pended rows; the re-pended population keeps its own probe index (jobs_assignment_routed_probe_idx, in 01.00.12_08), so each arm's probe stops at its own population's rows.

Bulk cancel and force-deregistration now make bounded committed progress

Unreleased. Changes the failure semantics of JobsClient.cancel_where() and deregister_actor(force=True); both remain correct to re-run.

These operations previously did all of their work in one unbounded transaction; they now drain their match set in bounded committed batches (event_writer_batch_size rows per transaction, each with a server-side statement_timeout). A mid-operation failure therefore leaves the batches that already committed as durable partial progress instead of rolling everything back, and a re-run continues where the stopped one left off, because already-cancelled rows fall out of the match set. If you relied on all-or-nothing semantics (e.g. aborting a tenant offboard on any error and expecting zero cancels), re-check the returned counts before retrying: they now reflect only what the completed batches did. See maintenance-sweeps.md for the full semantics, including why the drain terminates on the window count rather than the affected-row count.

Sub-jobs inherit parent tags by default

Unreleased. Silent for code that did not rely on sub-job tags being empty.

Every ctx.jobs.enqueue() call inside an actor body now propagates the parent job's tags to the sub-job, making sub-jobs findable by JobFilter(tags=...) and cancellable by cancel_where. Pass inherit_tags=False per-call to opt out. This is a behaviour change for any code that relied on sub-job tags being empty: inherited tags make sub-jobs visible to tag-based filters and bulk cancels.

WorkerSettings post-load validation runs on every load path

Unreleased. Silent unless a reload() or a validate=False load produces values an earlier load would have accepted.

dotenvmodel is bumped to 1.x (>=1.1.0,<2, the same bump as the environment-variable precedence change earlier in this guide) and WorkerSettings uses dotenvmodel's native post_load() hook instead of manual load()/load_from_dict() overrides. The base DotEnvConfig._load_fields invokes post_load automatically on every load path: load(), load_from_dict(), and reload(), including under validate=False, so a reload() that produces invariant-violating values now fails instead of silently succeeding. log_format validation also moved from dotenvmodel's choices= constraint (which load_from_dict(..., validate=False) skipped, so an invalid TASKQ_LOG_FORMAT could previously load silently) to a validator hook that runs regardless of validate=; its error message is now log_format must be one of ['console', 'json'], got <value>.

Every mixed-clock decision is single-arbiter on the store's clock

Unreleased. Silent; behaviour under clock skew changes.

The application process and the database server keep separate clocks that can diverge or step (VM pause/resume, NTP drift); every place that mixed the two domains in one decision is anchored to the database clock: workgroup supervisor freshness is computed server-side (a skewed supervisor host can no longer kill healthy children); cron ticks read the server clock inside the leader transaction, with the catch-up cutoff and beyond-window recompute server-anchored (no fire-loops or silently skipped backlog under leader-clock skew); rate limiting runs on the store's clock (PG window predicates and GCRA/token-bucket epoch math are server-side; peeks measure against the store clock too); prune/archive cutoffs and enqueue-pinned result TTLs are stamped server-side; the batch COPY path is server-stamped via an in-transaction fixup (status, created_at, scheduled_at, schedule_to_close, result_expires_at), so dedup windows hold under skew.

The admin session cookie is scoped to the admin mount path

Unreleased. Silent; expect one session lifetime of cookie overlap on upgrade.

taskq_session carried no path=, so it defaulted to / and the browser attached it to every request to the host application that mounts the admin UI, including routes with no reason to see an admin session. Both SSO backends now set path to their base_path, and logout clears it on the same path (a delete on a different path clears nothing, which would have left a live session behind). A stale path=/ cookie written by a previous version is not replaced by the new one: the browser keeps both and sends both, and the broader one can shadow the narrower until it expires. Operators upgrading should clear the taskq_session cookie, or expect one session lifetime (session_max_age_seconds, default 8h) of overlap.

ctx.progress() no longer blocks the actor

Unreleased. Silent; removes a synchronous Redis round-trip from the actor body.

Progress publishing was fire-and-forget in name only: ctx.progress() blocked the actor on a synchronous Redis round-trip. It now publishes via background tasks with a drain on shutdown.

Worker failure diagnostics are no longer swallowed

Unreleased. Silent; log volume and log fields change.

Timeout and generic-exception attempts log job_timeout / job_exception WARNING events carrying error_class / error_message / error_traceback; every terminal (non-retryable) failure across all five handlers emits exactly one job_failed ERROR event (job_id, actor, attempt, cause, error_class, plus handler context such as snooze_count / consume_budget / bucket_name): one alertable event per dead job, and per-attempt diagnostics at WARNING so retryable attempts produce zero ERROR noise. A retry the row's schedule_to_close deadline refuses at the write (the job lands failed with cause=DeadlineExceeded) is a terminal failure like any other: it emits the same job_failed event and fires on_retry_exhausted and the ErrorReporter once. Tracebacks are formatted from the explicit exception object rather than the ambient sys.exception(), so handler invocations outside an except block no longer record 'NoneType: None'. The terminal-write-failed event now includes job_error_traceback and infra_error_traceback. Timeout spans (lifecycle.scheduled / lifecycle.failed) now report the concrete exception class instead of hardcoded TimeoutError, agreeing with the log fields. Snooze / RetryAfter / ReservationUnavailable terminal outcomes and the stranded-jobs leader sweep also log their failure details instead of continuing silently.

on_retry_exhausted awaits any Awaitable

Unreleased. Silent; non-coroutine Awaitable callbacks now actually run.

on_retry_exhausted now uses inspect.isawaitable() instead of inspect.iscoroutine(), so a callback returning a non-coroutine Awaitable (e.g. a Task or a custom awaitable) is awaited instead of silently skipped.

TaskQ(redis_url=...) is validated at construction

Unreleased. Silent; invalid URLs fail earlier and with a different exception.

The URL routes through load_from_dict, so the RedisDsn field type coerces and validates it: an invalid URL now raises TypeCoercionError fail-fast at open() (previously a late ValueError from redis-py), and an empty or whitespace-only redis_url raises ValueError at construction instead of silently disabling Redis.

The .env-not-found warning filter is narrowed

Unreleased. Silent; real dotenvmodel warnings are visible again.

The .env-not-found warning suppression is narrowed to exactly that one warning: a logging.Filter matched on message prefix, instead of raising the whole dotenvmodel logger to ERROR, so real misconfiguration warnings (e.g. an invalid DOTENV_* value) stay visible.

start_to_close now cancels the running actor

Unreleased. Silent unless an actor runs past its deadline.

start_to_close now actually cancels the running actor on the transactional path. _run_actor_in_tx wrapped the actor in asyncio.shield() inside the wait_for enforcing the deadline, and a shield keeps the shielded awaitable running when its waiter is cancelled, so the timeout applied to the wait and never to the actor. The attempt was marked timed out and became eligible for retry on another worker while the original body kept executing, running every side effect past the timeout point twice. An actor that previously ran past its start_to_close deadline will now see CancelledError at that deadline, so any cleanup it needs on interruption belongs in a finally. The autonomous path already used a bare wait_for and is unchanged; the outer shield(_run_actor_in_tx()), which decouples external cancellation from an in-flight commit, is untouched.

Cancel state is cleared on every retry arm

Unreleased. Silent; cancels of jobs that retry mid-cancel now work.

A cancel that escalated to cancel_phase=2 in the same instant the actor raised an ordinary retryable exception survived the retry write: mark_retry, mark_snoozed and both mark_retry_after variants rewrote status/scheduled_at but left cancel_phase and cancel_requested_at on the row, and a retry reuses the same row. The next attempt was therefore dispatched already at FORCED, so the cancel controller's PG-observation fast-advance jumped straight to FORCED without ever calling task.cancel() and the job could never be cancelled again, only abandoned while its coroutine kept running. Both backends now clear the cancel columns on every retry arm. Terminal arms still keep both columns: they are the audit trail, and mark_abandoned's cancel_phase=2 guard reads them.

Keyed rate-limit key_fn errors no longer embed the payload

Unreleased. Silent; error text changes.

The RateLimitRegistry "key_fn returned an empty key" ValueError interpolated the whole payload (for payload {payload!r}); that exception propagates into the persisted error_message and the web admin through generic exception handling, and payload values are attacker-controlled. The message now names only the ref, matching the sanitization contract PayloadValidationError follows in taskq._validation.

Packaging, dependencies, and documentation corrections

Unreleased. Silent; install behaviour changes.

• humanize moved from the core install to the [fastapi] extra (it was bloating core installs).
• starlette and prometheus_client are declared as direct dependencies (they were transitive-reliance).
• Dependency upper bounds are added to asyncpg, redis, pydantic, fastapi, typer, dotenvmodel, uuid-utils, uvicorn, structlog, opentelemetry-instrumentation, prometheus-client; a deployment pinning a newer version than a bound now fails to resolve instead of silently drifting.
• Stale [web] extra references in the README and CI were replaced with [fastapi]; there is no [web] extra.
• Docs corrected: configuration.md claimed TASKQ_ENVIRONMENT selects .env.{env} files; ENV does; TASKQ_ENVIRONMENT is a deployment label that gates the unauthenticated-admin warning.

job_events rows past the retention period are deleted

Unreleased. Silent; event history older than the retention window disappears. A watch_reclaims consumer whose cursor the horizon passed now fails visibly (EventRetentionGapError) instead of skipping silently.

job_events rows older than TASKQ_EVENT_RETENTION_PERIOD (default 7 days) are now deleted by a leader sweep regardless of parent-job status; timedelta(0) disables it; the crash-reclaim outbox slice (kind='state_change' AND detail->>'reason'='lock_expired') is carved out of the ordinary window so an unread reclaim event survives it, but the carve-out is bounded, not unconditional. The same sweep deletes that slice once it is older than 100x the retention period (RECLAIM_OUTBOX_RETENTION_MULTIPLIER), because a fleet with no watch_reclaims consumer would otherwise retain every lock_expired event forever. Shortening the retention period shortens that window proportionally: a TaskQ.watch_reclaims consumer lagging past 100x retention loses events - and since the event-prune watermark (job_events_prune_state, the same change) records the highest deleted event id, the resumed stream now ENDS LOUDLY with EventRetentionGapError naming the cursor and the watermark instead of silently skipping to live. Size the retention period against your slowest consumer's worst outage, not only against event volume; the gate turns an over-long outage into an actionable error, not into a recovered feed. Recovery from EventRetentionGapError after a crash-before-consume is to recreate the watcher fresh (watch_reclaims(after_id=0)), accepting the loss the error reported: the deleted events cannot be refilled.

An expired result reads as expired: ResultUnavailable.reason and the expiry stamp

Unreleased. Silent; a late poller's error is now distinguished, and result_expires_at survives the sweep it stamps.

Before this change, the result-TTL sweep nulled result, result_size_bytes AND result_expires_at together, so a client polling JobHandle.wait after the TTL got a ResultUnavailable that read exactly like a job whose actor had returned None: the loss was observable but not explainable, and at the instance level it was indistinguishable from a job that never stored a result. The sweep now nulls only result and result_size_bytes and KEEPS the past result_expires_at stamp, and ResultUnavailable grew a reason attribute: "result_ttl_expired" (with the expiry instant in the message) when the row carries the past stamp, "not_stored" when it carries no stamp or a future one. One honesty note: the stamp proves the result's TTL elapsed, no more. The completion write stamps result_expires_at even when the actor returned None for a non-None result type (the sweep's AND result IS NOT NULL guard never touches such a row), so a past stamp cannot distinguish a swept result from a row that never stored one, and the message asserts only that: the TTL elapsed at T and no result is stored (swept after expiry, or the actor stored none). The in-memory backend's read view applies the identical rule, so tests see the same failure mode on both backends. A retried job cannot inherit a stale receipt: retry_job re-pends with result_expires_at = NULL and the terminal write recomputes from the actor's TTL at the new completion.

TaskQ-built client pools carry a 10 s per-query bound

Unreleased. Silent; a black-holed database now raises instead of parking the client forever.

Every pool TaskQ builds for a client (the DSN pool at TaskQ.open() and the pg_provider sugar's factory pools) now carries an asyncpg command_timeout of 10 s. Previously a black-holed Postgres (packets dropped, no RST) parked the client's first enqueue/get/cancel indefinitely: client processes arm no watchdogs, so nothing converted the hang into a crash. The query now fails with TimeoutError after 10 s.

The trade-off cuts the other way for a slow-but-alive database: a legitimate client query exceeding 10 s is now cancelled, so size client-side expectations (and any outer retry) accordingly. The bound is deliberately above the enqueue-path lock budgets (5 s defaults); server-side lock_timeout still fires first, so a lock refusal surfaces as the typed MaxPendingLockTimeoutError / UniqueForLockTimeoutError / IdempotencyKeyLockTimeoutError rather than a bare TimeoutError. Pools you supply yourself (pool= / pool_factory=) stay caller-owned: their timeouts are your choice.

The queue-depth alert fires on oldest-pending age, not raw depth

Unreleased. Operator-visible: the bundled TaskQQueueDepthHigh alert's expression changed; any override of it must be re-expressed.

The bundled Prometheus rule TaskQQueueDepthHigh (src/taskq/contrib/prometheus/rules.yaml) previously fired on raw depth; taskq_queue_depth > 1000; it now fires on the oldest pending job's age, max by (actor, queue) (taskq_jobs_oldest_pending_age_seconds) > 900, sustained for 5 minutes. Depth alone is ambiguous: a deep queue that drains is healthy throughput, and a queue-summed number cannot tell a busy shared queue apart from an actor nobody consumes: a misrouted actor produces no refusal and no failed job, its rows pile up pending while every probe stays green, and its age series rises without bound while its queue-mates stay flat. The per-(actor, queue) attribution names exactly whose TASKQ_QUEUES coverage to check. taskq_queue_depth is still emitted, so dashboards and any rules you wrote against it keep working, but if you overrode the bundled alert's threshold, re-express the override in seconds of oldest-pending age.

---

Bounded inputs

New upper bounds reject input that was previously accepted. Each is a clean error, not a crash, but a caller that exceeded the bound will now fail.

Surface	New bound	Failure
taskq queues set-max-concurrent --max-concurrent	>= 1 (was >= 0)	typer argument-parsing error, exit 2
taskq.worker.queue_ops.set_queue_max_concurrent	>= 1 or None (was >= 0)	ValueError
SubJobEnqueuer.enqueue_batch(items)	at least 1 item	ValueError
BatchFilter(limit=...)	<= 500 (default 100, 0 still means "no rows")	ValueError at construction
Admin /jobs, /jobs/count, /history (status)	values outside the closed status set	HTTP 400
Admin /jobs (tags)	255 chars each (no item-count cap)	HTTP 400
JobFilter: queue, actor, identity_key, tags	no NUL bytes	ValueError in __post_init__
ScheduleCreateArgs: actor, name, timezone, payload_factory, identity_key	no NUL bytes	ValueError in __post_init__
Admin text filters: actor, queue, search, identity_key, fairness_key, tags	no NUL bytes	HTTP 400
ScheduleCreateArgs.dst_strategy	a value in taskq.cron.DST_STRATEGIES	ValueError in __post_init__
BatchHandle.status() / wait_for_batch(): schema	schema-identifier regex	ValueError
taskq.testing.pg: schema	schema-identifier regex	ValueError

Notes:

• max_concurrent=0. 0 previously passed both the CLI and the ops-layer guard and then hit the table's CHECK (max_concurrent IS NULL OR max_concurrent >= 1), producing a raw asyncpg CheckViolationError traceback, so scripts passing 0 were already failing, just messily. NULL (via --clear) is the uncapped state; an emergency drain to 0 belongs to the per-actor taskq actor-config set --max-concurrent 0, which still accepts it.
• enqueue_batch([]). Previously returned [] silently when no connection was in play, while JobsClient.enqueue_batch already raised. Guard the call site if your item list can legitimately be empty.
• Admin status filter. Values are deduplicated in first-occurrence order, so a request repeating a status still succeeds and returns the same rows: only requests containing a value outside the closed status set are rejected, and the dedup alone bounds the list to the set's eight members, however long the request is.
• NUL bytes. Previously these reached Postgres and came back as an opaque asyncpg 22021: a 500 from the admin routes. The values were never storable; the rejection surfaces the fault at the boundary instead.
• dst_strategy. An unrecognized strategy previously constructed fine and took the default branch at cron-tick time. The known set is newly exported as taskq.cron.DST_STRATEGIES. A schedule that already declares firstof/allof also changes firing behaviour; see Breaking API changes.
• SQL interpolation guards. The schema parameter of the batch public API (BatchHandle.status(), wait_for_batch()) was previously interpolated without validation, a SQL-injection surface; it is now validated against the canonical schema-identifier regex before interpolation, and taskq.testing.pg follows the same rule.

---

Configuration that no longer loads

WorkerSettings gained several load-time validators. A configuration containing any of the values below stops the worker from starting; get a settings-load error instead of the opaque mid-startup failure it used to produce. Check these before rolling out, not during.

Setting	Now rejected	Previously
schema_name	longer than 63 characters	Postgres truncated it silently; Redis channel templates used the full string, so the two stores diverged
workgroup_instance	not a valid UUID	raw ValueError mid-registration
worker_label	contains a NUL	opaque asyncpg 22021 CharacterNotInRepertoireError at startup
queues[*]	outside the queue-name charset	accepted; jobs stranded on a queue no worker drains

The queue-name charset is: letters, digits, _, ., -, with the first character a letter or _.

Workgroup TOML configs follow the same load-time rule through WorkgroupConfig.from_toml:

• [[workers]] name is capped at 43 characters. The supervisor binds a health socket for every child at /tmp/taskq_health_<name>_<uuid>.sock (60 fixed chars around the name, and the path must fit every supported platform's AF_UNIX sun_path budget (104 bytes on macOS/BSD, 108 on Linux, NUL included). A name of 44-64 chars previously loaded and then died at child spawn with OSError: AF_UNIX path too long, restart-looping against the burst budget until give-up; longer names still do.
• queues = [] (on a worker or in [defaults]) is refused. A worker that consumes no queue dispatches nothing; omit the key to inherit [defaults].queues, or ["default"] when no default is set.

Both raise ValueError at config load, from taskq workgroup start / taskq workgroup validate before any process spawns.

Two further invariants apply only when watchdog_enabled=True:

• watchdog_loop_lag_budget + heartbeat_interval must be < lock_lease. A stalled event loop has to die before its leases expire, otherwise the leader sweep reclaims live jobs' locks mid-stall and the worker wakes to find its work reassigned.
• watchdog_loop_lag_budget must be > watchdog_check_interval. The lag detector samples once per check interval and schedules the beat it measures from the same poll, so a healthy loop's observed lag is roughly the check interval by construction. A budget at or below the sampling period trips on health rather than on stalls; measured, a budget of 1.0 against the default 1.0 s check interval force-exits an idle worker on its first armed poll.

If you tune either watchdog knob, move both together and keep the lag budget comfortably inside lock_lease.

These raise dotenvmodel's ValidationError / MultipleValidationErrors, not ValueError. The cross-field invariants (the lock_lease failed-beat-cascade check, since sized with the command timeouts, see the unreleased note below, and the grace-budget checks) previously raised ValueError, so callers that catch ValueError around WorkerSettings.load*() will no longer catch them: catch DotEnvModelError (the common base) to cover both single and aggregate cases, or ValidationError when at most one invariant can fire. ConstraintViolationError (field validators) was already not a ValueError; field-level validation (prune_retention_*, default_start_to_close, log_format, etc.) already raised it and is unaffected. See the dotenvmodel 1.x note under Breaking API changes for the dependency-level changes.

---

Structured-log event rename: state_change → state-change

Unreleased. Breaking for log pipelines. state_change shipped in v0.2.2.

The job state-transition event is now emitted as state-change, from every backend that logs a transition; one name now covers the Postgres and in-memory paths alike. Nothing fails; your saved searches and alert rules stop matching. Update any query, dashboard panel, or alert rule that selects on state_change before upgrading, or you lose visibility silently.

Three other event names were kebab-cased in the same pass, batch_streaming_enqueued, pg_credential_refresh_failed, and cancel_where_notify_failed, but none of them ever appeared in a release, so no consumer can be matching the old spellings.

---

Trailing newlines in queue names, tags, and keyed keys

The queue-name, tag, and keyed rate-limit key regexes are now anchored \A...\Z instead of ^...$. Python's $ also matches immediately before a trailing newline, so "default\n", "mytag\n", and "key\n" all satisfied the old patterns and are now rejected with a ValueError.

The realistic sources of a stray trailing newline are a shell $(...) substitution, a value read from a file, and an unstripped environment variable:

# This used to pass validation and now raises ValueError:
queue = pathlib.Path("/etc/taskq/queue").read_text()  # "default\n"

# Strip at the boundary:
queue = pathlib.Path("/etc/taskq/queue").read_text().strip()

A queue name with a trailing newline was never actually usable; jobs enqueued onto it were stranded, since no worker's queue = ANY($1) ever matched. The new error surfaces a fault that was previously silent.

Separately, queue names are now validated at both the enqueue and the actor-declaration chokepoints. The QueueName annotation is inert at runtime (its AfterValidator only fires inside pydantic model validation), so a typo'd queue name previously sailed through. It now raises at decoration time, import time in the common case, so a typo that used to strand jobs quietly will now stop your process from starting.

---

The default termination grace period rose from 75 seconds to 85

Unreleased. Action required before upgrading for any deployment that set its platform's stop grace against the old 75 s number: Kubernetes terminationGracePeriodSeconds, an Azure Container Apps or ECS stop timeout, Docker Compose stop_grace_period, or systemd TimeoutStopSec.

TASKQ_TERMINATION_GRACE_PERIOD (the worker's own SIGTERM-to-forced-exit budget) now defaults to 85 s, up from 75 s. Nothing rejects the old value: an explicitly set number keeps working, and a deployment that never set the variable picks the new default up on its next image roll.

Why. Two provider-release closes joined the modelled teardown tail. The bounded-close unwind that runs at the end of every shutdown (see taskq._close.worst_case_teardown_tail) went from 6 sequential closes (~32 s) to 8 (~42 s): the credential providers a managed-identity deployment resolves (one Postgres, one Redis) each close once, after every resource built through them. At the default phase graces (30 s + 10 s) the modelled worst case rose from 72 s to 82 s, so the old 75 s default no longer covered it and the new 85 s default does.

The action. A platform grace pinned to 75 s now sits ~10 s below what the worker needs, so the platform SIGKILLs the worker mid-teardown, right in the provider-release step that motivated the change. The kill truncates in-flight terminal writes, the jobs' leases expire, and the leader's crash-reclaim sweep re-runs the work later: the shutdown degrades to the crash-reclaim path instead of finishing cleanly. Before upgrading:

• raise the platform grace above the worker's worst case: at the shipped defaults, 85 s is the safe floor; or
• compute it from your settings: TASKQ_CANCELLATION_GRACE_PERIOD + TASKQ_CLEANUP_GRACE_PERIOD + ~42 s. The formula and its caveats (the conditional per-slot pool, the ~87 s sibling-crash path) are in deployment.md's fail-closed grace notes;
• and keep the platform grace at or above TASKQ_TERMINATION_GRACE_PERIOD itself, so the worker's own watchdog, not the platform, owns the shutdown deadline.

Two surfaces help confirm the sizing. The worker logs shutdown-budget-exceeds-termination-grace at startup when its own settings fall short of the model, but that warning compares settings to settings: the platform's grace is invisible to the worker, so a manifest still pinned to 75 s produces no warning anywhere and can only be caught by auditing the deployment config against the formula above. The settings tables in configuration.md and workers.md carry the new default.

---

Unreleased features

The features and notes below land with the next release. The canonical release notes for every release are generated by release-please from the repository's conventional commits (CHANGELOG.md is generator-owned, and a hand-written block there is invisible to the release-notes pipeline), is why the pending notes live in this guide. At release time the generated changelog becomes the authoritative record and these notes age out.

Jobs and batches

• JobsClient.cancel_where(filter, reason): bulk cancel all jobs matching a JobFilter in a single set-based operation. Pending/scheduled jobs go straight to terminal cancelled; running jobs get cooperative cancel (cancel_phase=1). Returns BulkCancelResult with counts and affected IDs. Empty filters are rejected with EmptyFilterError unless allow_empty_filter=True is passed. BulkCancelResult and EmptyFilterError are exported from the taskq top level.
• Batch failure policies (AbortBatchAfter): an opt-in failure_policy parameter on enqueue_batch() / enqueue_batch_streaming() creates a batches row and drives abort-on-consecutive-failure semantics via the apply_batch_terminal_outcome hook. When the threshold is reached the batch is aborted: pending/scheduled child jobs are cancelled and the batch row is set to aborted.
• Batch finalizer (transactional enqueue with batch): a finalizer parameter on enqueue_batch() / enqueue_batch_streaming() enqueues a finalizer job alongside the batch in the same transaction. The finalizer is NOT stamped with batch_id (deadlock prevention); wait_for_batch automatically excludes it from counts via the batch row's finalizer_job_id.
• Batch discovery (list_batches, BatchSummary): JobsClient.list_batches(BatchFilter) returns BatchSummary objects with live job-count aggregates. BatchFilter carries only batch-relevant fields (queue, active, batch_id, limit).
• enqueue_batch_streaming for unbounded iterables: accepts an Iterable[EnqueueItem] (including generators) and inserts in chunks of chunk_size (1-1000). All items share the same batch_id.
• wait_for_batch with expect_at_least, on_empty, exclude_job_id: expect_at_least raises EmptyBatchError when fewer than the expected number of jobs are present; on_empty controls behaviour when zero jobs and no batches row exist ("error" raises, "ok" returns empty status); exclude_job_id omits a specific job from counts (defaults to the batch row's finalizer_job_id).
• Backend protocol batch methods (10 new methods): enqueue_batch_atomic, create_batch, increment_batch_failures, reset_batch_failures, abort_batch, complete_batch, get_batch, list_batches, count_batch_non_terminal, prune_old_batches.
• Batches table migration (01.00.05_01): adds the batches table with columns for status tracking, failure counters, finalizer linkage, and batch-level metadata.

Sub-job enqueueing

• SubJobEnqueuer.enqueue() now accepts tags, inherit_tags, schedule_to_close, and start_to_close parameters. Sub-jobs inherit the parent job's tags by default (inherit_tags=True); pass inherit_tags=False to suppress inheritance for a specific sub-job.

Managed identities and connections

• Connection hook points for managed-identity / BYO connections: WorkerConnections dataclass with per-role pre-constructed resources (caller-owned) or zero-arg async factories (TaskQ-owned) for the worker's three PG pools, notify/leader dedicated connections, and Redis client. worker_main(..., connections=...) and open_worker_deps(..., connections=...) accept it; fields left None fall back to DSN construction. PoolFactory, ConnFactory, RedisFactory type aliases are exported from the taskq top level.
• Vendor-neutral credential provider abstraction (taskq.auth): PgCredentialProvider and RedisCredentialProvider async Protocols with reusable make_pg_pool_factory, make_dedicated_conn_factory, make_redis_client_factory builders. Any provider implementing the Protocols gets all factory builders for free. The PG factories pass the credential to asyncpg as user= / password= keyword arguments (which take precedence over both DSN userinfo and DSN query parameters), so the token never appears in the DSN string; enrich_pg_dsn remains as the string-helper variant (writes the credential into DSN userinfo; adds sslmode=require only when the DSN has no explicit sslmode, verify-full is never downgraded). All four helpers are exported from the taskq top level as well as taskq.auth.
• Per-worker Postgres credential providers in workgroup configs: a pg_credential_provider = "module:attr" key on a [[workers]] entry (or the pg_credential_provider= field on WorkerSpec) is forwarded to that worker's child command line as --pg-credential-provider, so two workers in one workgroup can use different providers.
• taskq[aad] extra: taskq.aad module with Microsoft Entra ID providers (EntraIdProvider, EntraIdPgProvider, EntraIdRedisProvider) backed by azure.identity.aio (the extra includes aiohttp, required by the async credentials). Providers constructed with credential=None lazily create one DefaultAzureCredential and reuse it; sync azure.identity credentials are supported and offloaded to a thread. See managed-identities.md.
• taskq[aws] extra: taskq.aws module with RdsIamProvider for AWS IAM RDS Postgres authentication, backed by boto3.
• taskq[vault] extra: taskq.vault module with VaultDynamicDbProvider for HashiCorp Vault database secrets engine dynamic credentials, backed by hvac.
• TaskQ stream hooks: pg_conn_factory and listen_conn parameters for the LISTEN/NOTIFY transport in TaskQ.stream(), so pool-only / AAD deployments can stream without a DSN. stream() now uses contextlib.aclosing to ensure the inner generator's finally (conn close) runs promptly on early return.
• migrate.apply_pending_locked hooks: conn (caller-owned) and conn_factory (TaskQ-owned) parameters replace the DSN-only path.
• Credential hot-reload (SIGHUP / interval / programmatic): hot-swaps every factory-backed PG pool, dedicated connection, and Redis client with freshly-built replacements (each factory fetches a fresh credential). Triggers: SIGHUP; TASKQ_RELOAD_INTERVAL (seconds, unset by default) for periodic reloads with no external signal: the only rotation path on Windows; and WorkerDeps.request_reload() / reload_credentials(deps) for embedders. Each factory call is bounded by TASKQ_RELOAD_FACTORY_TIMEOUT (default 30 s). The swap is atomic: the old pool stops serving new acquisitions immediately and is closed in the background with a bounded drain (default 5 s), then terminated; in-flight actor that outlives the drain sees its next acquire fail and the job retries on the new pool. DI-injected db: asyncpg.Pool actors resolve the new pool (LOOP-scope cache refresh) and progress flushing follows the swap. A SIGHUP arriving mid-reload (success or failure) triggers exactly one follow-up reload; reloads are skipped while shutdown is in progress. Each resource reloads independently; one factory failure is logged and does not abort the rest; the credentials-reloaded log line's failed field reports any resource that didn't rotate. Caller-owned resources are not swapped.
• NOTIFY listener resilience: the reconnect loop rebuilds a dropped LISTEN connection through the user-supplied notify_conn_factory (or the DSN closure it was opened with) instead of a stale/absent DSN. A caller-owned notify_conn that drops disables the listener (poll-based dispatch fallback) instead of crashing the worker.
• Ownership-contract enforcement: caller-owned pools/connections/Redis clients are never closed by TaskQ (including shutdown paths). A caller-owned leader_conn with no leader_conn_factory and no pg_dsn_direct is a startup ValueError (no rebuild path). TaskQ-owned dedicated connections (DSN- or factory-built) get TCP keepalive.
• taskq.worker re-exports WorkerConnections and reload_credentials (lazy, alongside the existing WorkerDeps / open_worker_deps).

Actor and retry hooks

• ErrorReporter Protocol for vendor-neutral terminal failure routing (Sentry, Datadog, DLQ) with NullErrorReporter default and a taskq.error_reporter.failures OTel counter. report() takes (job, exception) (the same argument order as on_retry_exhausted), and is guarded by the error_reporter_timeout setting (default 3 s). The registration's scope is validated at worker startup: a reporter registered at TRANSIENT scope (which the hook, running after the actor's scope closed, could never resolve) fails the boot with a DIError naming the scope and the allowed ones, where dispatch time previously raised per job. A dispatch that never bootstrapped (in-process runners) degrades instead: the hook is skipped behind a window-gated error-reporter-defect WARNING, and a provider resolving to a value without the reporter protocol warns the same way.
• retry_classifier hook on @actor for exception-instance-level retry classification (inspect attributes like HTTP status codes, return RetryOverride to refine kind/delay per occurrence). Non-RetryOverride returns are caught and logged rather than crashing, and the hook is skipped for non_retryable_exceptions and PayloadValidationError, matching the documented contract. RetryOverride and RetryClassifierHook are exported from the taskq top level.
• on_success hook on @actor for success callbacks (mirrors on_retry_exhausted with timeout guard).
• start_to_close per-attempt execution timeout with precedence chain: per-enqueue > @actor(start_to_close=...) > TASKQ_DEFAULT_START_TO_CLOSE worker fallback.
• KeyedReservationRef for dynamic per-key (session/tenant) concurrency caps computed from job payload at dispatch time.
• max_keyed_reservations setting to guard against unbounded keyed reservation growth.

Cron, queries, and admin

• Cron schedule ownership model (disabled_by, migration 01.00.19_02): cron_schedules gains a nullable disabled_by text column recording WHO disabled a schedule: 'auto' (the cron loop's failure-count auto-disable), 'operator' (schedule handle disable(), the CLI, the admin UI, actor deregistration), or NULL (enabled, or a row an old pod wrote during a mixed-version deploy; see below). A CHECK constraint enforces the two values. The worker's startup registration pass now re-enables a schedule auto-disabled on a transient failure blip only when the re-declaring spec is code-owned (CronScheduleSpec.owner="code", the default) and declares enabled=True; operator-disabled rows stay disabled across restarts, the guarantee the create-only registration design always intended but could not express. Upgrades are safe by construction: the follow-up backfill (migration 01.00.19_05) stamps every disabled row that predates it 'operator', so every schedule a human or auto-disable left disabled before the upgrade stays disabled until the operator acts -- the old behavior, unchanged. The residual NULL reading after the backfill covers the mixed-version deploy window: the previous release's failure UPDATE cannot name the marker, so an old pod's auto-disable during a rolling deploy lands enabled=false, disabled_by=NULL, and the boot revert also recovers a NULL-marked row carrying that write's fingerprint (consecutive_failures at or past the threshold, last_fire_error set) -- without the fix, such a row matched nothing and the schedule stayed disabled until a human re-enabled it (issue #460). Fresh installs get the column from the migration chain. The boot is not the only moment the old write can land: during the roll the previous release can hold (or win) leadership after every new pod has booted, and its cron tick keeps firing schedules, so its unmarked auto-disable can postdate every boot pass. Each new-release leader therefore re-runs the same ownership predicates over the code's declared specs when it assumes leadership (the cron-schedule-auto-disable-reverted-at-takeover log line): any old leader's disable is reverted the first time a new pod leads after it landed, not only at the next full restart. See cron.md.
• name and identity_key fields on CronScheduleSpec for per-property cron schedules and cron↔on-demand dedup.
• JobSortField enum and JobFilter.order_by for "latest run by business key" queries.
• Admin UI security settings: admin_actions_enabled and admin_ui_require_auth. The admin UI fails closed by default in non-dev environments: admin_ui_require_auth=True (default) raises RuntimeError at startup when no auth_dependency is configured, with explicit opt-out TASKQ_ADMIN_UI_REQUIRE_AUTH=false; the health endpoints follow the same fail-closed pattern: health_require_token=True (default) raises RuntimeError in non-dev when health_token is empty (TASKQ_HEALTH_TOKEN / TASKQ_HEALTH_REQUIRE_TOKEN=false to opt out). Destructive admin actions (run-schedule, retry-job, cancel-job) are gated behind admin_actions_enabled (default False); POST /schedules/{id}/run checks the schedule's enabled flag and has per-process cooldown rate limiting, and its cron payload_factory error redirect uses a generic error code instead of reflecting exception text.
• TASKQ_ADMIN_UI_SECURE_COOKIES (admin_ui_secure_cookies, default True): sets the Secure flag on the admin UI's CSRF cookie. The flag was previously derived from request.url.scheme, so behind a TLS-terminating edge (Azure Application Gateway, App Service) the app saw plain http and silently dropped Secure on exactly the deployments that need it; the session cookie, which already used a configured flag, kept it. Set it to False only for local http dev, where a Secure cookie is rejected by the browser and the UI stops working. A one-shot admin-ui-cookie-scheme-mismatch warning fires when the configured value contradicts the observed scheme; run uvicorn with --proxy-headers so X-Forwarded-Proto is honoured.
• TASKQ_ADMIN_UI_FRAME_ANCESTORS (admin_ui_frame_ancestors, default none): who may frame admin pages. Every admin response now carries Content-Security-Policy: frame-ancestors '<value>' and the legacy X-Frame-Options (DENY for none, SAMEORIGIN for self). Admin pages can no longer be iframed: a host application that embeds the admin UI in its own dashboard must set TASKQ_ADMIN_UI_FRAME_ANCESTORS=self or the frame renders blank. Only none and self are accepted; anything else fails at settings construction rather than silently emitting no header. CSRF is no defence against UI redress: the framed page is the real, authenticated, same-origin page, so a tricked click carries a valid token.
• SSO / SAML auth for admin UI: OIDC backend (taskq[oidc]), PKCE flow, JWKS validation, signed-cookie sessions; SAML backend (taskq[saml]): python3-saml, SP metadata, attribute extraction; shared AuthBundle/IdentityClaims abstraction (both backends use the same session handling and group/role allowlist); token_auth() helper for machine-to-machine bearer-token auth; TASKQ_SSO_BACKEND=none/oidc/saml CLI integration for standalone taskq ui serve. The taskq[oidc] extra no longer installs httpx; its authlib floor is now >=1.8.0 (authlib 1.8.0's httpx_client integration is httpx2-first, the direct OIDC calls use httpx2, and nothing under src/taskq imports httpx). OIDCSettings/SAMLSettings are separate DotEnvConfig classes with prefix scoping.

Testing and documentation

• Consolidated testing guide (testing.md).

Fixes and internal notes

• _di/solver.py debug log now reports the real cache_hit value instead of a hardcoded False.
• worker/_leader_sweeps.py logs a warning on invalid schema and includes error detail in exception handlers.
• worker/notify.py logs debug on NOTIFY payload parse failures.
• Test containers are shared singletons: one Postgres and one Dragonfly container per pytest invocation, shared across all xdist workers (filelock refcount, stale-leftover sweep) with per-module database and per-test schema isolation preserved; full suite ~152 s vs the ~226-240 s baseline.
• Docker/testcontainers calls in tests run off the event loop (asyncio.to_thread), so docker-py's blocking HTTP round-trips no longer stall the event loop mid-test.
• Behavioral timing tests assert in a single clock domain (one statement reads the server clock and the row together), so application/database clock divergence cannot corrupt an assertion; liveness freshness is bounded by the missed-at-most-one-tick contract.

Bulk enqueues partition max_pending admission per actor

Unreleased. Breaking for handlers of bulk cap refusals.

enqueue_batch() / enqueue_batch_fast() (and the chunked arm of enqueue_batch_streaming()) now admit the within-cap actors' items and refuse only the over-cap actors' items as whole groups, raising BatchMaxPendingExceededError after the admitted items are stored. Previously one capped actor aborted the whole call with MaxPendingExceededError and nothing enqueued.

• except MaxPendingExceededError no longer catches bulk cap refusals, the new error is deliberately not its subclass. Catch it explicitly; it names each refused actor (refusals), the refused item indices (refused_indices), and the admitted count (admitted_count).
• except BackpressureError now catches an error under which part of the batch is stored: a handler that blindly retries the whole batch duplicates the admitted items. Consult admitted_count / refused_indices and retry only the refused items, or rely on idempotency_keys.
• Durability is path-dependent: committed when the call owned its transaction; uncommitted on a caller-supplied open transaction (that transaction decides); on the streaming no-connection path a refusal surfaces after a durably committed chunk prefix (indices are stream-global). The atomic path keeps the legacy all-or-nothing contract and still raises plain MaxPendingExceededError.

taskq.cron.consecutive_failures is relabeled and bounded

Unreleased. Breaking for dashboards and alert rules keyed on the old label.

The metric's dimension is now the schedule row's actor (bounded to the first 100 distinct names a worker process sees; later names collapse onto the fixed _other_ value) instead of the per-schedule UUID. Dashboards grouping by schedule_id lose their series on upgrade. Per-schedule attribution lives on the cron fired / cron fire failed log lines and the cron fire span's taskq.cron_schedule_id attribute. The per-actor balance is reconciled against the database every tick; each tick re-derives the per-actor sum over the whole cron_schedules.consecutive_failures table, so disable, re-enable and delete actions taken in any process self-correct on the next tick with due work rather than stranding residue, and the value returns to zero once no schedule is failing. cron_schedules.consecutive_failures and the logs remain the authoritative per-schedule record; alert on taskq.cron.disabled_schedules > 0 for the auto-disabled condition.

The dead-connection retry no longer re-runs committed writes

Unreleased. Silent unless your database kills connections mid-flight; when it does, the behavior you get is the one you already wanted.

The retry guard behind the enqueue paths and the bulk-cancel drain (_with_fresh_connection_retry) covered the whole acquire-run-release cycle. A pooled connection whose server died between the op's last acknowledged statement and the pool's release-time reset raised asyncpg's InternalClientError from the RELEASE: the old wrapper read that as dead-on-acquire and re-ran the op with the same arguments (the pooled-connection re-run bug). Because the enqueue table's id is the primary key and the op carries its id, that re-run could not land a second row: it raised UniqueViolationError for an enqueue that had already committed and would run (observed live under an old-contract emulation: 11 UniqueViolations across 400 jittered kills). The harm was that error returned for work that SUCCEEDED, and the invitation it created: a caller that retried on the error issued a fresh-id enqueue, and THAT row landed, which is the route that actually runs the job twice. Two mechanisms now split the concern:

• The guard's checkout bounds the release (pool.release(conn, timeout=...), 5 s) and never raises a release failure: an op whose work committed gets its result back (one pool-release-failed WARNING is logged instead). The bound also un-wedges the hang the old path could hit: against a silently dead server (no FATAL, no FIN, a frozen or partitioned endpoint) the unbounded reset parked the caller's task AND pool.close() forever; it now times out after 5 s, asyncpg terminates the connection, and the pool reopens one on the next acquire.
• The retry is refused once the op marked a write durable (the INSERT's acknowledgement on the autocommit arms, the transaction COMMIT's on the batch/COPY/cancel arms). A connection that dies between the write and a LATER statement of the same attempt surfaces its error instead of re-issuing the write; the ambiguous outcome is handed to the caller rather than papered over with a re-run that conflicts with the committed row. A connection poisoned BEFORE the first statement still costs exactly one transparent retry: nothing was sent, so the retry cannot conflict with anything.

For context, the peer doctrine: procrastinate scopes its own dead-connection retry to the LISTEN connection only (psycopg_connector.py:313-353) and never auto-retries query paths. Pre-fix TaskQ sat at the opposite pole: any InternalClientError, including one raised after an acknowledged write, was retried. Post-fix TaskQ retries only the provably-nothing-sent case and surfaces ambiguous outcomes instead of re-running them: closer to the peer's conservatism than base, deliberately not as absolute, because the first-statement local failure it still absorbs is provably duplicate-free (nothing reached the server).

A mid-QUERY kill is unchanged: it raises ConnectionDoesNotExistError (a Postgres error, not retried by this wrapper) and whether the statement committed before the server died is unknowable from the client; idempotency keys remain the dedup channel for the retry YOU choose to issue in that case.

Worker pools send no per-connection GUCs in the startup packet

Unreleased. Breaking only in the sense that a worker that failed to boot behind a strict pooler now boots, and a client-side jit = off that silently rode every dispatcher connection is gone; set it server-side if you want it.

Every pool the worker builds opens with no server_settings=. asyncpg rides each entry in the Postgres startup packet, and a pooler that rejects unknown startup parameters (PgBouncer: unsupported startup parameter: jit) refuses the connect before authentication, so with a single TASKQ_PG_DSN pointing at the pooler, the eagerly-opened boot pools never came up and the error named neither the parameter nor the pool (the PgBouncer jit startup-parameter rejection). The dispatcher pool's jit = off entry was a guard whose measured win had already moved into the dispatch statement itself (the depth oracle passes with JIT enabled on a plain connection); a per-claim SET LOCAL jit = off is not a replacement: the claim runs in autocommit, so there is no transaction to scope it to. Operators who want the guard should set it where no startup packet is involved:

ALTER ROLE taskq SET jit = off;

or append ?options=-c%20jit%3Doff to the DSN; see ops.md. The per-slot transaction pool's inherited search_path/role remain startup parameters by design: they are session state a LOOP-scope connection declared, they must survive the pool's release-time RESET ALL (startup-packet values do; a post-connect SET does not), and the slot pool always rides the direct DSN. If your direct DSN itself routes through a pooler and you register a LOOP-scope connection, add ignore_startup_parameters=search_path,role (or whichever parameters you actually inherit) to the pooler's config.

The TaskQ client reads the .env cascade for its lock budgets and schema default

Unreleased. Silent widening; a constructor failure class is gone.

The TaskQ client's lock-budget probe (TASKQ_MAX_PENDING_LOCK_TIMEOUT_MS and its two siblings) previously read the process environment only, so a widening set in .env reached the worker's server-side budgets but not the client pool's derived command_timeout: the two sides silently disagreed (the lock-budget probe bug). The probe now resolves through the same layers the worker's TaskQSettings.load() reads (process environment, then the .env cascade, honoring DOTENV_OVERRIDE / DOTENV_READ_DOTFILES / DOTENV_READ_ENVIRON), never writing into os.environ.

TaskQ(...)'s default schema resolution also changed twice, once when this release's connection stack landed and once now:

• Previously (≤ v0.2.2): the default was the literal "taskq"; no environment or .env value was read at all.
• At this release's first connection-stack landings: the constructor ran a full validating TaskQSettings.load(), so TASKQ_SCHEMA_NAME was honored (process env and .env), but any OTHER malformed TASKQ_* value (e.g. TASKQ_ADMIN_PORT=not-a-port, a setting the client never reads) raised in an embedder's constructor.
• Now: the value resolves through the same cascade, then validates ALONE: a malformed TASKQ_SCHEMA_NAME still raises (it is the client's own field, reaching raw SQL; the worker's load fails the same value), and an unrelated malformed setting cannot break TaskQ() construction.

An explicit schema= argument keeps winning over every layer, so embedders that pass their loaded schema explicitly are unaffected.

The lock_lease invariant accounts for heartbeat command timeouts

Unreleased. Tightened validation: a lease between the old edge and the new floor stops loading.

The load-time invariant on WorkerSettings.lock_lease was lock_lease >= 4 * heartbeat_interval. It ignored the per-command timeouts a heartbeat tick pays: a failed beat's cycle spans the interval plus one heartbeat_command_timeout budget (the tick's command sequence AND its bounded rollback-or-close teardown sharing that ONE budget, made true by enforcement by the per-tick command budget), and the isolate decision lands on the (max_heartbeat_failures + 1)-th consecutive failed beat, so the lease could lapse before the isolate decision even at the defaults under contention.

The validator now requires

lock_lease >= max(heartbeat_interval, heartbeat_command_timeout) +
              (max_heartbeat_failures + 1) *
              (heartbeat_interval + heartbeat_command_timeout)

, the same sizing the lease-renewal threshold derives in taskq.worker.heartbeat._lease_renewal_threshold, so the threshold's floor can never exceed the lease it guards. The max(...) term is the LAST good beat's tail - from its mid-tick renewal point to the next tick's start - the span the failed cycles do not cover. At the defaults this is 10 + 4 * (10 + 2) = 58 against the 60 s lease: the shipped defaults still load. A config tightened to the old edge (lock_lease at exactly 4 * heartbeat_interval, or any lease below the new floor) now stops loading with the new message naming the command-timeout term: raise TASKQ_LOCK_LEASE past the floor, or lower TASKQ_HEARTBEAT_COMMAND_TIMEOUT / TASKQ_MAX_HEARTBEAT_FAILURES.

The bootstrap's ensure-slots pass is bounded

Unreleased. Robustness fix; a hang class is gone.

The worker bootstrap's per-reservation ensure pass called ConcurrencyReservation.ensure_slots with no bound; under dispatcher-pool starvation the bare pool.acquire waited forever, observed once as a 300 s CI hang). The pass now carries the same dispatcher_command_timeout budget every sibling bootstrap await has, and an acquire lapse logs the typed ensure_slots_timeout event (distinct from the generic ensure_slots_failed) and continues booting, the next restart re-runs the idempotent materialisation. ensure_slots, slot_rows_exist, and ConcurrencyReservation.acquire grew an optional timeout= forward to asyncpg.Pool.acquire for callers with a budget in hand (default None keeps the previous unbounded-acquire behaviour for callers that wrap their own).

The terminal-write fence survives the attempt ceiling

Unreleased. Correctness fix with one additive pre migration (01.00.18_02_pre_claim_epoch.sql; apply it before rolling the code, as with every pre file).

At the attempt ceiling the terminal writes could record the wrong outcome. The displayed attempt counter is stamped by the claim with a saturating increment (LEAST(attempt + 1, 32767)), so a row parked at the smallint ceiling (retry_kind='indefinite' climbs there after enough retries) claims at 32767 again and again. A reclaim plus a redispatch to the same worker then gave the stale execution and the live one the same (status, locked_by_worker, attempt) fence: the stale execution's mark_succeeded won, the job recorded the stale result, and the live execution's write read WorkerOwnershipMismatch. The displayed counter keeps its saturating semantics on purpose (the clamped row must still run, and its budget arms must still land), so the fix adds a second, non-saturating fence column instead of touching it.

Every row now carries claim_epoch bigint NOT NULL DEFAULT 0 (jobs and jobs_archive; ADD COLUMN with a constant default, metadata only, no table rewrite). The dispatch claim bumps it by exactly 1 per claim, and every terminal/ownership write that already fenced on attempt gained the conjunct claim_epoch = <the writer's own claim view>: mark_succeeded, mark_failed, mark_cancelled, mark_retry, mark_snoozed, both mark_retry_after variants, and mark_interrupted, every arm of the multi-arm arbiters included.

The invariant itself lives in the migration file. The operational shape: the value the writer presents is the claim_epoch of the JobRow its own dispatch returned (and of the actor context, ctx.claim_epoch, which the shutdown release reads); a caller that cannot present one (a None bind) no-ops through the same machinery the attempt fence uses, the same cannot-prove-it doctrine both fences apply. The in-memory testing backend mirrors the semantics exactly. No operator action is needed beyond applying the migration first; the column is never read by user-facing surfaces and the displayed attempt counter is unchanged.

The per-attempt due-time stamp (job_attempts.due_at)

Unreleased. Observability addition with one additive pre migration (01.00.20_04_pre_attempt_due_at.sql; apply it before rolling the code, as with every pre file). Nothing breaks; no consumer of the old shape changes.

Every job_attempts row (and its job_attempts_archive twin) now carries due_at timestamptz: the jobs.scheduled_at the attempt's claim took the row against — the due time the attempt was waiting for when it was dispatched. Before this, only the FIRST attempt's wait was reconstructable from the row: jobs.scheduled_at is overwritten by every retry's reschedule (mark_retry/mark_retry_after and the reclaim re-pends stamp it for the NEXT attempt), so attempts 2..N had no surviving due time and a retry chain's wait spans were unreconstructable from the ledger.

The stamp is written by every attempt-row writer (the terminal mark_* statements, the reclaim sweeps' batched INSERTs, the isolate write): each sources due_at from the job row's scheduled_at as it stood at that attempt's claim. On the arms that reschedule in the same statement that write an attempt row (mark_retry's retried arm, mark_retry_after's consuming snoozed arm), the statement pre-reads scheduled_at in a same-snapshot CTE — the pre-read is the claim-time value, the statement's RETURNING carries the NEXT attempt's. The chain therefore reconstructs from the ledger alone: due_at(k) → started_at(k) → due_at(k+1) is attempt k's wait, execution, and the next attempt's wait. There is deliberately NO index on the column: its reads are per-job (served by the primary key) or archive-analytics scans that already bound by time, and the archive twin's hypertable conversion keeps started_at as its chunk key untouched.

NULL semantics: NULL means a pre-migration attempt (or, during a rolling deploy, a row written by an old worker that named no such column). Historical due times are unrecoverable — jobs.scheduled_at was overwritten by every subsequent reschedule before this column existed — so there is no backfill; a backfill would fabricate data the ledger never recorded. Consumers reading attempt history should treat due_at IS NULL as "wait span unknown", never as "zero wait". The in-memory testing backend stamps the same semantics (the pre-write row's scheduled_at) so the differential tier exercises the column.

Migration 01.00.17_01 holds ACCESS EXCLUSIVE on jobs for the whole index rebuild

Unreleased. Operational note for the archive-candidate index migration; nothing breaks, but the apply blocks reads and writes on jobs for the duration of the build.

01.00.17_01_pre_archive_candidate_index_composite.sql rebuilds jobs_finished_at_idx as (status, finished_at) INCLUDE (id): a DROP INDEX then a CREATE INDEX, sharing the runner's one transaction. The migration file's own header (frozen text: the ledger records a SHA-256 of every bundled file and migrate up fails closed on checksum drift, so the header cannot be corrected in place) calls the rebuild "instant (one index, metadata-only lock windows)". Measured through the real migration runner, it is neither:

• The DROP INDEX takes ACCESS EXCLUSIVE on jobs within about a millisecond and, because it shares the transaction with the CREATE INDEX, holds it until COMMIT, i.e. for the whole build. Measured: roughly 0.03 s at 100k rows (36 MB), 0.37 s at 1M (305 MB), 3.1 s at 10M (2980 MB). For that whole window every enqueue, every dispatch UPDATE and even a plain count(*) queue behind it: the lock blocks reads and writes alike, not writes only.
• ddl_lock_timeout bounds the WAIT for the lock, not the hold: once the DROP INDEX holds jobs, a build that runs long runs long inside the lock (see the migration gave up waiting for a table lock).
• A build longer than dispatcher_command_timeout (5.0 s by default) is worse than a latency stall: the dispatch claim runs on the dispatcher pool under that per-query budget, so past the budget the fleet's dispatch iterations error instead of queueing, until the migration commits.
• The transaction is still the right shape on the abort side: a lock_timeout abort rolls back cleanly, the old index survives, the ledger stays empty, and no INVALID index is left behind. And exactly one index changes hands, so a moment of concurrent prune-and-retry sees the old plan shape, not an error.

The scale is modest at defaults: the shipped 30-day prune_retention_succeeded keeps the hot jobs table at roughly a month of terminal throughput, so 10M rows and a 3 s stall is reachable without being typical. Apply during a maintenance window, or when jobs is small or quiescent (e.g. right after a prune sweep), on any deployment where jobs is large.

Worker startup now rewrites actor_config.max_attempts and retry_kind

Unreleased. Correctness fix, no migration: both columns already exist (01.00.00_01_pre_initial.sql) and are NOT NULL with defaults.

Before this fix, sync_actor_config's UPSERT wrote these two columns only in the INSERT arm, i.e. the first time a row was created. The conflict arm left them untouched, and nothing else in the codebase wrote them, so the row kept the DDL defaults (3, 'transient') or whatever the first registration had seeded, forever. Server-side fires build their enqueue arguments from the stored row (cron ticks and the admin run-now in the cron leader), so an actor whose code declared, say, RetryPolicy(kind="indefinite", max_attempts=50) was fired server-side with 3 'transient' attempts. Producer-side enqueues were never affected: they carry the ActorRef's literal directly.

The conflict arm now assigns both columns from the registered literal on every boot. These columns are code-owned, the opposite of the capacity family (max_concurrent, max_pending, result_ttl, queue), which stay operator-owned-once-created and untouched by boots: no operator surface can write the retry contract (taskq actor-config set moves capacity, move-queue moves the queue, neither touches these), so the code literal is the only declaration of record and a changed @actor retry literal must reach the next server-side fire.

Operational consequence: if you have edited these columns out-of-band (direct SQL against actor_config), the next worker boot overwrites your edit with the code literal and logs actor-config-retry-contract-change (warning level, per actor, with both the registered and the stored pair). That warning means the stored retry contract changed at this boot; it is informational, never an error, and no force flag is involved. To make a hand edit durable, change the @actor(...) literal in code to match instead.

Shipped migration files are immutable

A migration file's bytes are part of its contract. The runner records a SHA-256 of the file's rendered SQL — every byte, comments included — in each database's schema_migrations ledger at apply time, and the drift guard refuses every future upgrade the moment a bundled file no longer renders to what a database's ledger recorded. The refusal is deliberate: a changed file means the schema was built from different SQL than the tree carries, and the runner will not build on unverified provenance.

Consequence for contributors: never edit, rename, or delete a migration file that has shipped. Ship the change as a NEW migration instead. This is not bureaucratic — editing 01.00.19_01 post-ship blocked every existing database from upgrading (the guard refused the customer's deploy at the migration step); the files are restored byte-exact and the improvements they carried ship as 01.00.21_02 (the fence-probe index's pg_index-keyed rebuild) and 01.00.21_03 (the batches CHECK constraints, NOT VALID + VALIDATE).

Operator consequence, one class only: a database that applied the REWRITTEN files (created from main between Sep 27 and this fix — e.g. built at pin bea355f7) holds the rewrites' checksums and will refuse against the restored files. The schema is correct — the carriers converge it — but the bookkeeping cannot be made true (the rewritten files no longer exist). For such a database: rebuild from a fresh database if practical; otherwise run taskq migrate up --allow-checksum-drift on it and keep that flag in its deploy pipeline — every run logs the drift as a warning, which is the honest state: that database's ledger refers to SQL that no longer exists anywhere.

Published variants: the one exception to immutability

Restoring a byte-exact file fixes the guard going forward, but databases that applied the PREVIOUS published bytes of an amended file carry that older checksum in their ledgers permanently — and no single file can render to two checksums. The _variants/ directory inside taskq.migrations carries the published history: for each amended file, the exact bytes of the version that shipped first (one .sql file per vintage, named for the commit that published it). The drift guard accepts a ledger checksum when it matches the current file OR any bundled variant's, each rendered with the checking database's own schema — so an accepted checksum is schema-agnostic by construction, and a checksum rendered under a different schema is still drift.

A ledger checksum nothing bundled produced — current file or variant — is drift exactly as before; the variants widen what counts as published history, never what counts as legitimate SQL. Variants are code: reviewed, bundled, and versioned with the package (the wheel ships them, and the guard degrades to fail-closed refusal if an install loses them) — adding one is a PR, never a runtime knob. Never add a variant for a file that has not actually shipped amended bytes: a variant legitimises a checksum forever.