Observability¶

TaskQ instruments itself with OpenTelemetry (vendor-neutral) and structured logging (structlog). No vendor SDK is bundled. You export data to any OTLP-compatible backend (Jaeger, Grafana Tempo, Honeycomb, Datadog, Sentry, Azure Monitor, PostHog) by pointing the standard OTel environment variables at your collector or agent.

1. OpenTelemetry: setup¶

Prerequisites¶

opentelemetry-api is a core dependency. The opentelemetry-sdk and OTLP exporter require the [otel] extra:

pip install "taskq-py[otel]"

Enabling and disabling¶

Variable	Default	Effect
`TASKQ_OTEL_ENABLED`	`true`	When `false`, all span and metric creation is suppressed; operations still succeed.

Worker startup¶

worker_main calls set_otel_enabled(settings.otel_enabled) before the TaskGroup opens. No application code needs to call this directly.

Exporter configuration¶

TaskQ emits through the OpenTelemetry API; the SDK and its exporters are what turn that into data at a backend. Configure them with the standard OTel environment variables; TaskQ does not override them:

Variable	Example
`OTEL_EXPORTER_OTLP_ENDPOINT`	`http://localhost:4317`
`OTEL_EXPORTER_OTLP_PROTOCOL`	`grpc` (default) or `http/protobuf`
`OTEL_TRACES_EXPORTER` / `OTEL_METRICS_EXPORTER` / `OTEL_LOGS_EXPORTER`	`otlp` (the default for traces and metrics once an endpoint is set), `console`, `prometheus`, `none`
`OTEL_SERVICE_NAME`	`my-app-worker`
`OTEL_RESOURCE_ATTRIBUTES`	`deployment.environment=production,k8s.pod.name=worker-0`

Under taskq worker the variables are enough. With the [otel] extra installed, the CLI installs SDK tracer and meter providers from them at startup through the SDK's own configurator (the same machinery opentelemetry-instrument uses) whenever any of OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER or OTEL_LOGS_EXPORTER is set. A bare endpoint selects otlp for traces and metrics (the specification's default; logs stay off unless OTEL_LOGS_EXPORTER names an exporter, because TaskQ logs through structlog). Startup logs one line saying what was wired:

otel-exporter-configured  traces=otlp  metrics=otlp  logs=  source=env

and source=none when nothing asked for an exporter. (That is the console renderer's shape; the default TASKQ_LOG_FORMAT=json emits the same line as the otel-exporter-configured event with traces/metrics/ logs/source — and the port fields — as JSON fields, so grep for the event name there.) Two other startup lines matter:

otel-exporter-unavailable (WARNING, with extra=otel or extra=prometheus); the variables are set but the package that provides the exporter is not installed. Nothing is exported; install the named extra.
otel-exporter-preconfigured (INFO): a provider was already installed before the worker started (an embedding application, a vendor distro such as configure_azure_monitor(...), or opentelemetry-instrument). The worker never replaces it.

An exporter the SDK cannot build (a misspelled name, a protocol no installed exporter speaks) fails startup with the reason, so the mistake is visible at deploy time rather than as a pipeline that looks configured and exports nothing. OTEL_SDK_DISABLED=true is honoured, and TASKQ_OTEL_AUTOCONFIGURE=false opts out entirely: set it when your application configures the SDK itself.

The alternative is the SDK's own launcher, which reads the identical variables: opentelemetry-instrument taskq worker --actors ... (the [otel] extra installs it). The worker detects the providers it set and changes nothing. Embedding applications that run worker_main without the CLI call taskq.obs.configure_exporters(settings) at process start, or configure the SDK directly; either way, before the worker records anything, because measurements taken before a provider exists are dropped, not replayed.

Common receiver addresses:

Datadog Agent: http://localhost:4317 (gRPC OTLP port)
Sentry Spotlight / OTel ingest: same
Azure Monitor: OTLP URL derived from the App Insights connection string
PostHog: PostHog Cloud OTLP endpoint

Validating the OTLP export path (collector round trip)¶

The export path is validated end to end in a dedicated lane: a real taskq worker, configured only through the standard OTel environment variables, exports spans and metrics over OTLP to a real collector container, and the test asserts on what the collector exported (the dispatch span under the configured service.name, the taskq.jobs.attempt_failures counter, and the masked form of a credential seeded into a failed job's error text). The same protocol that satisfies the collector is what the vendor intakes below consume.

make test-otel
# or directly:
uv run pytest --otel-validation -m otel_validation tests/otel_validation

The lane requires Docker and skips cleanly (it does not error) when the daemon is unreachable. It is opt-in: it never runs in the default test invocations.

Application Insights¶

TaskQ reads only the standard OTel variables, so the connection string's IngestionEndpoint is what the operator copies into the OTLP endpoint; the SDK appends the per-signal paths (/v1/traces, /v1/metrics) to that base. The connection string itself stays set exactly as the platform tooling expects it:

export APPLICATIONINSIGHTS_CONNECTION_STRING="InstrumentationKey=<guid>;IngestionEndpoint=https://<region>.in.applicationinsights.azure.com/"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://<region>.in.applicationinsights.azure.com/"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_SERVICE_NAME="my-app-worker"

Datadog¶

The agent's OTLP intake is a standard OTLP receiver on fixed ports: 4317 for gRPC (the default protocol) or 4318 for http/protobuf. Point the standard endpoint at the agent; DD_SITE and DD_API_KEY configure Datadog's own SDK family, which TaskQ does not ship, so TaskQ ignores them (they are only needed by processes exporting to the intake API directly, without an agent):

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
export DD_SITE="datadoghq.com"
export DD_API_KEY="<key>"   # for the agentless direct-intake shape only
export OTEL_SERVICE_NAME="my-app-worker"

Run the validation lane once from a deployment-shaped shell (the variables above, with the endpoint pointed at a local collector) before relying on it in production; the lane's assertions are the contract the export path must keep.

Instrumentation name¶

All spans and metrics are created under instrumentation name "taskq" (the value of INSTRUMENTATION_NAME).

Semconv compliance¶

TaskQ uses OTel messaging semconv attribute names (messaging.operation.type=publish, messaging.operation.type=process, messaging.consumer.group.name, etc.) so that dashboards built against the spec work without renaming.

2. Traces: span hierarchy¶

A complete job lifecycle produces four spans:

enqueue <actor>          (PRODUCER)
  └── process <actor>    (CONSUMER, linked to PRODUCER via span link)
        └── attempt.<N>  (INTERNAL)

A fifth INTERNAL span wraps the batch dispatch SQL query:

dispatch                 (INTERNAL)

Enqueue span¶

Name: enqueue <actor_name> (e.g. enqueue send_email)

Emitted by JobsClient.enqueue and SubJobEnqueuer.enqueue.

Attribute	Value
`messaging.system`	`"taskq"`
`messaging.destination.name`	queue name
`messaging.operation.type`	`"publish"`
`messaging.message.id`	job UUID (set after the DB write)
`taskq.actor`	registered actor name
`taskq.identity_key`	identity key string, or `""`

The enqueue span's trace_id and span_id are stored in the jobs table row (trace_id, span_id columns). At dispatch time the worker reads them and attaches them to the consumer span as a span link (links=[Link(...)]); it does not parent the consumer span to them. The consumer span therefore starts a new root trace, and a backend such as Application Insights shows enqueue and execution under two unrelated operation_Id values, joined only by the link. Parenting instead of linking is a separate change: it needs the W3C traceparent sampling flags persisted on the row, which the current two-column schema does not carry.

Dispatch span¶

Name: dispatch

Emitted by dispatch_batch each time the worker fetches a batch from PostgreSQL.

Attribute	Value
`taskq.queue`	first queue in the batch
`taskq.queues`	comma-separated list of queues
`taskq.batch_size`	requested batch limit

Errors on the SQL call set StatusCode.ERROR and call span.record_exception.

Consumer span¶

Name: process <actor_name> (e.g. process send_email)

Emitted by dispatch_one_job. Carries a span link back to the enqueue span so a trace viewer can correlate across the async boundary. If the stored trace_id or span_id is malformed (non-hex), the link is silently skipped and the consumer span is still created.

Attribute	Value
`messaging.system`	`"taskq"`
`messaging.destination.name`	queue name
`messaging.operation.type`	`"process"`
`messaging.message.id`	job UUID
`messaging.consumer.group.name`	`TASKQ_WORKER_GROUP` setting
`taskq.actor`	registered actor name
`taskq.attempt`	attempt number (1-based)
`taskq.identity_key`	identity key string, or `""`
`taskq.batch_id`	batch ID from job metadata, or `""`

The consumer span also carries lifecycle events added as span events:

Event name	When added
`lifecycle.running`	job transitions to `running`
`lifecycle.succeeded`	actor returns without error
`lifecycle.cancelled`	job is cancelled
`lifecycle.scheduled`	job is snoozed or reservation unavailable
`lifecycle.failed`	actor raises an unrecoverable error

Status is set to StatusCode.OK on success, StatusCode.ERROR on failure or cancellation.

Attempt span¶

Name: attempt.<N> (e.g. attempt.1)

An INTERNAL child of the consumer span. Wraps the actual actor function call plus transaction commit for the transactional execution path.

Trace context propagation¶

When a job is enqueued inside an active OTel span, the span context (trace_id as a 32-character hex string, span_id as a 16-character hex string) is written to the jobs row. At dispatch time the worker reads those columns, constructs a SpanContext with is_remote=True, and passes it as a trace.Link to the consumer span. This allows end-to-end tracing across the async enqueue-to-execute boundary without requiring the worker and the enqueueing process to share a trace context in-band.

Accessing the span inside an actor¶

ctx.span is the live consumer span, or None when OTel is disabled:

from taskq import actor
from taskq.context import JobContext


@actor
async def my_actor(payload: Payload, ctx: JobContext[Payload]) -> Result:
    span = ctx.span  # opentelemetry.trace.Span | None
    if span:
        span.set_attribute("custom.key", "value")
    ...

3. Metrics reference¶

All instruments are created from the "taskq" meter. Instruments marked unconditional are recorded even when TASKQ_OTEL_ENABLED=false because they represent safety-critical signals.

Serving the metrics: the Prometheus endpoint¶

Install the [prometheus] extra and taskq ui serve mounts GET /jobs/health/metrics, serving the series below in Prometheus text format:

pip install "taskq-py[prometheus]"
taskq ui serve   # scrape target: http://<host>:<port>/jobs/health/metrics

No provider setup is required. Mounting the route wires the data path, not just the route: at startup the process installs a PrometheusMetricReader-backed MeterProvider as the process-global OTel provider when nothing configured one, and logs prometheus-metrics-provider-wired. The wiring happens at startup because OTel drops measurements recorded before a provider exists; a provider installed at first scrape would serve an empty page forever. If you saw a 200 response with valid Prometheus text but zero taskq_* series, that is the failure this wiring exists to remove; check the startup log lines below before anything else.

If you configure your own provider, TaskQ never replaces it. Two shapes:

Your MeterProvider includes a PrometheusMetricReader (bound to the default prometheus_client registry, the one the endpoint scrapes): the router detects the bridge and changes nothing.
Your provider is OTLP-only (no PrometheusMetricReader): the route stays mounted, the scrape serves zero taskq_* series, and startup logs a prometheus-metrics-reader-missing WARNING. The shipped rules.yaml alert set references those series and can never fire in this state; add the reader yourself:

from opentelemetry import metrics
from opentelemetry.exporter.prometheus import PrometheusMetricReader
from opentelemetry.sdk.metrics import MeterProvider

# Before process start (before the worker or UI serves traffic):
reader = PrometheusMetricReader()  # default registry = the one the endpoint scrapes
metrics.set_meter_provider(MeterProvider(metric_readers=[reader]))

With TASKQ_OTEL_ENABLED=false no provider is installed and the mount logs prometheus-metrics-otel-disabled: an empty-of-taskq_* scrape is then a configuration statement, not a defect. One edge remains: prometheus-metrics-provider-blocked (WARNING) means a non-SDK global provider was set before the router was created, so OTel's set-once guard made the constructed bridge inert: set the provider once, at process start, with a PrometheusMetricReader attached.

Metrics are per-process. A scrape of a process shows what that process recorded. The taskq ui serve process records only its own activity (admin-triggered enqueues and cancels), so leader-sampled gauges (taskq.jobs.by_status, taskq.jobs.stranded, taskq.queue.depth, the sweep gauges) and worker-path counters (dispatch duration, consumed messages, attempt failures) are not in its scrape: they live in the worker processes. Scrape the workers for them, every pod: series are per-process, so a scrape of one pod says nothing about another.

The scrape topology in one line: the admin's /jobs/health/metrics serves the admin-process samplers ONLY (the SSE health pair taskq.admin.sse.connections / taskq.admin.sse.rejections_total, and whatever the UI process itself records); the fleet's series — every leader-sampled gauge and worker-path counter — live on each worker's own TASKQ_METRICS_PORT listener. Point Prometheus at the workers, every pod, for everything job-plane; point it at the admin process for the SSE surfaces.

The worker's scrape listener: TASKQ_METRICS_PORT. With the [prometheus] extra installed, setting a port makes taskq worker add a Prometheus pull reader to the meter provider it installs (see Exporter configuration) and serve every series this process records at http://<TASKQ_HEALTH_HOST>:<port>/metrics:

pip install "taskq-py[prometheus]"
TASKQ_METRICS_PORT=9464 taskq worker --actors myapp.actors:registry
# scrape target: http://<pod>:9464/metrics

The startup line reads otel-exporter-configured metrics=prometheus source=prometheus prometheus_port=9464 (the console shape; the default TASKQ_LOG_FORMAT=json emits the same fields on the otel-exporter-configured event); with an OTLP endpoint set as well, metrics=otlp,prometheus source=env,prometheus: both exporters share one provider. OTEL_METRICS_EXPORTER=prometheus with OTEL_EXPORTER_PROMETHEUS_PORT/_HOST is the SDK's own spelling of the same thing and works identically; TASKQ_METRICS_PORT is authoritative for the port when both are set, and the worker's own write of the port and host is scoped to the SDK call and rolled back, so neither value leaks to child processes. Unset, no listener is bound; when a provider is already installed by the embedding application, the worker wires nothing and the port stays unbound behind an otel-exporter-scrape-not-served WARNING.

The scrape endpoint is unauthenticated. It exposes actor names, queue names, and exception class names; never payloads or credentials, but real operational metadata. The listener binds the health host (TASKQ_HEALTH_HOST, 0.0.0.0 by default): keep it on the pod network, behind a network policy or an authenticating proxy if your cluster is shared.

The worker's health socket separately serves three hand-rendered process gauges (taskq_active_jobs, taskq_is_leader, taskq_shutdown_phase) via taskq health metrics (or GET /metrics on the optional TCP health listener). Those three are independent of OTel and need no extra; the OTel twins taskq.worker.active_jobs and taskq.worker.max_concurrency are on the real scrape.
Embedding applications that run worker_main themselves and already serve a scrape endpoint call taskq.contrib.prometheus.ensure_prometheus_meter_provider() once at process start (before the worker records anything) and serve the default registry as they do today; the FastAPI router (taskq.contrib.prometheus.create_metrics_router) is the same wiring mounted as a route.

A saturated rate limit is not a promotion stall¶

A growing scheduled count beside a flat pending count on taskq.jobs.by_status is the promotion-stall signature, but a saturated rate limit produces exactly that shape: denied jobs are rescheduled (scheduled with a future scheduled_at) without ever becoming pending. The two need different responses (unstick the promotion sweep vs. add capacity or wait out the limit), so do not alert on the shape alone. Discriminators:

Denial counters move only under rate limiting. Watch rate(taskq_ratelimit_denials_total[5m]) and rate(taskq_reservation_denials_total[5m]): a saturated bucket drives these; a promotion stall does not. The per-job record is the aggregated rate_limit_blocked_count column on jobs (a denial writes no job_events row; see the tip under Counters below).
The stall alert keys on the sweep, not the backlog. The shipped TaskQPromotionStalled rule fires on taskq_maintenance_leader_sweep_last_success_seconds{sweep_name="scheduled_to_pending"} going stale: promotion not running, which a healthy worker deferring rate-limited jobs never trips. Backlog shape rising while that sweep's last-success timestamp keeps advancing is saturation, not a stall.

Bucket names are deliberately not labels on the denial counters (caller-controlled cardinality); the fleet-wide rate tells you that admissions are denied, and rate_limit_blocked_count on the job row tells you which jobs absorbed them.

Counters¶

Metric name	Unit	Attributes	Description	Conditional?
`messaging.client.published.messages`	`1`	`actor`, `queue`	Jobs successfully enqueued.	yes
`messaging.client.consumed.messages`	`1`	`actor`, `queue`, `outcome`	Attempts consumed. `outcome` is one of `succeeded`, `failed` (terminal: a non-retryable class or the retry budget exhausted), `cancelled`, `scheduled` (the row went back to the queue: a retryable failure, a `Snooze`/`RetryAfter`, or an admission denial). Abandonment is never a consumer outcome; it has its own counter below. `taskq.jobs.attempt_failures` separates the failure share of `scheduled`.	yes
`taskq.jobs.attempt_failures`	`1`	`actor`, `error_type`, `retryable`	Attempts that ended in an actor failure, recorded at the failure handler once per handled exception. `retryable="true"`: the classifier rescheduled the attempt for another try; `"false"`: the failure is terminal. `error_type` is the exception class name. `TaskQRetryRateHigh` reads the `retryable="true"` share.	yes
`taskq.jobs.timeouts`	`1`	`actor`, `kind`	Jobs that exceeded a time budget. `kind="start_to_close"`: the per-attempt budget, counted at the timeout handler once per attempt whether or not it is retried; `kind="schedule_to_close"`: the whole-job deadline, counted at the deadline sweep and at the handler arms where the backend refused a retry, snooze or requeue with `DeadlineExceeded`. `taskq.deadline_exceeded_sweep.jobs_failed` remains the sweep's own (unconditional) count.	yes
`taskq.jobs.abandoned`	`1`	`actor`	Jobs abandoned: an operator cancel outlasted both grace periods and the running attempt was taken from the actor. Recorded by `mark_abandoned` itself on both backends; a shutdown interrupts instead and never produces it. `TaskQAbandonedJobs` reads this series.	yes
`taskq.cancellation.requested`	n/a	n/a	Incremented once per `JobsClient.cancel()` call regardless of outcome.	unconditional
`taskq.cancellation.phase_transitions`	`1`	n/a	Cancel phase transitions (0→1, 1→2, etc.).	yes
`taskq.notify.received`	n/a	n/a	NOTIFY callbacks delivered from asyncpg on the listener connection (all channels' payloads, matching and not).	unconditional
`taskq.notify.reconnects`	n/a	n/a	Successful listener reconnects: PG connection churn on the NOTIFY path. Each increment is one dedicated connection the listener had to re-establish; a sustained rate is the NOTIFY-connection-failure story (an unstable direct DSN, see troubleshooting.md), not individual job trouble.	unconditional
`taskq.notify.cancel_received`	n/a	n/a	Cancel NOTIFY callbacks delivered to this worker (payload parsed, `type="cancel"`, `worker_id` matching this worker) and turned into cancel wakeups.	unconditional
`taskq.notify.payload_parse_failed`	`1`	n/a	NOTIFY payloads discarded because they did not parse to the expected JSON envelope. Dropping is the correct outcome for a channel TaskQ does not exclusively own, but a rising rate is not a quiet channel: it is a foreign writer or a schema change on the channel publishing bodies the envelope cannot decode.	unconditional
`taskq.cancel.notify_sent`	n/a	`schema`	`pg_notify` calls fired for running-job cancel requests (the fast path beside the heartbeat poll, which remains authoritative).	unconditional
`taskq.backpressure.errors`	n/a	`actor`, `kind`	Synchronous refusals raised at enqueue. `kind` is the bounded enum `max_pending` / `max_pending_lock_timeout` (capacity admission) or `unique_for_lock_timeout` / `idempotency_lock_timeout` (identity-serialization refusals, counted beside their typed errors, not capacity signals, so alerting keyed on the capacity kinds is not tripped by them).	unconditional
`taskq.backpressure.capacity_refresh_failures`	`1`	`degraded`, `error_type`	Failed refreshes of the enqueue-side `actor_config` capacity cache. `degraded` is `"stale_snapshot"` (a previous snapshot is still being served) or `"no_snapshot"` (the cache never loaded and every enqueue is enforcing the `@actor` literal). `error_type` is the exception class name of the failed read.	unconditional
`taskq.deadline_exceeded_sweep.jobs_failed`	`1`	`actor`	Jobs failed by the deadline-exceeded sweep.	unconditional
`taskq.jobs.reclaimed`	`1`	`actor`, `disposition`	Running jobs reclaimed by the expired-locks sweep (the holder broke its liveness promise: lease expiry or heartbeat timeout). `disposition` is the code-fixed enum `repended` (attempts remained, the row went back to pending on its retry curve), `crashed` (budget exhausted, terminal), `cancelled` (a cancel request was in-flight when the holder died: the honest terminal label), plus `unknown`, the fallback a post-update status no CASE branch writes counts under, so map/code drift shows up as its own series instead of rolling the reclaim batch back. The per-actor crash split beside the sweep-name total `taskq.maintenance_leader.sweep_rows`; recorded once per (actor, disposition) pair after the sweep's transaction, by the Postgres sweep and the in-memory twin alike.	unconditional
`taskq.dispatch.failures`	`1`	`queue`, `error_type`	Dispatch rounds that raised before returning a claim set. A producer failing every round is otherwise silent on every dispatch signal: indistinguishable from an idle queue. `error_type` is the exception class name, separating a self-healing failure class (a lock timeout, a connection reset) from a permanent one (an auth failure, schema drift).	yes
`taskq.dispatch.corrupt_rows`	`1`	`queue`, `column`	Claimed rows whose jsonb columns refused to decode to the row contract, counted in the dispatch round's per-row decode path. `queue` is capped (like `taskq.dispatch.failures`); `column` is a closed set: `payload`, `metadata`, `progress_state`, `result` (or `unknown`, when the failing column cannot be determined). Each counted row is terminally failed in the same round with `error_class='CorruptJobDataError'`, so a poisoned row is visible here instead of as a producer loop that keeps dying on the unexpected-failure backstop: any non-zero rate is a producer writing garbage rows (schema drift, an interop writer).	yes
`taskq.enqueue.dedups`	`1`	`dedup_reason`	Enqueue dedup hits (an enqueue returned an existing row instead of writing one). `dedup_reason` is a bounded enum of three: `unique_for`, `idempotency_key`, or `zombie_ack_lost` (the first attempt committed its row but died before its acknowledgement; the retry's INSERT hit the committed row, and the identity stays pinned to that zombie's job id instead of erroring for work that succeeded). The per-hit log lines are budget-bounded at batch scale; this counter is the rate signal that survives the bound.	yes
`taskq.heartbeat.misses`	`1`	n/a	Heartbeat renewal failures (a tick whose renewal statement failed on a transient Postgres error). A miss is this counter plus the `heartbeat-tick-failure` log line only: no `job_events` row is written for it, although the schema's column comment lists a `heartbeat_miss` kind: the kinds actually written are `state_change`, `cancel_request` and `progress`.	yes
`taskq.worker.slot_pool.acquire_failures`	`1`	`error_type`	Bounded acquires from the per-slot transaction pool that failed (timeout or connection error); infrastructure, not a job outcome: the claimed job is left for lock-lease reclaim. `error_type` is the exception class name; the per-occurrence job id and full error stay in the `slot-pool-acquire-failed` log event.	yes
`taskq.leader.election_attempts`	`1`	n/a	Leader election attempts.	yes
`taskq.leader.election_failures`	`1`	n/a	Election attempts that did not win the lock.	yes
`taskq.error_reporter.failures`	`1`	`reporter_type`	`ErrorReporter` invocation failures.	yes
`taskq.progress.publish_failures`	`1`	`channel`, `error_type`	Redis publish failures for progress fanout. `channel` is `per_job` or `global`; `error_type` is the exception class name.	yes
`taskq.ratelimit.refund_failures`	`1`	`bucket`, `backend`, `error_type`	Rate-limit refund/rollback failures. `error_type` is the exception class name. The `bucket` label is capped: the first 100 distinct names a process sees keep their series, later names collapse onto `_other_`; per-bucket attribution stays on the `ratelimit-rollback-failure` log line.	yes
`taskq.ratelimit.denials`	`1`	`backend`	Rate-limit decisions that denied admission. Bucket names are not a dimension (caller-controlled cardinality).	yes
`taskq.ratelimit.acquire_dependency_failures`	`1`	`error_type`	Rate-limit acquires that failed on a store dependency (Redis or the PG fallback) and were failed closed as denials: an availability signal, distinct from `taskq.reservation.denials` (admission decisions). Read the two together before scaling a bucket; `error_type` is the exception class name.	yes
`taskq.reservation.denials`	`1`	`source`	Reservation/rate-limit admission denials surfaced to a worker handler. `source` is `reservation` or `rate_limit`. Bucket names are not a dimension.	yes
`taskq.sub_enqueue.failures`	`1`	`actor`, `error_type`	Sub-enqueue flush failures after the parent job committed: each counted unit is one child job that was reported as enqueued but was not. `actor` is the parent's actor; `error_type` is the exception class name.	yes
`taskq.worker.loop_stall_attributions`	`1`	`actor`, `kind`	Event-loop scheduling stalls attributed to the synchronous work holding the interpreter while the loop could not schedule, recorded by the lag watchdog's daemon thread at the warn and trip tiers. `kind="blocking_call"`: a synchronous call that released the GIL (I/O wait, a subprocess); `kind="gil_held"`: synchronous work that held it (a C extension without GIL release, or a hot pure-Python loop). `actor` names the registered actor whose frame sat under the stall, or the `_other_` overflow when no registered actor appeared in the sampled stack. See The watchdog family.	yes
`taskq.pruned.jobs`	`1`	`actor`, `status`	Jobs moved from `jobs` to `jobs_archive` by the prune sweep (Sweep 5).	yes
`taskq.archived.jobs`	`1`	`status`	Same prune-sweep event, status-only view (no actor dimension).	yes
`taskq.expired_archive.jobs`	`1`	`status`	Jobs hard-deleted from `jobs_archive` by the archive expiry sweep (Sweep 6).	yes
`taskq.maintenance_leader.sweep_rows`	n/a	`sweep_name`	Rows affected per sweep tick.	yes
`taskq.maintenance_leader.sweep_timeouts`	`1`	`sweep_name`	Sweep calls aborted by a deadline or server-side statement cancel, and gauge-sampler reads that did not complete for ANY reason (the sampler `sweep_name`s are `queue_depth` / `backlog_detection` / `actor_backlog` / `reservation_slots`). A non-zero rate means work is being aborted or going unobserved, not completing slowly.	yes
`taskq.leader.lock_contention`	`1`	`lock`	Advisory-lock acquisitions lost to another session, recorded by the losing side.	yes
`taskq.cron.lock_contention`	`1`	n/a	Cron ticks that returned without firing because another session held the cron advisory lock. A sustained rate equal to the tick rate means cron is not running anywhere (a partitioned holder never releasing the transaction-scoped lock); a brief low rate is leader-handover overlap.	yes
`taskq.cron.budget_deferrals`	`1`	`actor`	Cron fires deferred because the tick's funded factory budget had no fundable grant left for them: a schedule planned ahead consumed the budget, or the leftover fell below the minimum fundable grant. A brief burst is catch-up draining in tick-sized batches; a SUSTAINED rate means one schedule's payload factory is monopolizing the tick budget every tick: a slow-but-successful factory never strikes and never auto-disables, so its peers retry every tick without ever being funded (delayed, not lost; the deferral advances `next_fire_at` one leader tick). Per-schedule attribution is on the `cron-fire-budget-deferred` log line; the operator resolution (tighten `TASKQ_CRON_PAYLOAD_FACTORY_TIMEOUT` below the monopolizer's duration, or raise `TASKQ_DISPATCHER_COMMAND_TIMEOUT`) is the cron guide's tick-budget section and the `TaskQCronBudgetDeferrals` runbook. The `actor` label is capped like `taskq.cron.consecutive_failures`.	yes
`taskq.cron.skipped_slots`	`1`	`actor`	Cron schedule occurrences a fire attempt DROPPED because the schedule's `next_fire_at` fell behind `now() - TASKQ_CRON_CATCH_UP_WINDOW`: fires the system could not even attempt, counted in FIRE UNITS (a `*/5` schedule reported as 3 is three dropped fires, the unit the runaway predicate compares against the cron period). One increment per fire attempt carrying that attempt's full drop count; skipped slots are never replayed (the recompute lands `next_fire_at` in the future), so the count IS the record of what was dropped — except when the count walk's tick-budget slice expired mid-backlog (a fleet-down backlog too deep to count inside the tick): the `cron missed slots skipped` warning then flags `skipped_slots_partial=true` and the count is a floor, which the runaway predicate tolerates (a backlog that deep is past any period). This is the producer side of the runaway-fan-out detection — the system admitting it is losing the race; the clearance predicate itself is the fires-vs-clearance SQL in `src/taskq/insights.py` (the `taskq.insights` module). Read beside `taskq.cron.slots_behind` (the depth) and `taskq.cron.budget_deferrals` (a different, budget-shaped lag). Emitted from the tick's commit-gated emission, so a rolled-back tick (whose owed slots are re-attempted) reports nothing. The `actor` label is capped like `taskq.cron.consecutive_failures`. `TaskQCronSkippedSlots` reads it.	yes
`taskq.jobs.interrupted`	`1`	`actor`, `hold`	Running attempts released back to the fleet by a worker shutdown (`mark_interrupted` landed). The spent attempt stands: the attempt did start executing, so its increment is NOT refunded (see `interrupt_count` and the release contract in cancellation.md). `hold` buckets the release by whether the row was parked behind a hold (`">0"`, the actor was still running when the graces expired) or re-pended immediately (`"0"`), the split an operator reads to see whether deploys are interrupting responsive or unresponsive actors.	yes
`taskq.jobs.interrupted_noop`	`1`	`actor`	`mark_interrupted` calls declined by the fence (the row is no longer this worker's running row at the attempt epoch, or an operator cancel is in flight, which wins). A silent no-op on a release path would look like a release that never landed, so the fenced-out arm has its own counter; `actor` is empty when the fenced-out read cannot attribute one.	yes
`taskq.progress.flush_failures`	`1`	`stage`, `error_type`	Progress flush failures to Postgres, by stage. `stage="per_job"`: one job's flush UPDATE failed, that job's progress since the last flush is lost; `stage="pool"`: a pool could not be obtained at all (the loop-level getter failed, or the per-job acquire failed or was exhausted), so every job's progress is lost. The two are materially different incidents and the pool-stage sites log a different kind than the per-job one, so an alert rule can separate them. `error_type` is the exception class name.	yes
`taskq.maintenance_leader.sweep_unexpected_errors`	`1`	`sweep_name`	Sweep batches aborted by an error outside the deadline family (a constraint violation, a connection reset, any fault that is not a statement cancel). The deadline family lands on `taskq.maintenance_leader.sweep_timeouts`; everything else that aborts a prune-family batch lands here, because the sweep's row counters and success stamps only move on success, so a drain stopped by this class reads healthy on every other plane. One increment per aborted batch; the first aborted batch ends the sweep attempt, so the rate tracks the prune retry ladder. `TaskQSweepUnexpectedErrors` reads it.	yes
`taskq.worker.loop_unexpected_errors_total`	`1`	`loop`	Unexpected (non-transient) errors tolerated by a long-lived loop's backstop (`UnexpectedLoopErrorGuard`): the leader maintenance loops, the worker producer loop, and the progress flush loop. The guard re-raises after its consecutive-failure budget, so every increment is an error the loop chose to survive - per its own description, "anything above zero warrants investigation: either PG produced a shape the transient set should learn, or the loop has a bug." Prescribe the alert at > 0, not at a rate.	yes
`taskq.ratelimit.reclaim_drain_failures`	`1`	`error_type`	Failures of the keyed-reservation slot-row reclaim drain: a persistently failing drain strands `reservation_slots` rows, a STORAGE signal (the backlog forms on `taskq.ratelimit.reclaim_pending`). Distinct from `reclaim_heal_failures`, which is an AVAILABILITY signal; the two never share a counter. `error_type` is the exception class name.	yes
`taskq.ratelimit.reclaim_drain_rows`	`1`	n/a	Rows deleted by the keyed-reclaim drain, `reservation_slots` slot rows and `rate_limit_buckets` bucket rows alike (RETURNING-confirmed).	yes
`taskq.ratelimit.reclaim_heal_failures`	`1`	`error_type`	Failures of the acquire-path re-materialisation heal for keyed buckets whose slot rows a sibling worker's drain deleted: a failing heal denies new admissions for that bucket, an AVAILABILITY signal, the opposite polarity of `reclaim_drain_failures` (storage). `error_type` is the exception class name.	yes
`taskq.jobs.cancels_actored`	`1`	`actor`	Operator cancel requests this WORKER observed in its cancel-poll and began acting on, per actor: the executing side of the cancel story. `taskq.cancellation.requested` counts what the ISSUING process incremented — a `taskq jobs cancel` CLI call exits after writing the row, so the issuer's increment dies with it and the scrape shows nothing; this counter moves on the worker whose heartbeat poll is the durable surface, so the same cancel is visible in the worker scrape after the issuer is gone. One increment per cancel (the first observation: the cooperative arm, or the PG fast-advance when the request was first seen already-escalated); the ladder's re-walks record nothing. Read beside `taskq.jobs.cancel_pending` for how many are still in flight.	yes
`taskq.admin.sse.rejections`	`1`	`topic`, `surface`	SSE connection attempts refused because the topic's connection cap was already held (HTTP 429), recorded at the two 429 sites (the admin `/sse/{topic}` endpoint and the shared slot helper the progress stream uses). Same closed label vocabulary as `taskq.admin.sse.connections`, so the two join: rejections rising beside connections pinned at the cap is saturation; rejections alone is a burst the cap absorbed. Recorded in the admin process; exported on the admin's `/jobs/health/metrics`.	yes

One counter outside this table's worker/producer population: taskq.admin.audit.record_failed (admin-UI audit rows that could not be recorded after a backend-mediated mutation) is detailed in admin-ui.md's Audit trail section.

Which job is starving? The denial counters live on the row

The denial counters above are fleet-wide rates: they tell you the fleet is shedding admissions, not which job has been waiting. Because an admission denial writes no job_events and no job_attempts row (per-denial rows grow without bound under sustained contention); the per-job record is the aggregated rate_limit_blocked_count column on jobs (and snooze_count for actor-requested deferrals). Query it when one job is mysteriously slow while the fleet looks healthy:

SELECT id, actor, rate_limit_blocked_count, snooze_count, scheduled_at
FROM {schema}.jobs
WHERE status = 'scheduled' AND rate_limit_blocked_count > 0
ORDER BY rate_limit_blocked_count DESC
LIMIT 50;

Histograms¶

Metric name	Unit	Attributes	Description
`messaging.process.duration`	`s`	`actor`, `queue`, `outcome`	Job execution duration from dispatch to the attempt's end, labelled with the same `outcome` as `messaging.client.consumed.messages` (`succeeded` / `failed` / `cancelled` / `scheduled`), so a `start_to_close` timeout, which lands at exactly the budget, or a failure does not drag the success percentiles. Buckets: 5 ms – 600 s, seconds-scaled (the percentile reads the ops playbook prescribes need unit-scaled boundaries, not the SDK's unit-agnostic defaults).
`taskq.dispatch.duration`	`s`	`queue`	Batch dispatch SQL query latency (SQL execution only). Buckets: 1 ms – 5 s, seconds-scaled so `TaskQDispatchLatencyHigh`'s 50 ms threshold quantiles against real sub-50 ms data.
`taskq.dispatch.pool_acquire_duration`	`s`	`queue`	Seconds a dispatch round spent waiting for a dispatcher connection (pool wait, no SQL), kept out of `taskq.dispatch.duration` so a pool-exhausted pod cannot inflate the query-latency percentiles. Buckets: 1 ms – 30 s, seconds-scaled.
`taskq.jobs.queue_wait_seconds`	`s`	`actor`, `queue`	Seconds a job waited between becoming eligible (`scheduled_at`) and being claimed (`started_at`), both server-clock stamps on the dispatched row. Recorded once per dispatch, retries and re-pends included: the per-job companion of the sampled `taskq.jobs.oldest_pending_age_seconds`, which only shows the head of the line. Buckets: 1 ms – 1 h, seconds-scaled (the p99 read the playbook prescribes needs sub-second buckets for healthy fleets).
`taskq.lock.expires_in_seconds`	`s`	n/a	Lease remaining on this worker's job locks at the moment the heartbeat renewed them: `lock_lease` minus the measured gap since the previous renewal (nothing on the first), so a late or failed tick lowers the sample and `TaskQLockExpiringSoon` can fire; 0 when the renewal landed after expiry. A healthy worker reads `lock_lease − heartbeat_interval`. Buckets: 0, 5, 10, 15, 20, 30, 45, 60 s.
`taskq.heartbeat.tick_duration_seconds`	`s`	n/a	Wall-clock seconds per heartbeat tick. Buckets: 0.5 ms – 10 s, seconds-scaled.
`taskq.worker.event_loop_lag_seconds`	`s`	n/a	Event-loop scheduling latency measured by the lag watchdog: seconds between the watchdog thread asking the loop to run a callback and the loop running it: one sample per landed beat (about one per `TASKQ_WATCHDOG_CHECK_INTERVAL` on a healthy loop, microseconds each) plus the stall observed at a trip. The continuous signal under the warn/trip thresholds: a rising p99 is a loop being blocked (a sync call without `asyncio.to_thread`, a GC pause, a saturated CPU) before it is blocked long enough to page; the lock-TTL histogram and `TaskQLockExpiringSoon` follow it. Buckets: 5 µs – 30 s, microsecond-to-seconds scaled so healthy microsecond beats and a multi-second stall are distinct series. See The watchdog family.
`taskq.worker.loop_idle_fraction`	`1`	n/a	Sampled idle fraction of this worker's event loop, one sample per heartbeat window: the share of the watchdog's polls whose loop thread was parked in its idle selector wait. A sampled proportion, not a utilisation integral (sub-poll busy bursts invisible; unreadable frame samples count as not parked). Read beside the windowed stall tally: stalls attributed + low idle = a loop with no spare scheduling capacity; a healed worker's idle fraction recovers while its cumulative stall counts stand still. See The watchdog family and the limits.
`taskq.worker.shutdown_duration_seconds`	`s`	n/a	Wall-clock seconds from the first shutdown signal to clean worker teardown; recorded only on a clean exit (a watchdog trip force-exits without recording).
`taskq.maintenance_leader.sweep_duration_ms`	`ms`	n/a	Per-sweep-tick wall-clock duration.
`taskq.ratelimit.reclaim_drain_duration`	`s`	n/a	Wall-clock duration of one keyed-reservation reclaim drain statement. Recorded on success and failure alike (the caller passes it from a `finally`), so a timeout that aborted the drain still leaves a duration sample.

Reading percentiles off the served buckets¶

The seconds-scaled boundaries above are an approximation contract, and the approximation is bounded, not magic: histogram_quantile interpolates linearly inside the bucket whose cumulative count contains the rank, so the served quantile's error against the true quantile is bounded by the WIDTH of that containing bucket - never zero, and largest when the distribution's mass piles up against a boundary. The review suite (tests/test_prometheus_quantile_honesty.py) pins the bound per histogram against synthetic distributions with known order statistics, served through the real bridge; the measured shape of the error:

Read	True p99	Served p99	Error	Bounded by
`taskq.dispatch.duration` (healthy ~12 ms fleet, 1% tail to 900 ms)	~37 ms	~49.8 ms	~13 ms	the (25, 50 ms] bucket's 25 ms width
`taskq.jobs.queue_wait_seconds` (healthy sub-second, 2% stragglers ~45 s)	~45.0 s	~45.0 s	~0.03 s	the (30, 60 s] bucket's 30 s width
`taskq.lock.expires_in_seconds` (healthy lease 60 s − 10 s cadence)	~50.0 s	~59.9 s	~9.9 s	the (45, 60 s] bucket's 15 s width
`taskq.worker.event_loop_lag_seconds` (healthy ~20 µs beats, one 2 s stall)	~77 µs	~94 µs	~17 µs	the (50, 100 µs] bucket's 50 µs width

Three consequences operators should know:

The sub-millisecond event-loop range has real resolution, but the read is the TREND. Healthy ~20 µs beats spread across five boundaries below 1 ms; a p50 reads within tens of microseconds of truth. The relative error at that scale is large (tens of percent) and the histogram is not a stopwatch - the documented read ("a rising p99 is a loop being blocked") and the clean separation of a multi-second stall from the healthy mass are what it guarantees.
Alert thresholds on quantiles sit at bucket boundaries by design. TaskQDispatchLatencyHigh's 50 ms threshold IS the le="0.05" boundary: dispatch mass strictly below it cannot quantile to 50 ms or above (the healthy worst case quantiles to ~49.75 ms), so the rule fires exactly when dispatches cross the 50 ms bucket - and never on a healthy fleet, however large. The review suite proves both directions on marginal shapes in promtool.
A threshold that coincides with a HEALTHY cadence's steady-state bucket false-pages. TaskQLockExpiringSoon's 30 s line is the le="30" bucket edge: a healthy configuration whose lock_lease − heartbeat_interval lands in (20, 30] s (e.g. heartbeat 5 s with the 33 s cascade-floor lease) quantiles to ~29.9 s and pages forever. See the TaskQLockExpiringSoon runbook for the arithmetic and the override.

Observable gauges (polled)¶

Metric name	Unit	Attributes	Description
`taskq.queue.depth`	`1`	`queue`	Pending and scheduled jobs per queue. Sampled by the leader every 15 s; the 100 deepest queues keep their series, the rest collapse onto one `_other_` series carrying their SUMMED depth (a bookkeeping total: the unserved join excludes it, see Dimension cardinality).
`taskq.queue.live_workers`	`1`	`queue`	Workers whose `last_seen_at` is inside the liveness window (`TASKQ_ADMIN_WORKER_LIVENESS_SECONDS`), per queue they subscribe to, sampled in the same leader tick as `taskq.queue.depth`, same cap (the overflow collapses onto one `_other_` series carrying the SUMMED worker count — bookkeeping, not alertable: the two gauges' `_other_` sets differ, see Dimension cardinality). A queue with depth and no live worker is unserved (`TaskQQueueUnserved`); a dead-but-unswept worker row does not count.
`taskq.reservation.slots_used`	`1`	`bucket`	In-use reservation slots per rate-limit bucket. Sampled by the leader every 15 s.
`taskq.maintenance_leader.is_leader`	`1`	`worker_id`	`1` on the elected leader pod, `0` on all others.
`taskq.cron.disabled_schedules`	`1`	n/a	Count of currently disabled cron schedules.
`taskq.cron.slots_behind`	`1`	`actor`	How far behind its schedule a cron actor's fire was when it skipped, in FIRE UNITS (a `*/5` schedule reported as 3 was three dropped fires deep): the depth companion of `taskq.cron.skipped_slots`. Per actor, the most recent committed cron tick's worst schedule depth; a clean fire re-observes the actor at 0, so a depth that keeps climbing period over period is the runaway fan-out compounding, and one that decays to 0 was a one-time drop. Published by the leader's cron tick only (the tick is the only place the drop is computed), so a non-leader process carries no series — the leader-observable pattern, like `taskq.cron.disabled_schedules`. The `actor` label is capped like `taskq.cron.consecutive_failures`.
`taskq.heartbeat.consecutive_failures`	n/a	n/a	Consecutive heartbeat tick failures for this worker (sample-on-scrape).
`taskq.notify.connected`	`1`	`schema`	`1` if this process's NOTIFY listener connection is healthy, `0` otherwise (sample-on-scrape). A `0` beside a rising `taskq.notify.reconnects` rate is a listener that cannot hold a connection; see NOTIFY connection failures.
`taskq.worker.active_jobs`	`1`	n/a	Jobs in flight on this worker process: the OTel twin of the health socket's hand-rendered `taskq_active_jobs`, which no scrape reaches. One series per process; divide by `taskq.worker.max_concurrency` for utilisation.
`taskq.worker.max_concurrency`	`1`	n/a	This process's configured `TASKQ_MAX_CONCURRENCY`, the ceiling `active_jobs` saturates against. One series per process.
`taskq.jobs.running`	`1`	`actor`	Running jobs per actor, fleet-wide, sampled by every worker beside `taskq.jobs.by_status` from one grouped read over the running population. Which actors hold the slots while pending work waits; an actor with nothing running is absent, not 0.
`taskq.jobs.oldest_running_age_seconds`	`s`	`actor`	Seconds since the oldest running attempt of each actor started (`started_at`, server clock), from the same grouped read as `taskq.jobs.running`. Running age past the actor's p99 (`messaging.process.duration`) with `taskq.jobs.timeouts` flat is an actor with no `start_to_close`: nothing will end the attempt, and `running_lease_expired` stays 0 while the heartbeat renews it.
`taskq.worker.slot_pool.connections_in_use`	`1`	n/a	Connections of the per-slot transaction pool currently held by dispatching jobs (reported only when the worker runs one). A pool pinned at its maximum with zero acquire failures is saturation: visible below the acquire-failure cliff.
`taskq.maintenance_leader.sweep_last_success_seconds`	`s`	`sweep_name`	Unix timestamp of each sweep's last successful call. `time() - this value` is sweep staleness; a value that never moves while the process runs is a stalled sweep. Both operands of that difference are wall clock (the stamp is taken from `time.time()`, Prometheus's `time()` is scrape-time wall clock), so a backward clock step mutes any staleness alert built on it for the catch-up window; see the clock-domain caveat in the TaskQPromotionStalled runbook.
`taskq.maintenance_leader.sweep_batch_size`	`1`	`sweep_name`	Rows per committed batch each sweep is currently using. A value below `event_writer_batch_size` is the reduced (degraded) tier.
`taskq.maintenance_leader.sweep_batch_size_configured`	`1`	`sweep_name`	The batch size this worker's `event_writer_batch_size` configures for each sweep; emitted at the same call site as `sweep_batch_size` so the sweep-degraded alert compares the two label-matched.
`taskq.maintenance_leader.lease_expires_in_seconds`	`s`	n/a	Seconds until the leader lease expires, as stamped by the holder's last successful election or renewal (`leader_lease` seconds out, re-stamped every `heartbeat_interval`). Present only on the pod holding the lease and cleared on demotion, so a series that goes stale or vanishes is a leader that stopped renewing: the per-pod evidence behind the split-brain/no-leader alert.
`taskq.jobs.by_status`	`1`	`status`	Jobs per live status, counted exactly and sampled by every worker; terminal statuses are not sampled. A growing `scheduled` count next to a flat `pending` count is the promotion-stall signature.
`taskq.jobs.scheduled_count`	`1`	n/a	Count of `scheduled`-status jobs: the label-less twin of `taskq.jobs.by_status{status="scheduled"}`, sampled at the same tick. Exists so `TaskQScheduledBacklogGrowing` can compare a growth signal against `taskq.jobs.oldest_due_age_seconds` (also label-less) without a PromQL join modifier; a `status`-labeled series never matches a label-less one under vector `and`.
`taskq.jobs.oldest_due_age_seconds`	`s`	n/a	Seconds since the oldest scheduled job became due for promotion. Grows monotonically while promotion is stalled.
`taskq.jobs.actor_backlog`	`1`	`actor`, `queue`	Pending jobs per (actor, queue). No worker refuses to start because an actor's queue has no consumer: in a multi-worker fleet no supervisor can know what consumes a queue, so a misrouted actor piles up pending rows while every probe stays green. A queue-summed depth cannot tell that apart from a busy shared queue; this actor's own series rises while its queue-mates stay flat. Exact below the sampler's 1000-row per-pair cap and reading 1000 (a lower bound, never an under-count) at or above it (a series pinned at 1000 means "at least this deep"); the companion `oldest_pending_age_seconds` carries the part that grows without bound.
`taskq.jobs.oldest_pending_age_seconds`	`s`	`actor`, `queue`	Seconds since the oldest PENDING job became eligible, per (actor, queue). Depth alone is ambiguous: a deep queue that drains is healthy throughput, but an actor nobody consumes has a pending job whose age grows with wall clock. Distinct from `oldest_due_age_seconds`, which measures promotion (scheduled → pending) and reads 0 for pending work no consumer takes. Sampled from the same grouped snapshot as `actor_backlog`, so depth and age can never describe two different moments. When the per-actor read fails, both series go ABSENT for the interval: the honest state, never a stale value, and the failure counts on `taskq.maintenance_leader.sweep_timeouts` under `sweep_name="actor_backlog"`: an absent age series beside a rising timeouts rate is a dead sampler (the queue-depth alert's operand is gone, `TaskQSweepTimeouts` carries the truth), not a resolved alert.
`taskq.jobs.running_lease_expired`	`1`	n/a	Running jobs whose lock lease is past expiry (the zombie-running shape), with rows in a cancel phase (`cancel_phase != 0`) carved out: the reclaim sweep deliberately waits out the cancel grace ladder for those, so an expired lease mid-cancel is the protocol working, not a zombie; a cancel that never completes pages elsewhere: `TaskQAbandonedJobs` when its worker is alive to escalate through the phases, `TaskQHeartbeatMisses` when it died mid-cancel; reclaim honors the row to `cancelled` either way. Healthy reads 0: the reclaim sweep drains expired leases within a tick or two, so a sustained non-zero reading means reclaim is not draining. Sampled by every worker with `taskq.jobs.by_status`; the per-job truth (`locked_by_worker`, `lock_expires_at`) is on the admin `/jobs` lease column, not on a label.
`taskq.jobs.stranded`	`1`	`actor`, `reason`	Pending/scheduled jobs that can never be dispatched, sampled by the leader. `reason` is `no_actor_config` (the actor has no `actor_config` row) or `unserved_queue` (the queue dispatch routes the actor on has no live worker subscribed; a worker whose heartbeat has gone stale does not serve a queue, even before the stale-worker sweep removes its row). An empty reading means recovery; `TaskQStrandedJobs` reads this series.
`taskq.ratelimit.reclaim_pending`	`1`	n/a	Evicted keyed bucket names, reservations and rate limits alike, waiting for their rows (slot or bucket) to be reclaimed by the next drain tick. A scalar, not a per-bucket gauge: bucket names are caller-controlled and must not become label cardinality. Persistently non-zero beside a rising `taskq.ratelimit.reclaim_drain_failures` rate is the signature of broken reclamation.
`taskq.jobs.retrying`	`1`	`actor`	Non-terminal jobs (pending/scheduled/running) that are past their first attempt (`attempt > 0`), per actor, sampled by every worker with `taskq.jobs.by_status`. The retry ladder's live population: a job sitting at attempt 6 of 1000 is a healthy pending row on every other surface — `by_status` counts it pending, `taskq.jobs.attempt_failures` last moved five attempts ago. Beside `taskq.jobs.retry_headroom`, which says how close the worst row is to its ceiling. An actor with nothing retrying is absent, not 0.
`taskq.jobs.retry_headroom`	`1`	`actor`	The MINIMUM of `max_attempts - attempt` over the actor's live non-terminal rows with `attempt > 0`: how many attempts the actor's worst in-flight job has left, from the same grouped read as `taskq.jobs.retrying` (one statement, one moment). The haunt-detection operand: headroom 0 is a job ON its last attempt; a value pinned near 0 across ticks is a retry loop running out of ladder (see the alert suggestions).
`taskq.queue.depth_by_status`	`1`	`queue`, `status`	The depth gauge's status split, per (queue, status), sampled by the leader in the same tick and on the same connection as `taskq.queue.depth` (one grouped read's GROUP BY extended with `status`), so a growing `scheduled` share — the promotion-stall signature — is visible per queue. Kept on a sibling instrument rather than a relabel of `taskq.queue.depth`: the shipped alert set (`TaskQQueueUnserved`'s `on(queue)` join) and the cardinality proofs pin the depth gauge's one-series-per-queue label set. Same cap; the `_other_` pair (mixed statuses) is bookkeeping, not alertable.
`taskq.jobs.scheduled_horizon_seconds`	`s`	n/a	Seconds between now and the FURTHEST-out scheduled job's wake time (`MAX(scheduled_at) - now`, over `status='scheduled'`), sampled by every worker with `taskq.jobs.by_status`: how far ahead the future-armed wave reaches. 0.0 when nothing is scheduled; NEGATIVE when even the furthest job is overdue — every armed row has crossed its wake time and promotion is not keeping up. Label-less twin of `taskq.jobs.by_status{status="scheduled"}`, like `taskq.jobs.scheduled_count`, so the three backlog operands join without a PromQL join modifier.
`taskq.jobs.cancel_pending`	`1`	n/a	Non-terminal jobs with a cancel in flight (`cancel_phase > 0`), sampled by the leader with `taskq.queue.depth`. The durable cancel surface: a CLI cancel writes the row and exits, so this gauge is the only scrape-visible trace that the protocol owes a terminal write. Healthy drains to 0 within the cancel grace ladder plus a tick or two; a sustained non-zero reading is a cancel not finishing. Read beside `taskq.jobs.cancels_actored_total` (the worker-side counter that says the cancel was observed). Cleared to absent on demotion (never frozen at a stale level). No dimensions: the fleet total is the alertable shape; the per-job truth is on the row and the admin jobs page.
`taskq.claim.latency_p50_seconds` / `_p95_` / `_p99_`	`s`	`queue`	This worker's claim-statement latency percentiles over the recent window (computed at scrape from the worker's own bounded in-process window; `taskq.dispatch.duration`'s histogram carries the same signal for `histogram_quantile`). The MVCC-horizon predictor's distribution view: p50 moving with p99 is the whole scan degrading. Reports nothing for a queue this worker has not claimed for (empty, not zero).
`taskq.claim.claims_per_second`	`1`	`queue`	This worker's claim throughput over the recent window. Read beside the degradation ratio: a rising ratio with a collapsing rate is the claim path about to stall.
`taskq.claim.degradation_ratio`	`1`	`queue`	The live claim-latency p99 against its rolling 24h baseline (minute-bucketed p99s). THE pinned-MVCC-horizon signature: ~10x is a long transaction holding VACUUM off while the claim query's scans degenerate through dead tuples. Reports no data until the baseline has warmed (5 minutes), never a fabricated 1.0. `TaskQClaimLatencyDegraded` reads it; see the runbook.
`taskq.ratelimit.bucket_tokens`	`1`	`bucket`, `kind`	Admission budget currently left in each rate-limit bucket: `tokens_remaining` for a token bucket, remaining admissions for a sliding window — sampled by the leader every 15 s via the same peek (`RateLimitRegistry.peek_all`) the admin rate-limits page renders, so the scrape and the page can never disagree about a bucket's level. `kind` is the page's own bounded derivation: `token_bucket` / `sliding_window_log` / `sliding_window_gcra`. Statically-registered buckets only as NAMED series (bounded by the code the user ships); keyed-materialised buckets (`base_name:key`) create NO per-key series — their tokens are summed per kind under `bucket="_other_"` (the `taskq.ratelimit.reclaim_pending` precedent: keyed values never become label cardinality). A named series pinned at 0 is an exhausted admission budget; read beside `taskq.ratelimit.denials` for the rate. Cleared to absent on demotion.
`taskq.admin.sse.connections`	`1`	`topic`, `surface`	SSE streams currently open, per (topic, surface). `surface` is the closed enum `admin` (the admin UI's `/sse/{topic}` stream) or `progress` (the per-job progress stream); `topic` is the closed vocabulary the endpoints validate before the slot lookup (`queues`/`jobs`/`workers`/`history` for admin, the one constant `progress-stream` family key for progress; anything else collapses onto `_other_` at the instrument). Recorded in the process that serves the streams, so this series is exported on the admin's `/jobs/health/metrics` and never on a worker scrape. A value pinned at the configured cap with `taskq.admin.sse.rejections_total` rising is SSE saturation.

The watchdog family¶

The in-worker watchdog (taskq.worker._watchdog; workers.md: In-worker watchdog) emits one instrument per detector so a hang is attributable without a shell on the pod. All are recorded by the worker process itself; none carry identity labels.

Metric name	Kind	Unit	Attributes	Description
`taskq.worker.event_loop_lag_seconds`	histogram	`s`	n/a	Per-beat event-loop scheduling latency (above). Healthy: microseconds; a blocked loop yields one sample the length of the block once it recovers. Buckets: 5 µs – 30 s.
`taskq.worker.watchdog_loop_lag_warns_total`	counter	`1`	`detector` (`event-loop-lag-warn`)	Tier-1 lag warnings: the loop exceeded `TASKQ_WATCHDOG_LOOP_LAG_WARN_BUDGET`: once per stall (the latch clears on the next beat), with a thread dump and a deferred task-stack dump. Non-terminal.
`taskq.worker.watchdog_trips_total`	counter	`1`	`detector` (`event-loop-lag`, `stale-loop-tick`, `shutdown-deadline`)	Terminal trips: the worker force-exited with `EXIT_WATCHDOG` because the loop exceeded `TASKQ_WATCHDOG_LOOP_LAG_BUDGET`, an interval-driven loop stopped ticking, or shutdown outlived `TASKQ_TERMINATION_GRACE_PERIOD`. In-flight jobs are reclaimed by the leader on lock-lease expiry.
`taskq.worker.loop_tick_age_seconds`	gauge	`s`	`loop`	Seconds since each interval-driven sibling loop (heartbeat, producer, the leader loops, ...) last ticked: the `/ready` body's `loop_tick_ages`, exported. A loop whose age grows past its period × grace factor is the stale-loop-tick trip in the making.
`taskq.worker.loop_stall_attributions`	counter	`1`	`actor` (bounded), `kind` (`blocking_call`, `gil_held`)	Attributed event-loop stalls: each warn-tier and trip-tier stall names the registered actor whose frame sat under the work holding the interpreter, and classifies it (below).
`taskq.worker.sibling_crashes_total`	counter	`1`	`loop`	Sibling task exits by exception (never cancellations): the crash that sets the shutdown event and takes the worker down.
`taskq.worker.shutdown_duration_seconds`	histogram	`s`	n/a	Clean shutdown wall-clock time; a trip records nothing here, so a missing sample beside a `shutdown-deadline` trip is the expected shape.
`taskq.worker.loop_idle_fraction`	histogram	`1`	n/a	Sampled idle fraction of this worker's event loop, ONE SAMPLE PER HEARTBEAT WINDOW: the share of the lag watchdog's polls in the window whose event-loop thread was parked in its idle selector wait (the CPython asyncio loop's no-ready-callbacks state), published beside the windowed stall tally in the `workers` row metadata as `loop_idle`. Buckets: 0, 0.05, 0.1, 0.2 ... 0.9, 1.0 (bounded - the value is a fraction). Read its limits before alerting on it: it is a sampled proportion, not a utilisation integral.

Read them together: event_loop_lag_seconds rising → watchdog_loop_lag_warns_total → watchdog_trips_total{detector="event-loop-lag"} is one stall escalating through the tiers; loop_tick_age_seconds{loop="heartbeat"} climbing while the lag histogram stays flat is a loop that is scheduling but not ticking (blocked on an await: a pool acquire, a wedged connection), the stale-loop-tick shape.

Stall attribution: which actor is blocking the loop¶

The lag watchdog's daemon thread joins two signals while a stall persists, sampling the event-loop thread's stack every poll into a small ring (the last 16 samples):

Loop lag (the beat gap): the loop is not scheduling.
The watchdog thread's own wakeup gap: how far each poll wait overshoots the requested interval. An overshoot beyond one full extra interval means the watchdog thread itself was starved of the GIL: what synchronous work that never releases the interpreter does to a bystander thread.

Loop lag with a quiet watchdog thread classifies the stall blocking_call: a synchronous call that released the GIL (an I/O wait, a subprocess), where sys._current_frames sampling from the still-running watchdog thread lands exactly on the blocking frame. Loop lag with a starved watchdog thread classifies it gil_held: the interpreter is held (a C extension that does not detach, or a hot pure-Python loop); that sample is approximate and its line points at or just after the C call.

Each attribution is emitted as the event-loop-stall-attributed WARNING (with actor, job_id when exactly one running job matched the actor, frame as file:line:function of the deepest non-taskq frame, kind, lag_seconds, the sample count, a truncated stack, and the remedy in remedy), bumps taskq.worker.loop_stall_attributions, and records into the rolling tally the heartbeat merges into the worker's workers row metadata, which is what /admin/workers' Stall hotspots column and taskq doctor read. No span event is recorded on the running attempt's span: span operations are not thread-safe, and the attempt's span lives on the loop thread, so writing to it from the watchdog's daemon thread would risk corrupting it mid-block.

Windowed stall tally and loop idle fraction¶

The stall tally's counts are cumulative since worker boot, which answers "which actor ever stalled this process" but not "is it stalling NOW": a worker that stalled once yesterday re-publishes the same counts on every beat and reads identically to one stalling on every beat. Two windowed signals ride the SAME heartbeat publish (the same workers-row metadata write, one extra dict merge, no extra round trip), both drained per heartbeat window so the published value always describes exactly the span since the previous publish:

loop_stalls_window (metadata key beside the unchanged loop_stalls): the delta of the cumulative per-actor per-kind counts between consecutive heartbeat publishes — the per-window stall rate. Shape: {"window_seconds": <span>, "stalls": {actor: {kind: count}}}. A healed worker's stalls map is empty (or missing the actor) while its cumulative counts stand still; an active staller shows fresh counts every window. Delta counts clamp at zero, so the tally's eviction of its coldest actors can never fabricate a negative rate. A worker's FIRST publish has no predecessor window: the delta is the full cumulative tally and the span is the process age — the honest summary a fresh beat can give.
loop_idle (metadata key) and taskq.worker.loop_idle_fraction (the histogram): the fraction of the window's watchdog polls where the event-loop thread was PARKED in its idle selector wait — an idle CPython asyncio loop with no ready callbacks sits in selectors.py:select directly under base_events.py:_run_once, the exact frame shape the watchdog's existing stall sampler reads. One sample per watchdog poll (armed polls only — the same gate the beat rides), classified parked / not-parked, aggregated per window and published as {"idle_fraction", "samples", "samples_parked", "samples_unreadable", "window_seconds"}.

What the idle signal does and does not measure, stated exactly:

It is a sampled proportion, not an integral. At the default poll cadence that is ~2 binary verdicts per second; sub-second busy bursts below the sampling resolution are invisible, and the fraction carries binomial noise of order 1/sqrt(samples) (samples is published beside it — do not compare two windows with different sample counts).
It measures the loop thread's state, not job concurrency or CPU: a loop stepping many coroutines reads the same as one running a single hot callback. taskq.worker.active_jobs / max_concurrency remains the job-count view; this is the only loop-time view.
The complement is the honest read: a window whose idle_fraction is ~1.0 is a loop that held no ready work at any sampled instant (it scheduled immediately every time the watchdog looked). A LOW fraction is suggestive, not diagnostic — read it beside loop_stalls_window (stalls attributed + low idle = a loop with no spare scheduling capacity) and taskq.worker.event_loop_lag_seconds (the responsiveness view, which saturates only on real blocks).
The classifier is defined against CPython's selector-based event loop (the only loop TaskQ runs on). Any readable frame shape that is not the parked pair counts as not parked, and an unreadable frame chain (samples_unreadable) also counts as not parked — the metric biases toward busy. The idle direction is guarded: the zero-timeout select _run_once makes when ready callbacks are queued (measured: a hot call_soon spinner sits inside select(0) for nearly all of its wall time, so shape-only classification read a 100%-busy loop as ~100% parked) is classified NOT parked via the _run_once frame's timeout local; the residual over-report window is one sample where that local cannot be read.

Read all three views on one clock: the windowed tally, the idle fraction, and the lag histogram are drained/recorded at the same heartbeat publish and cover the same span.

Sweep samples: rows and duration are different populations¶

taskq.maintenance_leader.sweep_rows and taskq.maintenance_leader.sweep_duration_ms share a call site but not a sample population: a timed-out sweep records its duration and bumps taskq.maintenance_leader.sweep_timeouts, but records no row sample; the batch was aborted, so no rows were committed. Read row counts with the sweep_timeouts counter in hand before drawing conclusions from a missing or zero row sample.

Up-down counters¶

Metric name	Unit	Attributes	Description
`taskq.cron.consecutive_failures`	`1`	`actor`	Per-actor count of consecutive cron execution failures. Between ticks the deltas are `+1` per failed fire and `-count` on a successful reset, but the series is not left to accumulate those deltas: each tick reconciles it against the database's own per-actor sum over the whole `cron_schedules` table, so enables, disables, and deletes performed by any process (a client, the CLI, the admin UI; none of them emit metric deltas) self-correct on the next tick with due work, and the value returns to zero once no schedule is failing. The authoritative per-schedule counts are the `cron_schedules.consecutive_failures` column and the `cron fired` / `cron fire failed` / `cron schedule auto-disabled` logs (plus the `taskq.cron_schedule_id` attribute on the `cron fire` span); alert on `taskq.cron.disabled_schedules > 0` (the shipped `rules.yaml` alert) for the auto-disabled condition. The `actor` label is capped: the first 100 distinct names a process sees keep their series, later names collapse onto `_other_`.

Dimension cardinality¶

No metric carries worker_id, schedule_id, job_id or any other identity value as a dimension, and new instruments must not add one. The maintenance-sweep labels (sweep_name, lock, status) are bounded enums: a handful of values fixed by the code, not identity values, so they are allowed as dimensions.

The queue label on the job-side emitters (messaging.client.published.messages, messaging.client.consumed.messages, taskq.dispatch.duration, messaging.process.duration) is bounded by the code: the first 100 distinct queue names a process sees keep their own series, and every later name collapses onto the fixed _other_ value. The cap never evicts, so the series count is hard-bounded at 101 per metric regardless of how many queue names the deployment mints; unbounded, 5,000 distinct names produced 100k+ time series in the cardinality benchmark. Per-queue attribution is not lost: the queue name still rides on the enqueue/dispatch/consume span attributes and log lines, where cardinality is free. actor remains user-defined and unbounded on those emitters: keep actor names a bounded enum.

The leader-sampled gauges (taskq.queue.depth, taskq.queue.live_workers) rank by VALUE, not by first-seen admission: the 100 deepest (or most-workered) queues keep their own series and the rest collapse onto ONE _other_ series carrying their SUMMED value, so the fleet-wide total stays exact through the cap (the fleet suite proves 150 real queues serve exactly 101 series with the exact partition and exact total). That sum is bookkeeping, not an alertable shape: the two gauges admit DIFFERENT sets of 100 queues (depth ranks by pending jobs, live workers by subscribers), so their _other_ series cover different queues and a join between them - depth in _other_ with no live workers in _other_ - compares unrelated sets. TaskQQueueUnserved therefore excludes _other_ on both sides; a queue that falls off the top-100 on both gauges at once has no per-queue signal by design, and per-queue attribution for it lives on the span attributes and log lines.

The cron counter's actor label is the exception, and it is capped the same way queue is: create_schedule accepts any string actor at creation time (validation is deferred to fire time by design), and a dangling, misspelled or tenant-generated name fails every tick's planning loop (each failure emitting the raw string as a label value), so taskq.cron.consecutive_failures admits only the first 100 distinct actor names a process sees and collapses the rest onto _other_. Every other actor-labeled instrument receives actors that flowed through registration (ActorRef names on the job side, actor_config-resolved names on the cron success/suppression paths) and carries them as-is.

The bucket label on taskq.ratelimit.refund_failures is capped the same way queue is, and for the sharpest reason yet: a keyed bucket's handle name is base_name:key and the key part is caller-controlled payload data (a tenant id, a session id) with no cardinality bound, so carried as-is every distinct key ever seen would mint one never-released series for the life of the process. The first 100 distinct bucket names a process sees keep their own series and later names collapse onto _other_. Per-bucket attribution is not lost: the concrete handle name rides the ratelimit-rollback-failure log line emitted at the same catch site, where cardinality is free.

Azure Monitor counts every unique combination of metric name, dimension key and dimension value published in the last 12 hours as an active time series, caps a subscription at 50,000 of them per region, allows 10 dimension keys per metric, and documents fewer than 100 valid values per dimension as the practical ceiling (up to 300 is a grey area; beyond that, use custom logs). worker_id is a fresh UUID per worker process, so on Kubernetes every deploy, restart and autoscale event mints new series. Exceeding the cap throttles ingestion for every custom metric in the subscription, and throttled points are not backfilled: the data is gone, so this is not repairable after the fact.

Per-worker and per-schedule attribution is on the channels where cardinality is free:

Logs: worker_id is bound via contextvars onto every log line; schedule_id is on the cron fired, cron schedule auto-disabled, cron-tick-lock-contended and cron-fire-budget-deferred lines.
Spans: taskq.worker_id and taskq.cron_schedule_id are attributes of the cron fire span.

The record_* helpers still accept worker_id / schedule_id: they are part of the published taskq.obs surface and the callers already hold the value. They are not recorded as dimensions.

The one exception is taskq.maintenance_leader.is_leader, which is still labeled by worker_id: the shipped Prometheus alert (sum(taskq_maintenance_leader_is_leader) != 1) detects split-brain by summing one series per pod, so the dimension is essential to that invariant. Operators exporting to Azure Monitor should drop it with an SDK View.

Metric recording and sampling independence¶

Counters and histograms in the dispatch and consume paths are recorded outside their corresponding span bodies. This ensures that a 100% sampled span does not inflate metric counts relative to a partially-sampled trace.

Ready-made alert rules¶

The repo ships alert rules for the metrics above: import them instead of writing from scratch:

src/taskq/contrib/prometheus/rules.yaml: 24 rules (queue depth, heartbeat misses, terminal-failed share, retried-failure share, abandoned jobs, lock TTL, leader split-brain, dispatch latency, claim-latency degradation, progress failures, disabled cron, scheduled-backlog growth, promotion stall, sweep timeouts, sweep unexpected errors, sweep degraded tier, maintenance-lock contention, rate-limit dependency outage, cron lock contention, cron budget deferrals, cron skipped slots, unserved queue, stranded jobs, expired-lease zombies)
src/taskq/contrib/kubernetes/prometheus_rule.yaml: the same rules as a PrometheusRule CRD for Kubernetes

The rules fire on the series above, so they only work where those series are scraped: see Serving the metrics: the Prometheus endpoint for the (zero-config) wiring and its per-process caveat; rules that read leader-sampled gauges or worker-path counters must be evaluated against worker scrapes, not the ui serve process.

The operational "which metric catches which failure mode" table is in ops.md, Observability and alerting, and ops.md §12, Scaling playbook maps the tunable failure modes to the knob that addresses them. Where each shipped alert lands:

Alert	Playbook row (ops.md §12)	Runbook
`TaskQQueueDepthHigh`	Queue depth growing, live workers idle (or the saturation row when utilisation is high too)	none
`TaskQQueueUnserved`	Pending backlog with `stranded > 0`	TaskQQueueUnserved
`TaskQStrandedJobs`	Pending backlog with `stranded > 0`	TaskQStrandedJobs
`TaskQScheduledBacklogGrowing`	Cron piling up at tick	TaskQScheduledBacklogGrowing
`TaskQPromotionStalled`	Cron piling up at tick	TaskQPromotionStalled
`TaskQCronLockContention`	Cron piling up at tick	TaskQCronLockContention
`TaskQCronBudgetDeferrals`	the cron guide's tick-budget lever	TaskQCronBudgetDeferrals
`TaskQCronSkippedSlots`	the cron guide's tick-budget lever (same tuning surface: period, actor capacity, catch-up window)	TaskQCronSkippedSlots
`TaskQCronScheduleDisabled`	cron schedule failure, upstream of any sizing knob	none
`TaskQRateLimitDependencyOutage`	Rate-limit denials	TaskQRateLimitDependencyOutage
`TaskQRunningLeaseExpired`	`terminal-write-failed` logs / disowned jobs	TaskQRunningLeaseExpired
`TaskQHeartbeatMisses`, `TaskQLockExpiringSoon`	worker-to-Postgres health, not actor config (same row as the terminal-write failures)	none
`TaskQDispatchLatencyHigh`	Queue depth growing, live workers idle when depth also grows; otherwise PgBouncer/pool trouble (the database-health row)	none
`TaskQFailedJobRateHigh`	failure classification (ops.md §6), not a scaling knob	TaskQFailedJobRateHigh
`TaskQRetryRateHigh`	failure classification (ops.md §6)	TaskQRetryRateHigh
`TaskQAbandonedJobs`	actor ignores cancellation (ops.md §2)	TaskQAbandonedJobs
`TaskQSweepTimeouts`, `TaskQSweepDegraded`, `TaskQSweepUnexpectedErrors`, `TaskQLeaderLockContention`	maintenance sweep load, no worker knob	TaskQSweepTimeouts, TaskQSweepDegraded, TaskQSweepUnexpectedErrors, TaskQLeaderLockContention
`TaskQLeaderSplitBrainOrNoLeader`	leadership, no sizing knob	none
`TaskQProgressPublishFailures`	Redis health on the progress path, no worker knob	none

The remaining tuned-by-dashboard conditions (actor saturation, event-loop lag) ship no alert on purpose; both are read off the gauges in the playbook table.

Alert suggestions for the new series¶

The shipped rules.yaml pins the core set; these series also support alert shapes worth wiring per deployment (suggested, not shipped — tune the for: windows to your fleet's tick cadences):

The haunt: a retry loop running out of ladder. The detection pair is taskq_jobs_retry_headroom reaching 0 — an actor's worst live job is ON its last attempt:

# A job on its final attempt: page BEFORE the ceiling-writes-terminal.
min by (actor) (taskq_jobs_retry_headroom) == 0

Read it with taskq_jobs_retrying (the population): headroom 0 with a retrying count of 1 can be one legitimately-dying job; headroom pinned near 0 across ticks with a retrying count that will not drain is the retry loop — an actor whose failure is deterministic and whose max_attempts is high enough to hide in for hours. A job at attempt 6 of 1000 is invisible to every other series (it is a healthy pending row to taskq_jobs_by_status, and taskq_jobs_attempt_failures last moved five attempts ago): this pair is the only live view of the ladder.

SSE saturation. Streams pinned at the configured cap while refusals continue:

# topic's stream budget exhausted and clients still knocking
max by (topic, surface) (taskq_admin_sse_connections) == <cap>
  and rate(taskq_admin_sse_rejections_total[5m]) > 0

(<cap> is TASKQ_ADMIN_MAX_SSE_CONNECTIONS; both series are exported on the admin's /jobs/health/metrics.) Rejections alone with headroom under the cap is a burst the cap absorbed; sustained is a dashboards-open fleet or a scrape-loop client.

A cancel the protocol never finished. taskq_jobs_cancel_pending pinned above 0 past the grace ladder (plus a couple of sampling ticks) means a cancel request is stuck in flight; a simultaneous taskq_jobs_cancels_actored_total that stops moving while the gauge stays up is the worker stopped acting on it (died mid-cancel — TaskQHeartbeatMisses covers the worker side).
An exhausted rate-limit bucket. taskq_ratelimit_bucket_tokens{bucket="<bucket>"} == 0 with a rising taskq_ratelimit_denials_total rate is saturation of a NAMED bucket — capacity, not a stall (see A saturated rate limit is not a promotion stall). The _other_ series at 0 says SOME keyed bucket is exhausted; which one is on the admin rate-limits page, deliberately not on a label.

Runaway fan-out: what the operator sees first¶

A cron-triggered sync fans out to actor jobs. When those jobs take longer to clear than the cron period fires them, the schedule falls further behind every period until its next_fire_at crosses now() - TASKQ_CRON_CATCH_UP_WINDOW — and the tick's catch-up branch starts DROPPING occurrences: fires the system cannot even attempt. That skip is the system admitting it is losing the race, and it is fully instrumented so the runaway is a paged alert, never a silent loss:

What the operator sees first: TaskQCronSkippedSlots fires on a sustained taskq.cron.skipped_slots rate. The counter is in FIRE UNITS (a */5 schedule reported as 3 is three dropped fires) and never replays — the count is the record of what was dropped.
What to check next, in order:
taskq.cron.slots_behind{actor} — the depth. Climbing period over period: the fan-out is compounding. Decaying to 0 on the next fire: a one-time drop (failover, paused fleet), not a runaway.
taskq.cron.budget_deferrals — a DIFFERENT lag: budget-shaped (deferred one tick and retried, nothing dropped), not a skip. Both can fire on the same actor; their remedies are different knobs.
taskq.jobs.actor_backlog / taskq.jobs.oldest_pending_age_seconds — the job-plane pile-up behind the cron-plane drops.
The fires-vs-clearance SQL in src/taskq/insights.py (taskq.insights, the runaway predicate): jobs cleared per schedule period vs. the cron period, joined through the durable metadata->>'cron_schedule_id' stamp every cron-enqueued job carries. It names the schedule and quantifies the gap.
What the system guarantees while you investigate: every dropped slot is counted (the counter), its depth visible (the gauge, the cron missed slots skipped warning, and the cron fired line both carry skipped_slots), and the fan-out job itself lands with its schedule stamp — nothing vanishes without a trace.

4. Structured logging¶

Setup¶

worker_main calls setup_logging(level=settings.log_level, log_format=settings.log_format). setup_logging is idempotent: calling it a second time is a no-op.

Configuration¶

Variable	Default	Values
`TASKQ_LOG_FORMAT`	`json`	`json` (production), `console` (development)
`TASKQ_LOG_LEVEL`	`INFO`	Any stdlib level name: `DEBUG`, `INFO`, `WARNING`, `ERROR`
`TASKQ_LOG_EVENTS_LEVEL`	`info`	`info` (default, byte-identical to the pre-knob stream), `warning` (the failure/anomaly stream only), `off` (WARNING-and-above only), `debug` (adds the per-tick internals)

log_format rejects any value outside {"json", "console"} at WorkerSettings load time. log_events_level rejects any value outside {"info", "warning", "off", "debug"} (case-insensitive) the same way.

The event-stream verbosity knob (`TASKQ_LOG_EVENTS_LEVEL`)¶

Every state change is durably recorded in the job_events ledger in the same transaction as the state itself — the DB ledger is the audit trail. The per-job JSON log lines are a streaming duplicate of those rows, and the hot-path audit measured that duplicate at ~23% of the worker's on-CPU time. The knob suppresses the streaming duplicate only; job_events receives every event at every level (asserted end-to-end in the system tier), and the call sites stay unconditional — a structlog processor in the canonical chain (the cheapest point, before the serialization cost) applies the classification, so a call site cannot drift from the setting.

Level	`state-change` / `cancel_phase_change` / `dispatch` / lifecycle INFO	Anomaly stream (`job-failed`, `heartbeat-tick-failure`, `isolate-self-*`, reclaims, `worker-watchdog-trip`, backpressure refusals, sweep/cron failures)	`loop-lag` / `poll-cadence` internals
`info` (default)	emitted	emitted	suppressed
`warning`	suppressed	emitted (even the anomaly lines logged at INFO)	suppressed
`off`	suppressed	WARNING-and-above emitted; INFO-level anomaly lines suppressed	suppressed
`debug`	emitted	emitted	emitted

Two contracts to know:

off never blinds the operator to failures. WARNING-and-above anomaly lines emit at every level. off is for deployments that consume the events elsewhere; a failing tick still leaves a line on stderr.
Unclassified events fail open: an event not in the classification table passes at every level, so a future event can never be silently suppressed.

The knob and OTel together¶

With OTel configured, the spans carry the same operational events (dispatch spans, consumer spans, attempt spans, the span events the worker records), so warning/off keeps the traces while cutting the duplicate log stream: an OTel-instrumented deployment can run TASKQ_LOG_EVENTS_LEVEL=off and lose nothing but the log lines whose content the spans and the job_events ledger already carry — while job-failed and the watchdog trips still hit stderr. The trace_id/span_id fields the processor chain injects (see the chain below) remain the correlation key from a surviving log line back to its span.

Processor chain¶

The structlog processor chain applied to every log call (in order):

filter_by_level: the stdlib-level gate — a call below the configured level costs nothing
_events_level_filter: the TASKQ_LOG_EVENTS_LEVEL knob — drops the classified happy-path events at warning/off, the per-tick internals at every level but debug, passes everything else (including an unclassified event, which fails open). Placed here deliberately: it is the cheapest point BEFORE the serialization cost, and the call sites stay unconditional
merge_contextvars: pulls in worker_id (and any other context vars bound by the worker)
add_log_level
add_logger_name
StackInfoRenderer
TimeStamper(fmt="iso", utc=True): ISO 8601 UTC timestamp in timestamp field
_otel_span_processor: injects trace_id and span_id from the active OTel span, if any
EventRenamer("event"): ensures the event key is always event
JSONRenderer (production) or ConsoleRenderer (development)

The two filters (1 and 2) are deliberately NOT wrapped in the no-raise safety wrapper: DropEvent is a BaseException the wrapper would let through anyway, and the filters' whole job is to be the cheapest gate.

Job-context fields¶

bind_job_context returns a new immutable BoundLogger pre-bound with:

Field	Always present?
`job_id`	yes
`actor`	yes
`queue`	yes
`attempt`	yes
`trace_id`	yes (empty string when no active OTel span)
`worker_id`	via contextvars (bound once at worker startup)
`identity_key`	only when non-`None`
`span_id`	only when non-`None`
`batch_id`	only when non-`None`

span_id, identity_key, and batch_id are omitted entirely when None (they are not serialized as null).

Key log events¶

Event	Level	Kind	Key fields	When
`state-change`	info	`state_change`	`from_state`, `to_state`	Any job status transition (the event name is hyphenated, the `kind` is the snake_case ledger value)
`cancel_phase_change`	info	`cancel_phase_change`	`from_phase`, `to_phase`	Cancel phase escalation
`heartbeat-tick-success`	debug	n/a	`worker_id`, `tick_duration_ms`, `jobs_extended`, `is_leader`	Each successful heartbeat tick
`heartbeat-tick-failure`	warning	n/a	`worker_id`, `consecutive_failures`, `error_class`, `error`	Each failed heartbeat tick
`heartbeat-hook-failure`	warning	`state_change`	`worker_id`, `cause`, `error`	Cancel controller failure inside heartbeat transaction
`heartbeat-tick-unexpected-error`	error	n/a	`worker_id`	Unexpected exception in heartbeat loop
`terminal-write-retry`	warning	`terminal_write_retry`	`job_id`, `actor`, `write`, `attempt`, `max_attempts`, `retry_in_ms`, `infra_error_class`, `infra_error_message`	A terminal write hit an infrastructure error (connection reset, pool-acquire timeout) and is about to be retried: one line per retried attempt (waits of 50, 200 and 800 ms, jittered, up to four attempts)
`terminal-write-retry-budget-exhausted`	warning	`terminal_write_retry_budget_exhausted`	`job_id`, `actor`, `write`, `attempt`, `elapsed_ms`, `budget_ms`, `infra_error_class`, `infra_error_message`	The write's retries ran out of wall time (5 s from the first attempt) before they ran out of attempts: each attempt was itself slow, the shape of a black-holed Postgres, so no further retry is made and `terminal-write-failed` follows
`pool-release-failed`	warning	`pool_release_failed`	`operation`, `error`	A pooled connection's release failed or timed out (5 s bound) after an enqueue/bulk-cancel op: the pool's release-time reset against a dead or parked connection. asyncpg terminates the connection either way and the op's own result stands; the WARNING is the operator's signal that the database is killing connections (mass count during a failover or restart)
`pool-conn-dead-on-acquire`	warning	`pool_conn_dead_on_acquire`	`operation`, `error`	The pool handed an enqueue/bulk-cancel op a connection the server had already killed; its first statement failed locally with asyncpg's `InternalClientError` and the op was retried once on a fresh connection: one line per retry, the failover/restart tax an operator can count per call site
`terminal-write-failed`	error	`terminal-write-failed`	`job_id`, `actor`, `actor_succeeded`, `job_error_class`, `infra_error_class`, `infra_error_traceback`	Every attempt of a terminal write failed; the row stays `running`, the worker disowns it (its lease is no longer renewed) and lock-lease expiry reclaims it
`dispatch`	info	`dispatch`	`from_state`, `to_state`, `count`, `worker_id`, `queues`, `limit_n`	Each dispatch batch
`slot-pool-acquire-failed`	error	`slot_pool_acquire_failed`	`pool`, `acquire_timeout`, `job_id`, `error_class`, `error_message`	A job's bounded acquire from the per-slot transaction pool failed: the job is left for lock-lease reclaim, not failed
`slot-conn-terminated-transaction-in-flight`	warning	`slot_conn_terminated_transaction_in_flight`	`job_id`	A cancelled slot's connection was still inside its transaction at release time, so it was terminated instead of returned to the pool: the server rolls the transaction back, the job row stays `running`, and lock-lease expiry reclaims it
`slot-pool-release-skipped-pool-closed`	warning	`slot_pool_release_skipped_pool_closed`	`job_id`, `error_class`, `error_message`	The slot pool was closed underneath the job (credential-rotation drain or teardown); release is refused or the proxy was already released back: the connection is already gone and the job's real outcome is preserved
`slot-pool-release-skipped-conn-dead`	warning	`slot_pool_release_skipped_conn_dead`	`job_id`, `error_class`, `error_message`	The slot connection was terminated underneath the job (credential-rotation drain or teardown close-timeout): even the transaction probe cannot run on it; the terminator owns its disposal and the job's real outcome is preserved
`consume-rate-limit-denied-noop`	debug	n/a	`from_state`, `to_state`, `cause`	Reservation denied but no state transition occurred
`prune-completed`	info	`prune`	`status`, `count`, `cutoff`, `duration_ms`	Per-status completion line from the prune sweep (Sweep 5); its failures are `prune-failed` (error) and the batch rows it removed are `batches pruned` (info)
`archive-expiry-completed`	info	`archive_expiry`	`status`, `count`, `expire_before`, `duration_ms`	Per-status completion line from the archive expiry sweep (Sweep 6); its failures are `archive-expiry-failed` (error)

state_change events also carry cause, bucket_name, delay_seconds, and similar context fields depending on the transition.

Payload redaction¶

obs.redact_payload(payload) returns the first 16 hex characters of the SHA-256 digest of the JSON-serialized payload. Raw payload content is never written to logs. Use this when you want to correlate log lines with a specific payload without exposing its contents:

ctx.log.info(
    "state_change",
    kind="state_change",
    from_state="pending",
    to_state="running",
    payload_hash=obs.redact_payload(payload),
)

Exception redaction¶

Exception text reaches both spans and JSON logs, and from there whatever telemetry backend is configured. Before it does, TaskQ drops Postgres DETAIL: lines and masks credentials in URI-shaped text.

Only DETAIL is dropped. It is the line that quotes caller-supplied row values: Key (idempotency_key)=(tenant-4417) already exists., and TaskQ's idempotency_key, identity_key and fairness_key routinely hold tenant or subject identifiers. HINT: (Postgres's suggested fix) and CONTEXT: (the PL/pgSQL call stack) are structural, carry no row values, and are kept.

By default an operator sees:

duplicate key value violates unique constraint "jobs_idempotency_key_key"
HINT:  Perhaps you meant to reference the column "jobs.queue".
CONTEXT:  PL/pgSQL function taskq.enqueue(text) line 12 at SQL statement

Variable	Default	Effect
`TASKQ_EXCEPTION_REDACTION_ENABLED`	`true`	When `false`, `DETAIL:` lines are not dropped, on both the span and the log path.
`TASKQ_EXCEPTION_MESSAGE_MAX_CHARS`	`2000`	Bound on the scrubbed `error_message` carried by the `job_exception` / `job_timeout` log lines and the span's exception event; a truncated message ends with the dropped character count. The durable `ErrorInfo` row (what the admin UI and `JobRow.error_message` read) keeps the full text.

Setting it to false is a debugging aid, not a production setting. What you gain is the offending row value, usually the fastest way to identify which caller collided. What you expose is that value (a tenant or subject identifier) written to spans, to logs, and to every configured telemetry vendor, where it is retained under that vendor's policy rather than yours. The worker logs an exception-redaction-disabled WARNING on every startup while it is off, so an audit of startup logs will find it.

URI credential masking (postgresql://taskq:***@host) is not covered by this toggle and is always applied. There is no debugging case for shipping a password to a telemetry backend. The mask also covers the empty-username form (postgresql://:***@host, which is what a DSN renders when only a password is set) and password-family query parameters (?password=***).

Logging inside an actor¶

ctx.log is a structlog.stdlib.BoundLogger already bound with job_id, actor, queue, attempt, and trace_id. worker_id arrives via contextvars. You do not need to add these fields manually.

from taskq import actor
from taskq.context import JobContext


@actor
async def my_actor(payload: Payload, ctx: JobContext[Payload]) -> Result:
    ctx.log.info("processing-started", item_count=len(payload.items))
    for item in payload.items:
        ctx.log.debug("processing-item", item_id=item.id)
    return Result(...)

5. Log format examples¶

JSON (`TASKQ_LOG_FORMAT=json`)¶

{
  "event": "state_change",
  "kind": "state_change",
  "job_id": "018e1234-abcd-7000-8000-000000000001",
  "actor": "send_email",
  "queue": "default",
  "attempt": 1,
  "from_state": "running",
  "to_state": "succeeded",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "worker_id": "018e5678-ef01-7000-8000-000000000002",
  "timestamp": "2025-01-15T10:23:45.123456Z",
  "level": "info",
  "logger": "taskq.worker._consumer"
}

Console (`TASKQ_LOG_FORMAT=console`)¶

2025-01-15T10:23:45.123456Z [info     ] state_change    actor=send_email attempt=1 from_state=running job_id=018e1234-... queue=default to_state=succeeded

The console renderer uses structlog.dev.ConsoleRenderer. Field ordering and coloring depend on the structlog version.

6. Testing observability¶

Setting up a test tracer¶

taskq.testing.otel.setup_tracer creates an in-process TracerProvider backed by ListSpanExporter and patches obs.get_tracer for the duration of the test:

import pytest
from taskq.testing.otel import setup_tracer, ListSpanExporter


async def test_span_emitted(monkeypatch: pytest.MonkeyPatch) -> None:
    provider, exporter = setup_tracer(monkeypatch)

    # ... run enqueue or dispatch ...

    consumer = exporter.span_named("process my_actor")
    assert consumer is not None
    assert consumer.kind == trace.SpanKind.CONSUMER

ListSpanExporter helpers:

Method	Returns
`span_named(name)`	First `ReadableSpan` with that name, or `None`
`spans_named(name)`	All `ReadableSpan` objects with that name
`events_on(span_name, event_name)`	All events named `event_name` on the first span named `span_name`
`spans_with_kind(kind)`	All spans with the given `SpanKind`

Setting up a test meter¶

taskq.testing.otel.setup_meter creates a per-test MeterProvider backed by InMemoryMetricReader and patches the four core dispatch-path instruments:

from taskq.testing.otel import setup_meter, counter_value, histogram_points


async def test_metrics_recorded(monkeypatch: pytest.MonkeyPatch) -> None:
    reader = setup_meter(monkeypatch)

    # ... run enqueue and dispatch ...

    assert counter_value(reader, "messaging.client.published.messages") >= 1
    assert counter_value(reader, "messaging.client.consumed.messages") >= 1
    assert len(histogram_points(reader, "messaging.process.duration")) >= 1

Metric query helpers:

Helper	Returns
`collect_metrics(reader)`	All `Metric` objects from the reader
`counter_value(reader, name)`	Summed integer value for a counter
`counter_data_points(reader, name)`	`list[NumberDataPoint]` for a counter
`histogram_points(reader, name)`	`list[HistogramDataPoint]` for a histogram

Autouse fixtures¶

taskq.testing.otel exports two autouse pytest fixtures that are imported into conftest.py:

_otel_enabled_guard: snapshots and restores _otel_enabled around each test
_logging_configured_guard: resets structlog configuration and removes ProcessorFormatter handlers around each test

These fixtures run automatically for any test that imports from taskq.testing.otel. See ../api-reference/testing.md for the full fixture inventory.

Verifying trace context propagation¶

The integration tests in tests/test_otel_integration.py show the full end-to-end pattern: enqueue, manually advance the job to running, call dispatch_one_job, then assert that the consumer span carries a link whose trace_id and span_id match the producer span:

producer = exporter.span_named("enqueue _integration_test_actor")
consumer = exporter.span_named("process _integration_test_actor")

assert len(consumer.links) == 1
assert consumer.links[0].context.trace_id == producer.get_span_context().trace_id
assert consumer.links[0].context.span_id == producer.get_span_context().span_id

7. Disabling OTel¶

Set TASKQ_OTEL_ENABLED=false to suppress all span and metric creation. Structured logging is independent of this flag and remains active.

When OTel is disabled: - safe_start_span yields a NonRecordingSpan (no-op). - All record_* helpers return immediately without touching the meter. - ctx.span is None. - The unconditional counters (taskq.cancellation.requested, taskq.backpressure.errors, taskq.deadline_exceeded_sweep.jobs_failed, taskq.jobs.reclaimed, taskq.backpressure.capacity_refresh_failures) continue to record because they use the module-level instrument directly without checking the flag.

8. External OTLP collector¶

Add an OTel collector alongside the TaskQ worker services. The collector receives spans and metrics on the standard OTLP gRPC port and forwards them to your backend.

# docker-compose.yml: collector service only
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-config.yaml"]
    volumes:
      - ./otel-config.yaml:/etc/otel-config.yaml:ro
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP

Point the worker at the collector (the [otel] extra installed; the worker wires the OTLP exporters from the endpoint at startup; see Exporter configuration):

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \
OTEL_SERVICE_NAME=taskq-worker \
taskq worker --actors myapp.actors:registry
# startup log: otel-exporter-configured traces=otlp metrics=otlp logs= source=env

9. Error reporting (ErrorReporter Protocol)¶

OTel exception events and structured logs cover most error-observability needs. For error handling that needs vendor-specific routing (DLQ to Sentry, a custom alerting webhook, or an external audit log), TaskQ defines an ErrorReporter Protocol that you implement and register as a DI provider.

The Protocol¶

ErrorReporter is a typing.Protocol with a single async method:

from typing import Protocol
from taskq.backend._protocol import JobRow


class ErrorReporter(Protocol):
    async def report(self, job: JobRow, exception: BaseException) -> None: ...

The worker invokes report() after a job reaches a terminal failed state, passing the final JobRow and the exception that caused the failure. The argument order is (job, exception): matching OnRetryExhausted (see Retries: on_retry_exhausted hook). The call is fire-and-forget with respect to the job lifecycle; a failing reporter does not alter the job's terminal state.

The invocation is wrapped by invoke_error_reporter, which guards the call with asyncio.wait_for using a timeout of 3 seconds (the error_reporter_timeout default). TimeoutError and all other exceptions raised by report() are caught, logged at WARNING, and counted on the taskq.error_reporter.failures counter; they never propagate to the consumer loop.

NullErrorReporter (default)¶

When no ErrorReporter is registered, the worker uses NullErrorReporter, whose report() is a no-op. This is the default out of the box: no error reporting happens beyond OTel exception events and structured logs.

Registering a custom reporter¶

Implement the Protocol and register it as a DI provider. The worker resolves it from the ProviderRegistry at dispatch time:

from taskq.di import ProviderRegistry, Scope
from taskq.obs.error_reporter import ErrorReporter
from taskq.backend._protocol import JobRow


class SentryErrorReporter:
    """Routes terminal failures to Sentry as breadcrumbs."""

    async def report(self, job: JobRow, exception: BaseException) -> None:
        # Send to Sentry, a webhook, an audit table, etc.
        # job carries the full final row: id, actor, status, error_class,
        # error_message, error_traceback, attempt, identity_key, ...
        # exception is the BaseException that caused the terminal failure.
        ...


registry = ProviderRegistry()
# Register the Protocol type with a pre-built instance:
registry.register_value(ErrorReporter, Scope.PROCESS, SentryErrorReporter())

The reporter is resolved through the standard DI scope chain (see Dependency Injection). Use Scope.PROCESS for stateless reporters (most cases) or Scope.LOOP if the reporter holds a connection that should be reused for the event-loop lifetime.

Keep report() fast and resilient

report() runs on the worker's consume path. The library's invoke_error_reporter wrapper enforces a 3-second asyncio.wait_for timeout (the error_reporter_timeout default) and catches all exceptions, so a hanging or crashing reporter cannot block the terminal-state write indefinitely or crash the worker. Even so, keep external calls short and catch internal exceptions; a reporter that consistently times out delays terminal writes by up to 3 seconds per failure and increments the taskq.error_reporter.failures counter on each miss.

The `taskq.error_reporter.failures` metric¶

When a registered ErrorReporter.report() itself raises an exception, the worker catches it, logs a warning, and increments the taskq.error_reporter.failures counter:

Metric name	Unit	Attributes	Description	Conditional?
`taskq.error_reporter.failures`	`1`	`reporter_type`	`ErrorReporter` invocation failures.	yes

The reporter_type attribute is the class name of the reporter that raised (e.g. "SentryErrorReporter", via type(reporter).__name__). NullErrorReporter never raises, so this counter stays at zero unless a custom reporter is installed and failing.

actors.md: @actor decorator, JobContext, ctx.log, ctx.span
workers.md: worker lifecycle, WorkerSettings, pool configuration
ops.md: operations & adoption: which metrics catch which failure mode
maintenance-sweeps.md: why the sweep instruments exist and how to read them together
runbooks.md: the alerts these metrics feed, with confirm/remediate steps
../api-reference/testing.md: test fixtures, setup_tracer, setup_meter
cancellation.md: cancel phases, cancel_phase_change log events