Dead-Letter Queue Handling for Failed Lease Extractions

A dead-letter queue (DLQ) is where a lease extraction goes to be remembered rather than discarded. When a document survives its retry budget and still cannot be turned into a valid abstraction, the worst possible outcome is silent loss — a lease that never posted, a rent roll with a hole no one notices until an audit. The precise problem this page resolves is the DLQ lifecycle: what a failed async batch pipeline should route into a dead-letter queue, exactly what context to preserve alongside it, how to triage the accumulating pile, and how to redrive a fixed document back into extraction without ever committing the same rent twice.

This is deliberately not another backoff guide. Deciding whether a failure is transient (retry) or permanent (dead-letter) is the job of error handling & retry logic; that page owns the classifier and the exponential backoff loop. This page begins one step later — at the moment a failure has already been judged permanent, or has exhausted its attempts — and follows it through the envelope, the triage dashboard, the redrive, and finally retention. The DLQ is the pipeline’s institutional memory of everything it could not process, and treating it as a first-class, queryable, replayable store is what separates a recoverable outage from a lost portfolio.

Architectural context

Every stage of Parsing & Extraction Workflows that can fail permanently drains into one place. Upstream, the retry layer classifies each exception and either loops it through backoff or hands it off; when it hands off, it hands off here. A 400 schema rejection, a fundamentally corrupt PDF, or a document that has failed its fourth attempt all arrive at the DLQ carrying a verdict, not a question. Transient blips never reach it — routing a recoverable timeout into the dead-letter queue would bury a self-healing failure among genuine defects and inflate the depth metric operators alert on.

Two things must never be confused with the DLQ. Low-confidence-but-valid documents — a scan the model simply read poorly — are not failures at all; they divert through fallback routing logic to a human review queue, because a reviewer can still read them. And the redrive path leans entirely on the same content-hash identity that the idempotency patterns for S3-event-triggered lease ingestion establish at ingestion — without a stable key, replaying a dead-lettered document risks a duplicate commit. The DLQ answers exactly one question: “this document is permanently stuck; what do we need to fix it and safely try again?”

The dead-letter envelope

A DLQ entry is worthless if it only says “failed.” What makes a dead-letter queue drainable is a structured envelope that carries everything an operator or a replay tool needs to act without re-deriving context from logs. The envelope is a contract, and it should be validated on write so the queue can never be poisoned with a malformed record.

Two rules keep the envelope honest. The content_hash must be a real content hash, not a UUID or a timestamp — a time-based key would let the same document occupy the DLQ under two identities and defeat idempotent redrive. And the original bytes live behind object_key, never inline: a DLQ stuffed with base64 PDFs exhausts broker memory and turns depth alerting into noise.

Failure class, DLQ routing, and redrive strategy

Not everything that fails belongs in the DLQ, and not everything in the DLQ is redriven the same way. The table below maps the outcome of failure classification (which happens upstream in the retry layer) to whether it dead-letters and how it should be replayed once addressed.

The poison-message row is the one most systems get wrong. A document that dead-letters, gets redriven, and dead-letters again in an identical way is a poison message: left unchecked it forms an infinite loop that re-inflates the queue every drain cycle. Detecting it is simple — cap redrive_count — but the response must be to quarantine and escalate, never to keep replaying.

Recommended implementation

The two moving parts are a to_dlq() router that builds and persists the validated envelope when a failure is judged permanent, and a redrive() that re-injects a fixed document idempotently. Both hinge on the content hash. The envelope model below is deliberately distinct from the transient-classification machinery — it assumes the verdict is already “permanent” and concerns itself only with capture and replay.

import hashlib
from datetime import datetime, timezone
from typing import Optional

from pydantic import BaseModel, Field, field_validator


class DeadLetter(BaseModel):
    """Validated envelope for a permanently-failed lease extraction."""
    document_id: str
    object_key: str                       # pointer to original bytes, never inline
    content_hash: str                     # SHA-256 hex — the redrive identity
    error_class: str                      # exception type / error_code for grouping
    stack_trace: str
    retry_count: int = Field(ge=0)        # attempts spent before dead-lettering
    redrive_count: int = Field(default=0, ge=0)
    first_seen: datetime
    last_seen: datetime

    @field_validator("content_hash")
    @classmethod
    def _is_sha256(cls, v: str) -> str:
        if len(v) != 64 or any(c not in "0123456789abcdef" for c in v):
            raise ValueError("content_hash must be a 64-char SHA-256 hex digest")
        return v


def content_hash(document_bytes: bytes, processing_version: str) -> str:
    """Same bytes + same pipeline version => same key, every time."""
    h = hashlib.sha256()
    h.update(document_bytes)
    h.update(processing_version.encode())
    return h.hexdigest()


def to_dlq(dlq, *, document_id: str, object_key: str, chash: str,
           exc: Exception, retry_count: int) -> DeadLetter:
    """Route a permanent failure into the DLQ, merging with any prior entry.

    Assumes the caller has already classified `exc` as permanent (or
    attempts-exhausted); classification itself lives in the retry layer.
    """
    now = datetime.now(timezone.utc)
    existing = dlq.get(chash)                       # keyed on content hash
    if existing is not None:
        # Same document failing again — bump last_seen, don't duplicate.
        existing.last_seen = now
        existing.retry_count = max(existing.retry_count, retry_count)
        dlq.put(existing)
        return existing

    record = DeadLetter(
        document_id=document_id,
        object_key=object_key,
        content_hash=chash,
        error_class=type(exc).__name__,
        stack_trace="".join(__import__("traceback").format_exception(exc)),
        retry_count=retry_count,
        first_seen=now,
        last_seen=now,
    )
    dlq.put(record)                                 # unique on content_hash
    return record

Redrive is where idempotency earns its keep. Re-injecting a dead-lettered document must not create a second abstraction if the original commit had actually succeeded, and it must refuse to loop forever on a poison message.

POISON_THRESHOLD = 3


def redrive(dlq, pipeline, chash: str) -> dict:
    """Re-inject one dead-lettered document idempotently, keyed on its hash."""
    record = dlq.get(chash)
    if record is None:
        return {"status": "not_found", "content_hash": chash}

    # Poison-message guard: quarantine instead of looping.
    if record.redrive_count >= POISON_THRESHOLD:
        dlq.quarantine(chash)
        return {"status": "quarantined", "content_hash": chash,
                "redrive_count": record.redrive_count}

    # The pipeline commit is itself keyed on content_hash with a unique
    # constraint, so re-enqueueing a document already committed is a no-op
    # rather than a double count of rent.
    record.redrive_count += 1
    dlq.put(record)
    pipeline.enqueue(object_key=record.object_key, content_hash=chash)
    dlq.remove(chash)                               # leaves the DLQ; re-enters only if it fails again
    return {"status": "redriven", "content_hash": chash,
            "redrive_count": record.redrive_count}

The safety property is that redrive() never commits anything itself — it only re-enqueues, and the pipeline’s own hash-keyed upsert is what guarantees no double write. That is why the redrive key and the ingestion idempotency key must be the same content hash; if they diverge, a replay slips past the unique constraint and posts a duplicate. Bulk redrive after a fix is then just this function mapped over every entry sharing an error_class.

Edge cases specific to commercial leases

• Redrive causing a duplicate commit. A document that timed out after its PMS write but before the acknowledgement looks failed and gets dead-lettered, then redriven. Because the commit is an upsert keyed on content_hash, the replay converges on the existing row instead of double-counting the base rent — the single most important invariant in the whole lifecycle.
• Unbounded DLQ growth. A systemic upstream break can flood the queue with tens of thousands of entries, and a DLQ with no retention becomes a permanent liability. Apply an age-based TTL against first_seen: entries unresolved past the window move to cold archive (still recoverable, no longer counted in depth), so the live queue stays a working set, not a landfill.
• Poison-message loops. An entry that fails identically on every replay will re-enter the DLQ forever. The redrive_count cap quarantines it after a few attempts and escalates to a human, breaking the loop instead of amplifying it.
• Losing the original payload. If the envelope stored only a stack trace and not the object_key, the document could never be redriven — the failure would be diagnosable but not recoverable. Always preserve a pointer to the immutable original bytes; the abstraction is disposable, the source lease is not.
• A transient error mislabeled permanent. If the upstream classifier wrongly tags a 503 as fatal, a recoverable document lands in the DLQ. Triage catches this: a burst of same-error_class entries that all succeed on a single redrive is the signature of a misclassification, and the fix belongs back in error handling & retry logic, not in the DLQ.

When to escalate

The DLQ is an operational surface, and its depth is a leading indicator. Escalate out of the automated drain path when:

• Queue depth crosses a threshold — a sustained rise past a defined ceiling means the drain rate has fallen behind the failure rate. Alert operations and open a triage session before the backlog becomes a portfolio-wide gap.
• A single error_class spikes suddenly — a systemic defect (a changed OCR API contract, a bad deploy) is filling the DLQ faster than any manual redrive can empty it. Circuit-break the upstream stage so it stops feeding the queue, fix the root cause, then bulk-redrive by hash.
• redrive_count reaches the poison threshold — the document is not fixable by replay; route it to manual review with its full envelope so a person decides its fate.
• Entries age past the retention window — anything approaching TTL expiry needs a human decision to redrive or let it archive, so no lease is silently aged out of existence.

Frequently asked questions

What belongs in a dead-letter queue versus a retry? Only permanent failures and attempts-exhausted tasks. A transient timeout, a 429, or a 5xx is retried with backoff and never dead-lettered — routing recoverable failures into the DLQ buries self-healing blips among real defects and corrupts the depth metric operators alert on. Classification happens upstream in the retry layer; the DLQ receives only the verdict “permanent.”

How do I redrive without double-committing rent? Key both the redrive and the pipeline’s commit on the same SHA-256 content hash, with a unique constraint on the write. Redrive only re-enqueues; it never commits directly. Because the downstream upsert is hash-keyed, replaying a document whose original commit had actually succeeded converges on the existing row instead of posting a duplicate abstraction.

How do I stop a poison message from looping forever? Track a redrive_count on the envelope and quarantine any entry that reaches a small cap — three is a reasonable default. A document that fails identically on every replay is not fixable by replay, so it must be escalated to manual review rather than re-enqueued indefinitely.

How long should dead-lettered leases be retained? Apply an age-based TTL against first_seen. Keep the live window long enough to cover a realistic fix-and-redrive cycle, then move unresolved entries to cold archive — still recoverable for audit, but no longer counted in the live depth. This keeps the queue a working set rather than an unbounded liability.

Related

• Scaling Async Lease Parsing Pipelines with Celery and Redis — the distributed broker whose permanently-failed tasks feed this dead-letter queue.
• Designing Celery Task Queues for Lease Portfolio Batches — queue topology and routing that determine which tasks can end up dead-lettered.
• Idempotency Patterns for S3-Event-Triggered Lease Ingestion — the content-hash identity that makes redrive safe against double commits.
• Async Batch Processing — the bounded-concurrency controller that emits the per-task failures this queue captures.
• Error Handling & Retry Logic — the transient-versus-permanent classification and backoff that decide what reaches the DLQ in the first place.

← Back to Async Batch Processing

← Back to Async Batch Processing