Error Handling & Retry Logic

Commercial lease abstraction pipelines rarely execute flawlessly on the first pass. Scanned PDFs introduce layout anomalies, NLP models occasionally misclassify clause boundaries, and third-party OCR APIs throttle requests during peak ingestion windows. This page sits inside Parsing & Extraction Workflows and owns one narrow but load-bearing problem: deciding, for every failure a lease document can throw, whether to retry it, where to route it, and how to do so without corrupting a rent roll. For PropTech developers and property management operations teams, that demands far more than rudimentary try/except blocks — it requires structured error classification, deterministic backoff, and idempotent state management across thousands of documents.

The specific workflow challenge here is failure economics. A production ingestion system must immediately differentiate between transient failures — a network timeout fetching a document, a 429 from a cloud OCR provider, a momentary 5xx from the property management system (PMS) — and fatal errors such as a malformed JSON response, a fundamentally unparseable scan, or a permanent 400 schema rejection. Transient issues warrant automated recovery; retrying a fatal error only burns retry budget, inflates latency, and delays the dead-letter routing an operator needs to triage the document by hand. Get the classification wrong in either direction and the pipeline either thrashes or silently drops leases.

Where this fits in the pipeline

Error handling is a cross-cutting layer, not a stage of its own. It wraps every fallible boundary in the extraction path and decides what happens when that boundary fails:

• Upstream of it: the PDF/DOCX ingestion pipelines that fetch and normalize raw files, and the OCR preprocessing workflows that deskew scanned pages — both of which raise the transient I/O and provider errors this layer absorbs.
• This layer: classify each exception, apply backoff to recoverable ones under a strict attempt cap, enforce idempotency so a replay never double-writes, and divert the unrecoverable to a dead-letter queue with full diagnostic context.
• Downstream of it: validated abstractions flow into the canonical lease data models, while low-confidence or dead-lettered results pass through fallback routing logic toward a human review queue instead of being committed to the books.

A retry layer that also tries to parse clauses or resolve amendment precedence becomes impossible to reason about under load. It answers one question — “this operation failed; do we try again, and if not, where does the failure go?” — and hands everything else off.

Prerequisites and environment setup

The reference implementation targets Python 3.11+ with a small, vendorable dependency set so the retry wrapper can be baked into any worker image.

Install with pip install "tenacity>=8.2" "pydantic>=2.6" requests. Two environment assumptions matter. First, every external call — OCR API, metadata service, PMS write — must surface a typed exception that the classifier can read; swallowing errors into a generic Exception destroys the signal the whole layer depends on. Second, every retried operation must be idempotent at the persistence boundary: retries are guaranteed, so a commit keyed on anything other than a stable document hash will eventually double-count rent. Treat both as preconditions before wiring in any backoff.

Failure classification: the decision table

Classification is the heart of this layer, and it is best expressed as an explicit table rather than buried in branching code. Every exception maps to a category, a category maps to an action, and the action determines retry behavior. The same matrix that drives fallback routing logic at the confidence boundary drives retry routing at the failure boundary.

The distinction between “no — dead-letter” and “no — route to review” matters: a 400 is an engineering defect to triage, while a low-confidence scan is a valid document the model simply could not read confidently. Conflating them sends parseable leases to a developer’s exception log and buries real bugs in a reviewer’s queue.

Primary implementation

Below is a production-grade retry wrapper built on tenacity, configured specifically for lease document processing. It uses a typed exception hierarchy, a classification predicate, jittered exponential backoff, and structured logging for observability. Exponential backoff with jitter is what prevents the thundering-herd problem when a throttled OCR provider recovers and every worker retries in lockstep.

import logging
import random
from enum import Enum
from typing import Any, Optional, Callable
from functools import wraps

import requests
from tenacity import (
    retry, stop_after_attempt, wait_exponential_jitter,
    retry_if_exception, before_sleep_log,
)

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
)
logger = logging.getLogger("lease_abstraction.retry")


class FailureClass(str, Enum):
    """Coarse classification that determines routing."""
    TRANSIENT = "transient"   # retry with backoff
    FATAL = "fatal"           # dead-letter, do not retry
    SOFT = "soft"             # valid doc, route to manual review


class LeaseExtractionError(Exception):
    """Typed exception carrying the routing signal the pipeline needs."""
    def __init__(
        self,
        message: str,
        error_code: str,
        failure_class: FailureClass,
        document_id: Optional[str] = None,
    ):
        super().__init__(message)
        self.error_code = error_code
        self.failure_class = failure_class
        self.document_id = document_id


def _is_recoverable(exc: BaseException) -> bool:
    """Predicate tenacity calls to decide whether to retry an exception."""
    # Raw transport failures (timeouts, connection resets) are always transient.
    if isinstance(exc, (requests.exceptions.Timeout,
                        requests.exceptions.ConnectionError)):
        return True
    # Our typed errors carry their own verdict.
    if isinstance(exc, LeaseExtractionError):
        return exc.failure_class is FailureClass.TRANSIENT
    return False


def lease_retry(func: Callable) -> Callable:
    """Retry recoverable failures with jittered exponential backoff, capped at 4 attempts."""
    @retry(
        retry=retry_if_exception(_is_recoverable),
        # 2s, 4s, 8s ... capped at 60s, each with random jitter to de-correlate workers.
        wait=wait_exponential_jitter(initial=2, max=60),
        stop=stop_after_attempt(4),
        before_sleep=before_sleep_log(logger, logging.WARNING),
        reraise=True,  # surface the final exception so the caller can dead-letter it
    )
    @wraps(func)
    def wrapper(*args, **kwargs):
        return func(*args, **kwargs)
    return wrapper


@lease_retry
def fetch_lease_metadata(document_id: str, api_endpoint: str) -> dict[str, Any]:
    """Fetch lease metadata, translating HTTP status into a routing decision."""
    try:
        resp = requests.get(f"{api_endpoint}/leases/{document_id}", timeout=10)
        resp.raise_for_status()
        return resp.json()
    except requests.exceptions.HTTPError as e:
        status = e.response.status_code
        if status == 429 or status >= 500:
            # Throttle or remote fault — recover automatically.
            raise LeaseExtractionError(
                f"Recoverable upstream status {status}", str(status),
                FailureClass.TRANSIENT, document_id) from e
        # 4xx other than 429 is a permanent client/schema error — never retry.
        raise LeaseExtractionError(
            f"Permanent client status {status}", str(status),
            FailureClass.FATAL, document_id) from e
    except ValueError as e:
        # Body was not valid JSON — a fatal parse failure, not a transient blip.
        raise LeaseExtractionError(
            "Malformed JSON in metadata response", "PARSE",
            FailureClass.FATAL, document_id) from e

The wrapper deliberately re-raises the final exception (reraise=True) rather than swallowing it, so the caller can read failure_class and route deterministically. A 429 and a 503 recover transparently; a 400 or a malformed body short-circuits straight to the caller for dead-lettering, never wasting four attempts on an error that cannot improve.

Idempotency and state management

Retry logic without idempotency guarantees introduces severe data-integrity risk. When a timeout fires after the PMS has committed but before the worker receives the acknowledgement, a naive retry duplicates the abstraction and double-counts rent. The fix is to key every write on a stable identity rather than on execution order. Tie an idempotency key to a cryptographic hash of the document bytes and enforce a unique constraint on document_id + processing_version, so repeated execution converges on a single record. This is the same typed-identity contract that metadata normalization standards require at the canonical boundary, applied here to make retries safe.

A small state machine makes partial progress recoverable. Tracking each document through INGESTED → PARSING → EXTRACTED → VALIDATED → COMMITTED means a retry resumes from the last durable checkpoint instead of re-running the whole pipeline, and a crash mid-stream never leaves a half-written abstraction that corrupts a downstream financial model.

import hashlib
from pydantic import BaseModel, field_validator


class DeadLetterRecord(BaseModel):
    """Validated envelope for a non-recoverable document; the DLQ contract."""
    document_id: str
    idempotency_key: str
    error_code: str
    failure_class: str
    stack_trace: str
    diagnostic: str            # human-readable summary for the triaging operator
    raw_payload_ref: str       # pointer to stored original, never the bytes inline

    @field_validator("idempotency_key")
    @classmethod
    def key_is_content_hash(cls, v: str) -> str:
        # Reject anything that is not a 64-char SHA-256 hex digest, so the
        # DLQ can never be poisoned with an unstable, time-based key.
        if len(v) != 64 or not all(c in "0123456789abcdef" for c in v):
            raise ValueError("idempotency_key must be a SHA-256 hex digest")
        return v


def idempotency_key(document_bytes: bytes, processing_version: str) -> str:
    """Stable key: same bytes + same pipeline version => same key, every replay."""
    h = hashlib.sha256()
    h.update(document_bytes)
    h.update(processing_version.encode())
    return h.hexdigest()

Validation and quality gates

Failure handling is itself validated, not trusted. Three gates keep the layer honest:

• Typed classification at the boundary. Every fallible call raises a LeaseExtractionError carrying a FailureClass, so routing is driven by data, not by string-matching log messages. An unclassified exception is treated as fatal by default — fail closed, never silently retry the unknown.
• Confidence scoring as a soft-failure path. A document the OCR engine read with low confidence is not an error to retry; it is a SOFT failure that diverts to manual review. A sensible default review threshold is 0.80, tuned against a labeled holdout — high enough that a wrong base rent never auto-commits. Anything below it routes through fallback routing logic rather than the retry loop.
• Validated dead-letter records. The DLQ is not a dumping ground for raw exceptions. Each entry is a pydantic-validated DeadLetterRecord carrying the idempotency key, error code, stack trace, a human-readable diagnostic, and a pointer to the stored original — so an operator can triage without re-deriving context, and a replay tool can re-submit the document by key with no duplication.

Non-recoverable failures — an unparseable scan, a persistent schema mismatch — belong in the dead-letter queue immediately, never in the retry loop. When fatal exceptions accumulate beyond a defined threshold, a circuit breaker should trip to protect upstream services from cascading degradation while operators investigate. This is the same dead-letter contract the async batch processing controller emits per task; this page owns the classification and backoff that decide what reaches it.

Troubleshooting

For the specific case of scanned documents whose coordinates drift between pages — a frequent source of SOFT failures and false fatal classifications — the spatial-validation and dynamic-anchoring techniques in Handling OCR Drift and Layout Shifts in Scanned Lease Documents decide whether a page should be retried after re-preprocessing or routed straight to review.

Performance and scale notes

Backoff math is a budget, not a default. With wait_exponential_jitter(initial=2, max=60) and a four-attempt cap, worst-case latency for a single document is roughly 2 + 4 + 8 seconds of waiting plus jitter — bounded, and short enough that a stuck document doesn’t hold a worker hostage. Over a large portfolio, the dominant cost is not the retries themselves but the blocked concurrency they create: a worker sleeping through a 60-second backoff is a worker not extracting leases. Pair the retry layer with the bounded-concurrency model in async batch processing so backoff sleeps happen off the critical path and don’t pin the worker pool.

Two scale rules keep a high-volume run healthy. First, cap total attempts globally, not just per call — a portfolio-wide retry storm against a degraded provider should trip a circuit breaker rather than multiply load. Second, make the dead-letter queue cheap to drain: because every record is keyed on a content hash, a recovery job can re-submit thousands of dead-lettered documents idempotently once the root cause is fixed, with no risk of double-committing the ones that already succeeded.

Frequently asked questions

How do I tell a transient failure from a fatal one? Read the typed signal, not the log text. Transport errors (timeouts, connection resets), HTTP 429, and 5xx are transient and retry with backoff. Every other 4xx, a malformed response body, and an unparseable document are fatal and dead-letter on the first failure. Anything unclassified is treated as fatal — fail closed.

What confidence threshold should trigger manual review instead of a retry? Start at 0.80 for the aggregate extraction score and tune against a labeled holdout. A low-confidence read is a soft failure — the document is valid, the model just wasn’t sure — so it routes to the review queue through fallback routing, never into the retry loop, because retrying will not raise the confidence.

How many times should a lease extraction be retried? Four attempts with jittered exponential backoff capped at 60 seconds covers virtually all transient provider blips while bounding worst-case latency. Beyond that, additional retries rarely succeed and only delay the dead-letter routing an operator needs; trip a circuit breaker instead if failures are widespread.

How do I handle lease amendments that override base clauses when a retry replays a document? Process the amendment as its own document with its own content hash and resolve precedence downstream in the lease data models using effective dates. Because the idempotency key is the content hash, replaying the amendment is a no-op against the base lease — it never overwrites it in place, preserving the audit trail.

Related

• Handling OCR Drift and Layout Shifts in Scanned Lease Documents — spatial validation and dynamic anchoring for the scanned-document failures this layer classifies.
• Async Batch Processing — the bounded-concurrency controller whose per-task failures feed this retry and dead-letter machinery.
• OCR Preprocessing Workflows — deskew and cleanup upstream, reducing the soft failures that would otherwise route to review.
• Fallback Routing Logic — where low-confidence and dead-lettered documents divert for human review instead of auto-committing.
• Metadata Normalization Standards — the typed-identity contract that makes idempotent, replay-safe commits possible.

← Back to Parsing & Extraction Workflows

← Back to Parsing & Extraction Workflows