Exponential Backoff for OCR API Failures

When a lease-abstraction pipeline calls a cloud document-AI service — AWS Textract, Azure Document Intelligence, or Google Document AI — the call fails often enough that “just retry” is not a strategy, it is a liability. This page sits inside Error Handling & Retry Logic and resolves one narrow decision: given a transient OCR API failure, exactly how long should a worker wait before the next attempt, how many attempts and how much total time is it allowed to spend, and when should it stop calling a degraded provider altogether? Naive fixed-interval retries either give up too early on a throttled service that would have recovered, or — far worse — synchronize every worker into a retry wave that keeps a rate-limited endpoint pinned in the red.

The failure economics are specific to remote OCR. A 429 Too Many Requests during a peak ingestion run is recoverable in seconds if you back off politely, but a 400 UnsupportedDocumentException on a corrupt scan will fail identically on the tenth attempt as on the first. The retry layer’s job is to spend its budget only where waiting actually helps, and to route everything else straight out of the loop. This guide differs deliberately from Handling OCR Drift and Layout Shifts in Scanned Lease Documents: that page fixes quality failures where the API succeeds but returns misaligned coordinates; this page fixes availability failures where the API call itself does not complete.

Architectural context

This backoff logic wraps exactly one boundary in the Parsing & Extraction Workflows domain — the network call from a worker to a third-party OCR provider — and nothing else. Upstream, the OCR preprocessing workflows deskew and clean pages before they are submitted, and the async batch processing controller decides how many of these calls run concurrently. When a call exhausts its budget, the document does not vanish: it flows to dead-letter queue handling for failed lease extractions, and a sustained provider outage that trips the circuit breaker escalates to fallback routing logic so a human is alerted rather than a queue silently draining into failure. The parent cluster owns the taxonomy of transient-versus-fatal; this page owns the timing math and the breaker that act on it.

Which OCR errors are retryable

Retrying is only correct when a later attempt could plausibly succeed without any change to the input. That rules out anything caused by the document or the request itself. The table below is the contract the classifier enforces for the three major providers; the status codes are stable across them even though the exception class names differ.

A 401 is worth calling out: it looks transient because it “might work later” once someone rotates a key, but nothing the pipeline can do at runtime fixes it, so it must alert rather than burn the budget silently.

The backoff schedule

Full jitter means the sleep is drawn uniformly from [0, min(cap, base * 2**attempt)] rather than being a fixed exponential value. The randomness is the whole point: it de-correlates a fleet of workers that all failed at the same instant, so a recovering provider sees a smeared trickle of requests instead of a synchronized thundering herd. With base = 1.0s and cap = 30.0s:

Worst-case cumulative wait across six attempts is bounded near 61 seconds, but the expected wait is roughly half that because each sleep averages the midpoint of its range. The attempt cap alone is not enough — a run of maximum draws could still blow past an SLA — so the implementation below also enforces a wall-clock total_budget and stops at whichever limit is hit first.

Recommended implementation

The retry loop is deliberately explicit rather than a decorator hiding the math, because OCR retries need three things a generic @retry rarely exposes together: full jitter, a Retry-After override that supersedes the computed sleep, and a shared attempt-plus-time budget. It returns the typed FailureClass from the parent Error Handling & Retry Logic cluster so downstream routing stays consistent.

import random
import time
import logging
from dataclasses import dataclass, field
from enum import Enum
from typing import Any, Callable, Optional

logger = logging.getLogger("lease_abstraction.ocr_backoff")

RETRYABLE_STATUS = frozenset({429, 500, 502, 503, 504})
PERMANENT_STATUS = frozenset({400, 401, 403, 404, 413, 415, 422})


class OCRAPIError(Exception):
    """Raised by the provider adapter, carrying the routing signal."""
    def __init__(self, message: str, status: Optional[int],
                 retry_after: Optional[float] = None, document_id: str = ""):
        super().__init__(message)
        self.status = status
        self.retry_after = retry_after   # seconds, parsed from Retry-After header
        self.document_id = document_id


class RetryOutcome(str, Enum):
    SUCCESS = "success"
    PERMANENT = "permanent"          # 4xx -> dead-letter, never retried
    BUDGET_EXHAUSTED = "exhausted"   # transient but out of attempts/time -> dead-letter
    CIRCUIT_OPEN = "circuit_open"    # provider degraded -> short-circuit -> dead-letter


@dataclass
class RetryBudget:
    max_attempts: int = 6
    total_budget_s: float = 90.0     # wall-clock ceiling across all attempts
    base_delay_s: float = 1.0
    cap_delay_s: float = 30.0
    started_at: float = field(default_factory=time.monotonic)

    def full_jitter(self, attempt: int) -> float:
        ceiling = min(self.cap_delay_s, self.base_delay_s * (2 ** attempt))
        return random.uniform(0.0, ceiling)

    def time_left(self) -> float:
        return self.total_budget_s - (time.monotonic() - self.started_at)


def _classify(exc: OCRAPIError) -> bool:
    """True if the OCR error is transient and worth another attempt."""
    if exc.status in PERMANENT_STATUS:
        return False
    if exc.status in RETRYABLE_STATUS or exc.status is None:
        return True   # None == raw timeout / connection reset
    return False       # unknown status: fail closed, treat as permanent


def call_ocr_with_backoff(
    fn: Callable[[], dict[str, Any]],
    breaker: "CircuitBreaker",
    budget: Optional[RetryBudget] = None,
    document_id: str = "",
) -> tuple[RetryOutcome, Any]:
    """Invoke an OCR provider call under jitter, a retry budget, and a breaker."""
    budget = budget or RetryBudget()

    if not breaker.allow():
        logger.warning("Circuit open for OCR provider; skipping %s", document_id)
        return RetryOutcome.CIRCUIT_OPEN, None

    last_exc: Optional[OCRAPIError] = None
    for attempt in range(budget.max_attempts):
        try:
            result = fn()
            breaker.record_success()
            return RetryOutcome.SUCCESS, result
        except OCRAPIError as exc:
            last_exc = exc
            if not _classify(exc):
                logger.warning("Permanent OCR error %s on %s: %s",
                               exc.status, document_id, exc)
                breaker.record_success()   # provider is healthy; the doc is not
                return RetryOutcome.PERMANENT, exc

            breaker.record_failure()
            # Provider's Retry-After wins over our computed sleep, if present.
            sleep_s = exc.retry_after if exc.retry_after is not None \
                else budget.full_jitter(attempt)

            remaining = budget.time_left()
            if attempt == budget.max_attempts - 1 or sleep_s >= remaining:
                logger.error("Retry budget exhausted for %s after %d attempts",
                             document_id, attempt + 1)
                return RetryOutcome.BUDGET_EXHAUSTED, exc

            logger.info("Transient OCR %s on %s; sleeping %.2fs (attempt %d)",
                        exc.status, document_id, sleep_s, attempt + 1)
            time.sleep(sleep_s)

    return RetryOutcome.BUDGET_EXHAUSTED, last_exc

Note the two subtleties that make this safe for money-bearing lease data. A PERMANENT error records a breaker success, because the provider is healthy — it is the document that is wrong, and counting it as a provider fault would trip the breaker on a batch of corrupt scans. And Retry-After supersedes the jittered value entirely: when Textract tells you to wait 20 seconds, guessing 3 seconds only earns another 429.

A minimal circuit breaker

The breaker stops the pipeline from hammering a provider that is already down. After a threshold of consecutive failures it opens, short-circuiting every call to the dead-letter path for a cool-down window, then allows a single trial request (half-open) to test recovery before fully closing.

class CircuitState(str, Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"


class CircuitBreaker:
    def __init__(self, fail_threshold: int = 5, reset_timeout_s: float = 60.0):
        self.fail_threshold = fail_threshold
        self.reset_timeout_s = reset_timeout_s
        self._failures = 0
        self._state = CircuitState.CLOSED
        self._opened_at = 0.0

    def allow(self) -> bool:
        if self._state is CircuitState.OPEN:
            if time.monotonic() - self._opened_at >= self.reset_timeout_s:
                self._state = CircuitState.HALF_OPEN   # let one probe through
                return True
            return False
        return True

    def record_success(self) -> None:
        self._failures = 0
        self._state = CircuitState.CLOSED

    def record_failure(self) -> None:
        self._failures += 1
        if self._failures >= self.fail_threshold:
            self._state = CircuitState.OPEN
            self._opened_at = time.monotonic()
            logger.critical("OCR circuit breaker OPEN after %d failures", self._failures)

Edge cases specific to commercial lease OCR

• Thundering herd without jitter. If a hundred workers all hit a 429 at 14:00:00 and back off by a fixed 2 ** attempt, they retry in perfect lockstep at 14:00:02, re-throttling the provider. Full jitter smears those retries across the whole window; dropping the jitter to save “wasted” early wait is the single most common cause of a self-inflicted outage.
• Retry-After larger than the budget. A provider under heavy load may return Retry-After: 120 while your total_budget_s is 90. Honoring it would blow the SLA, so the loop treats sleep_s >= remaining as budget-exhausted and dead-letters immediately rather than sleeping past the deadline — the document is re-queued later, not held hostage on a worker.
• Non-idempotent OCR calls retried. Asynchronous Textract StartDocumentAnalysis starts a job and bills per page; blindly retrying a timed-out start can launch two jobs for one lease and double the cost. Retry only after confirming no JobId was returned, or make the start idempotent with a client-supplied token keyed on the document hash — the same content-hash identity used across async batch processing.
• Partial page-batch failure. A 60-page scanned lease submitted as one job can succeed for 55 pages and fail three with a transient fault. Retrying the whole document re-OCRs the 55 good pages and re-bills them; track per-page results and retry only the failed page range, then merge.
• 429 storms across tenants. A provider quota is account-wide, so one tenant’s bulk backfill can throttle every other tenant’s real-time calls. The breaker is shared per provider account, not per document, so a storm trips it once and all in-flight leases short-circuit cleanly to the dead-letter queue instead of each independently exhausting its budget.

When to escalate

Backoff and the breaker are containment, not resolution. Escalate out of the retry loop when:

• The retry budget is exhausted on a genuinely transient error — the document is not corrupt, the provider is just unavailable right now. Route it to dead-letter queue handling for failed lease extractions with the last status code and attempt count, so a drain job can re-submit it idempotently once the provider recovers.
• The breaker stays open past one or two reset windows — that is a sustained outage, not a blip. Fire an alert and divert affected documents through fallback routing logic to a manual-review or alternate-provider path, rather than letting the queue quietly drain into dead letters unattended.
• Permanent errors spike on a normally clean feed — a wave of 400/415 usually means an upstream change (a new export format) that belongs in OCR preprocessing workflows, not the retry layer.

Frequently asked questions

Which OCR API errors should I actually retry? Only 429 rate limits, 5xx server errors, and raw connection timeouts or resets — cases where an identical later request could succeed. Every other 4xx (400, 401, 403, 413, 415, 422) and any malformed response body is permanent and must dead-letter on the first failure, because the input or configuration is the problem and waiting changes nothing.

Why full jitter instead of plain exponential backoff? Plain 2 ** attempt backoff keeps every worker that failed at the same moment synchronized, so they all retry together and re-throttle the recovering provider. Full jitter draws each sleep randomly from zero up to the exponential ceiling, spreading retries across the window so the provider sees a smooth trickle instead of a thundering herd.

Should I honor the Retry-After header over my own backoff? Yes. When the provider sends Retry-After, it is telling you exactly when capacity returns; your computed jitter is only a guess and a shorter guess just earns another 429. Use Retry-After when present, fall back to full-jitter backoff when it is absent, and if the requested wait exceeds your remaining time budget, dead-letter the document to retry later rather than sleeping past the deadline.

What does the circuit breaker add on top of a retry budget? The retry budget bounds one document’s attempts; the breaker bounds the whole pipeline’s pressure on a degraded provider. After a run of consecutive failures it opens and short-circuits new calls straight to the dead-letter path for a cool-down, so a thousand in-flight leases do not each independently burn a full budget against a service that is already down.

Related

• Error Handling & Retry Logic — the parent guide that owns the transient-versus-fatal taxonomy this backoff logic acts on.
• Handling OCR Drift and Layout Shifts in Scanned Lease Documents — the quality-failure sibling: what to do when the API succeeds but the coordinates are wrong.
• Partial Extraction Recovery for Incomplete Lease Parses — recovering usable fields when only part of a document extracts cleanly.
• Dead-Letter Queue Handling for Failed Lease Extractions — where budget-exhausted and breaker-open OCR calls land for triage and idempotent replay.

← Back to Error Handling & Retry Logic

← Back to Error Handling & Retry Logic