Partial-Extraction Recovery for Incomplete Lease Parses

Most extraction runs do not fail cleanly — they fail partially. A parser reads the base rent and term dates confidently, misses the option-notice window entirely, and returns a CAM-charge figure it is only 40% sure of. The precise decision this page resolves is what to do with that mixed result: not whether to retry the whole document, but how to keep the fields that parsed, isolate the ones that did not, and recover the gaps at the lowest possible cost. This is a distinct problem from the transient-versus-fatal classification that Error Handling & Retry Logic owns — a partial parse is not an error the operation threw; it is a successful operation whose output is incomplete, and re-running the entire parse to recover three missing fields wastes OCR budget and clobbers the fields that were already correct.

The economic case is simple. A commercial lease abstraction has thirty-plus fields; a single unreadable table cell should not send the whole document back to the queue or into manual review. Field-level recovery lets downstream systems — payment schedules, renewal calendars — start acting on the ninety percent that is known while a targeted second pass chases the remainder.

Architectural context

Partial-extraction recovery sits just after the first extraction pass and just before the canonical commit, inside Parsing & Extraction Workflows. Upstream, the OCR preprocessing workflows clean the pages and the parser emits per-field results with confidence scores; this layer reconciles those results. It leans on confidence scoring & quality gates to decide which fields clear the bar, commits the clean subset to the canonical lease data models marked partial, and routes the remaining gaps through fallback routing logic — either to a cheap targeted re-extraction or, for load-bearing fields, to human review. It answers exactly one question per document: given a field-by-field status map, what do we commit now, what do we re-extract, and what do we defer?

Field status drives the action

Recovery begins by classifying every field, not the document. Each field lands in one of four states, and the state — combined with whether the field is required — determines the action. Leading with this table keeps the logic declarative instead of buried in branches.

A field’s status is not permanent — the point of recovery is to move fields up this table. A missing required field blocks the commit only if it is still missing after the targeted re-extraction pass; a low-confidence value may cross the gate on a second engine and become committable.

Merge-precedence spec

When multiple passes or engines return the same field, the merge is deterministic, not last-writer-wins:

1. Highest confidence wins — the value with the greater score is authoritative.
2. Present beats missing — any parsed value outranks a null, regardless of score.
3. On a confidence tie within a small epsilon, a required field routes to review; an optional field takes the value from the more trusted engine per a fixed precedence order.
4. A committed high-confidence field is never overwritten by a later lower-confidence pass — recovery only fills gaps and upgrades, it never downgrades.

Recommended implementation

Model the result as a PartialLease where every field carries its own status and confidence, then express commit, re-extract, and merge as pure functions over that structure. Money is Decimal; dates are ISO-8601.

from __future__ import annotations
from decimal import Decimal
from enum import Enum
from datetime import date
from typing import Optional, Any
from pydantic import BaseModel, Field, model_validator


class FieldStatus(str, Enum):
    PRESENT_HIGH = "present_high"   # parsed, score >= gate
    PRESENT_LOW = "present_low"     # parsed, score below gate
    MISSING = "missing"            # no value produced
    CONFLICT = "conflict"          # passes disagree


class FieldResult(BaseModel):
    value: Optional[Any] = None
    confidence: float = Field(ge=0.0, le=1.0, default=0.0)
    status: FieldStatus = FieldStatus.MISSING
    source: str = "pass_1"


class PartialLease(BaseModel):
    document_id: str
    base_rent: FieldResult          # required
    term_start: FieldResult         # required
    term_end: FieldResult           # required
    cam_charges: FieldResult        # optional
    notice_window_days: FieldResult # required for renewal calendars

    REQUIRED = ("base_rent", "term_start", "term_end", "notice_window_days")
    GATE = 0.80

    def gaps(self) -> list[str]:
        """Fields that still need recovery: missing or below the gate."""
        return [
            name for name in self.model_fields
            if isinstance(getattr(self, name), FieldResult)
            and getattr(self, name).status in (FieldStatus.MISSING, FieldStatus.PRESENT_LOW)
        ]

    def blocking_gaps(self) -> list[str]:
        """Required fields not yet committable — these block a clean commit."""
        return [n for n in self.gaps() if n in self.REQUIRED]

    @model_validator(mode="after")
    def _grade(self) -> "PartialLease":
        for name in self.model_fields:
            fr = getattr(self, name)
            if not isinstance(fr, FieldResult):
                continue
            if fr.value is None:
                fr.status = FieldStatus.MISSING
            elif fr.confidence >= self.GATE:
                fr.status = FieldStatus.PRESENT_HIGH
            else:
                fr.status = FieldStatus.PRESENT_LOW
        return self

The merge combines two passes field by field, applying the precedence spec so a targeted second pass can only fill or upgrade a field, never overwrite a committed high-confidence value.

EPSILON = 0.03


def merge_field(primary: FieldResult, candidate: FieldResult,
                required: bool) -> FieldResult:
    """Best-of-confidence merge for one field across two passes."""
    # A committed high-confidence field is authoritative — never downgrade it.
    if primary.status is FieldStatus.PRESENT_HIGH:
        return primary
    # Present always beats missing.
    if candidate.value is None:
        return primary
    if primary.value is None:
        return candidate
    # Both present: disagree within epsilon on a required field -> route to review.
    if abs(primary.confidence - candidate.confidence) <= EPSILON \
            and primary.value != candidate.value and required:
        conflicted = candidate.model_copy(update={"status": FieldStatus.CONFLICT})
        return conflicted
    # Otherwise the higher-confidence value wins.
    return candidate if candidate.confidence > primary.confidence else primary

Finally the commit gate: it writes the clean subset with a partial marker, hands the gaps to recovery, and only blocks when a required field is still unrecovered.

def commit_gate(lease: PartialLease, re_extract) -> dict:
    """Commit committable fields; re-extract gaps; block only on required gaps."""
    if lease.gaps():
        recovered = re_extract(lease.document_id, lease.gaps())  # targeted, cheap
        for name in lease.gaps():
            merged = merge_field(getattr(lease, name), recovered[name],
                                 required=name in lease.REQUIRED)
            setattr(lease, name, merged)
        lease = PartialLease.model_validate(lease.model_dump())  # re-grade

    blocking = lease.blocking_gaps()
    committed = {n: getattr(lease, n).value for n in lease.model_fields
                 if isinstance(getattr(lease, n), FieldResult)
                 and getattr(lease, n).status is FieldStatus.PRESENT_HIGH}
    status = "complete" if not blocking else "partial"
    return {
        "document_id": lease.document_id,
        "record_status": status,      # downstream reads this before acting
        "committed_fields": committed,
        "deferred_fields": blocking,  # routed to review, not committed
    }

The targeted re_extract callback re-runs only the named fields — a second OCR engine on the CAM table region, a regex pass anchored to the notice clause — which costs a fraction of a full re-parse and leaves every already-clean field untouched.

Edge cases specific to commercial leases

• A required field is still missing after re-extraction. blocking_gaps() is non-empty, so the record commits as partial with base_rent and dates live but notice_window_days deferred. It never commits a null required field as if it were complete — the renewal calendar simply gets no notice date until recovery succeeds.
• Two passes return conflicting base rents. A first pass reads Decimal(“4500.00”) at 0.62 and a second reads Decimal(“4900.00”) at 0.64. The scores tie within epsilon on a required field, so merge_field marks it CONFLICT and routes to review rather than silently posting the marginally-higher-confidence figure to the books.
• Low-confidence but present, and correct. A CAM figure parses at 0.55 — below the gate but actually right. It is not committed; it goes to the review queue pre-filled with the low-confidence value, so a reviewer confirms rather than keys it from scratch.
• A partial record completes later on an amendment. A subsequent amendment supplies the missing notice window. Recovery treats the amendment as its own document; because the base record is keyed and already committed, the new field fills the gap and flips record_status to complete without rewriting the fields that were already correct.
• Downstream consumes a partial record safely. A payment schedule reads record_status and the committed field set, generates rows from base_rent and the term, and skips the deferred CAM component — proceeding on what is known and deferring the rest rather than failing the whole schedule.

When to escalate

Targeted re-extraction is not always the answer. Escalate out of the auto-recovery loop when:

• A required-field gap survives re-extraction — route it through fallback routing logic to manual review; the field-specific handling for absent metadata is covered in Implementing Fallback Routing for Missing Lease Metadata Fields.
• The same field is chronically partial across many documents — that is a systemic signal, not a per-document one. Fix it upstream in the OCR preprocessing workflows (deskew, denoise, better region cropping) rather than paying for a second pass on every lease.
• Confidence stays marginal after two engines — stop re-extracting and defer to review; more passes on an unreadable scan will not raise the score. Where the gap comes from transient provider failures rather than genuinely unreadable content, the retry approach in Exponential Backoff for OCR API Failures is the right tool instead.

Frequently asked questions

How is partial recovery different from just retrying the document? A retry re-runs the whole operation because it failed — it threw a transient error. Partial recovery runs when the operation succeeded but produced an incomplete result. Retrying the full parse wastes OCR budget on fields that already parsed correctly and risks overwriting them; targeted recovery re-extracts only the missing or low-confidence fields and merges them back.

When should a record be committed as partial instead of held back entirely? Commit as partial whenever every required field is present and above the confidence gate, even if optional fields are still missing. Holding the whole record back to chase an optional CAM figure blocks the payment schedule and renewal calendar from acting on rent and term data that is already known and correct.

How do I merge results when two extraction passes disagree? Apply a fixed precedence: highest confidence wins, present beats missing, and a committed high-confidence field is never overwritten. On a near-tie for a required field the values conflict, so route that field to review rather than guessing. A committed value only ever gets filled or upgraded, never downgraded.

Can downstream systems safely act on a partial lease record? Yes, if they read the record status and the committed field set first. A payment schedule generates rows from the committed rent and term and skips deferred components; a calendar emits the deadlines it has and waits on the ones still in review. The partial marker is the contract that lets them proceed on the known and defer the rest.

Related

• Error Handling & Retry Logic — the parent layer that classifies thrown failures; partial recovery handles the complementary case of a successful-but-incomplete result.
• Exponential Backoff for OCR API Failures — the retry path for transient provider failures, distinct from field-level gap recovery.
• Handling OCR Drift and Layout Shifts in Scanned Lease Documents — spatial techniques that reduce the missing and low-confidence fields this page recovers.
• Fallback Routing Logic — where unrecovered gaps and conflicts divert for human review instead of auto-committing.
• Implementing Fallback Routing for Missing Lease Metadata Fields — field-specific handling for the required-field gaps that block a clean commit.

← Back to Error Handling & Retry Logic

← Back to Error Handling & Retry Logic