Calibrating Confidence Thresholds for Lease Field Extraction

The Confidence Scoring & Quality Gates cluster gives you the architecture — a per-field score, an auto-commit gate, a review queue. It does not tell you where to draw the line. This page resolves exactly one question the architecture leaves open: given a scored extractor, what number does the base_rent gate compare against, 0.90 or 0.96, and how do you defend that choice with data rather than a hunch? The answer is a calibration procedure — build a labeled holdout, measure how precision and recall trade off per field, and read the threshold straight off a stated risk budget like “no more than one wrong base-rent auto-commit in two hundred.”

A single global cutoff is the wrong tool. A wrong tenant name is an embarrassment; a wrong base rent posts real money to a ledger and flows into every downstream schedule. Those errors do not deserve the same tolerance, so each field class gets its own threshold derived from its own cost of being wrong. Calibration is the mechanism that turns “how much is a mistake in this field worth” into a concrete floating-point cutoff.

Architectural context

Calibration sits between scoring and the gate. Upstream, the extractor emits a raw score per field — a softmax probability, an ensemble agreement ratio, or a heuristic from field mapping strategies. This procedure converts each field’s scores, measured against ground truth, into a threshold. The gate then compares live scores to those thresholds: at or above, auto-commit; below, divert through fallback routing logic to human review. It is deliberately decoupled from error handling & retry logic, which handles documents that fail to parse — calibration governs documents that parse successfully but might be quietly wrong.

The calibration spec table

The output of the whole procedure is one row per field class. You choose the middle column — the risk budget — from what a mistake costs; the last two columns fall out of the holdout. These figures are illustrative for a mixed office-and-retail portfolio with a well-behaved extractor; your numbers will differ, but the shape holds — financial fields buy safety with review load, low-stakes fields run loose.

Read a row as a sentence: “keep wrong base_rent auto-commits under one in two hundred, which forces the threshold up to 0.96, which sends about 18% of documents to review for that field.” The false-accept rate (FAR) — wrong values divided by auto-accepted values — is the number that matters, not raw accuracy. Accuracy folds in the documents you routed to review, where a human catches the error; FAR measures only the mistakes that slip through the gate unseen. That is the risk you are actually budgeting.

Recommended implementation

Given holdout rows of (score, was_correct), the algorithm is: sort the candidate thresholds, and for each, compute the FAR among the values it would auto-accept. Pick the lowest threshold whose FAR still sits at or under the target — lowest, because a higher threshold only buys safety you did not ask for at the cost of extra review. Everything here is pure standard library; money values stay in Decimal so an equality check on rent never drifts on binary-float rounding.

from __future__ import annotations
from dataclasses import dataclass
from decimal import Decimal


@dataclass(frozen=True)
class HoldoutRow:
    """One labeled extraction: the raw score the extractor emitted for a
    field, and whether the value it produced exactly matched ground truth."""
    field: str
    score: float
    was_correct: bool


@dataclass(frozen=True)
class Calibration:
    field: str
    threshold: float
    false_accept_rate: float   # wrong / auto-accepted, at or above threshold
    accept_rate: float         # auto-accepted / total
    review_load: float         # 1 - accept_rate
    support: int               # auto-accepted count backing the estimate


def calibrate_field(
    rows: list[HoldoutRow],
    target_false_accept: float,
    min_support: int = 50,
) -> Calibration | None:
    """Return the LOWEST threshold whose auto-accept false-accept rate is at
    or below target_false_accept, backed by at least min_support samples.
    Lowest threshold => least review load that still meets the risk budget."""
    total = len(rows)
    if total == 0:
        return None

    # Candidate cutoffs are the observed scores: any threshold between two
    # adjacent scores accepts the same set, so scores are the only breakpoints.
    for t in sorted({r.score for r in rows}):
        accepted = [r for r in rows if r.score >= t]
        if len(accepted) < min_support:
            continue
        wrong = sum(1 for r in accepted if not r.was_correct)
        far = wrong / len(accepted)
        if far <= target_false_accept:
            accept_rate = len(accepted) / total
            return Calibration(
                field=rows[0].field,
                threshold=t,
                false_accept_rate=far,
                accept_rate=accept_rate,
                review_load=1.0 - accept_rate,
                support=len(accepted),
            )
    return None  # target unreachable at this min_support — see "When to escalate"

Feeding it a money field looks like this — the was_correct flag is a Decimal equality, so a committed 4500.00 only counts as correct when it matches the human-verified value to the cent:

truth = {"lease-8841": Decimal("4500.00"), "lease-8842": Decimal("3120.50")}
raw = [
    ("lease-8841", 0.972, Decimal("4500.00")),
    ("lease-8842", 0.883, Decimal("3120.05")),  # OCR dropped a digit: wrong
]
base_rent_rows = [
    HoldoutRow("base_rent", score, committed == truth[doc])
    for doc, score, committed in raw
]

cal = calibrate_field(base_rent_rows, target_false_accept=0.005)
# Calibration(field='base_rent', threshold=0.96, false_accept_rate=0.004, ...)

Check the score is calibrated first

The procedure above assumes the score means something — that documents scored 0.96 are wrong about 4% of the time. Raw softmax outputs and heuristic scores routinely violate this; a model can be 99% confident and 80% right. Before trusting any threshold, bin the scores and compare the mean score in each bin to the observed accuracy. A wide, systematic gap means the score is not a probability, and the tidy 0.96 you read off is arbitrary.

def reliability_bins(rows: list[HoldoutRow], n_bins: int = 10):
    """Empirical calibration curve: bucket by score, compare mean predicted
    score to observed accuracy. Large, consistent gaps mean the raw score
    needs isotonic or Platt scaling before a threshold can be read off it."""
    bins: list[list[HoldoutRow]] = [[] for _ in range(n_bins)]
    for r in rows:
        bins[min(int(r.score * n_bins), n_bins - 1)].append(r)
    out = []
    for b in bins:
        if not b:
            continue
        mean_score = sum(r.score for r in b) / len(b)
        observed = sum(1 for r in b if r.was_correct) / len(b)
        out.append((round(mean_score, 3), round(observed, 3), len(b)))
    return out  # [(0.93, 0.91, 140), (0.97, 0.98, 210), ...]

The two industry-standard fixes are Platt scaling — fit a one-parameter logistic that squashes scores toward observed frequencies — and isotonic regression — fit a monotonic step function that maps each score band to its empirical accuracy. Both are conceptually “learn a correction curve from (score, was_correct) and apply it before the gate.” Isotonic is more flexible and needs more data; Platt is robust on small holdouts. You do not need a heavy dependency to start — a monotonic pass over the reliability_bins output gives a serviceable isotonic-style map. Calibrate the score, then run calibrate_field on the corrected scores.

Edge cases specific to commercial leases

• Tiny holdout. Below roughly 50 auto-accepted samples, a FAR of “0 wrong out of 40” is not evidence the field is safe — it is noise. min_support guards the point estimate; for the decision itself, use the upper bound of a Wilson interval on the FAR, not the raw fraction, so a lucky holdout does not talk you into a reckless threshold.
• Class imbalance. A field that is correct 99% of the time makes every threshold look excellent on FAR while hiding that you auto-commit almost nothing useful. Always report accept_rate alongside FAR; a threshold that meets the budget but accepts 3% of documents is not a win, it is review-queue overflow in disguise.
• Per-property-type drift. Office, retail, and industrial leases word their rent and CAM clauses differently, so a single blended threshold can pass in aggregate while quietly failing on retail. Calibrate per (field, property_type) segment when the holdout supports it; the aggregate number hides the segment that is actually bleeding.
• Overfitting the threshold. If you tune 0.96 on the same holdout you report FAR from, you have fit the noise. Hold out a separate test fold, round thresholds to two decimals, and never chase a 0.001 improvement — that precision is not real.
• Score not calibrated. Covered above, and it is the most common failure: teams read a threshold off an uncalibrated score and are baffled when production FAR is triple the holdout’s. Run reliability_bins first, every time.

When to escalate

calibrate_field returning None is a signal, not a bug: it means you cannot hit the target false-accept rate at any threshold with enough support — the extractor simply is not accurate enough on this field, and no cutoff will rescue it. Escalate out of pure threshold-tuning when:

• The target is unreachable at acceptable review load. If meeting 0.5% FAR on base_rent requires a 0.99 threshold that sends 60% of documents to review, the fix is upstream, not the gate. Tighten extraction — better OCR, stricter field mapping strategies, an ensemble — or route that field’s low-confidence cases through fallback routing logic by design rather than fighting the number.
• Production FAR drifts above the holdout’s. New document formats or a new extractor version invalidate the calibration. Treat a rising live FAR the way you treat rising parse failures in error handling & retry logic: alert, freeze auto-commit for the affected field, and re-calibrate.
• Re-calibration cadence. Re-run the procedure on a monthly cadence and on every trigger — a new property type in the portfolio, a swapped OCR engine, or a model retrain. Document mix drifts continuously; a threshold set in Q1 is a guess by Q3.

Frequently asked questions

What false-accept rate should base rent use? Around 0.5% — no more than one wrong auto-committed base rent in two hundred — because the value posts money and propagates into every downstream payment and accounting schedule. Set it from what a wrong rent costs to detect and reverse, not from a default.

How big does the holdout need to be? Enough that each field has at least 50 auto-accepted samples above its candidate threshold, and ideally several hundred, so the false-accept estimate is stable. With a small holdout, decide on the upper bound of a Wilson interval rather than the raw fraction.

Do I need one threshold for the whole document or one per field? One per field class. A wrong tenant name and a wrong base rent carry wildly different costs, so they get different risk budgets and therefore different thresholds. A single global cutoff over-reviews cheap fields and under-protects expensive ones.

How often should I re-calibrate? Monthly as a baseline, and immediately whenever the document mix shifts — a new property type, a new OCR engine, or a retrained extractor. Watch live false-accept rate as the trigger; when it climbs above the holdout’s, the old threshold is stale.

Related

• Confidence Scoring & Quality Gates — the parent architecture this procedure feeds: the scoring model, the gate, and the review queue that consume the thresholds you calibrate here.
• Error Handling & Retry Logic — governs documents that fail to parse; calibration governs documents that parse but may be wrong.
• Handling OCR Drift and Layout Shifts in Scanned Lease Documents — a common cause of the score drift that forces re-calibration.
• Fallback Routing Logic — where below-threshold fields divert for human review instead of auto-committing.
• Field Mapping Strategies — the extraction layer to tighten when a field cannot hit its target false-accept rate at an acceptable review load.

← Back to Confidence Scoring & Quality Gates

← Back to Confidence Scoring & Quality Gates