Confidence Scoring & Quality Gates

Confidence scoring is the mechanism that lets a lease abstraction system decide, without a human in the loop, whether an extracted record is trustworthy enough to post to a rent roll. It sits inside Parsing & Extraction Workflows, immediately after the hybrid extractor has produced candidate fields and immediately before anything is written to the property database. The extractor’s job is to guess values; this stage’s job is to attach a defensible number to each guess, fold those numbers into a single document-level score, and route the document deterministically to one of three destinations — auto-commit, manual review, or a dead-letter queue. The specific problem solved here is turning a fuzzy pile of per-field signals into a repeatable, auditable accept/reject decision that a compliance team can later defend.

The failure this prevents is the silent one. A pipeline with no scoring layer treats a base rent read at 60% certainty exactly like one read at 99% certainty, and the wrong figure posts money before anyone notices. A pipeline with naive scoring — a single mean over all fields — is barely better, because it lets a perfectly extracted tenant name and address paper over a garbled rent figure. Everything below is the engineering of a gate that refuses to be fooled that way: where confidence signals actually come from, how to aggregate them so high-risk financial and temporal fields dominate, how to calibrate the accept threshold against labeled data, and how to route everything that falls short into fallback routing logic rather than best-effort guessing.

Prerequisites and environment setup

The reference implementation is deliberately small and pure — a scoring layer should be cheap enough to run on every document and easy enough to unit-test that its thresholds can be tuned with confidence. It targets Python 3.11+ and pydantic v2, with Decimal for money and ISO-8601 dates throughout.

Install with pip install "pydantic>=2.6". Two assumptions carry through the rest of the page. First, every signal that enters the scorer is already normalized to the closed interval [0.0, 1.0] — a raw OCR engine reporting a 0–100 integer or a model returning an un-normalized logit must be mapped before it reaches FieldConfidence, and the pydantic model enforces the range so an out-of-band value fails loudly rather than skewing an aggregate. Second, the scorer never sees raw extraction strings alone; it sees the parsed, typed candidate produced upstream by the extractor, because a cross-field consistency check like “does the implied annual rent match the stated annual rent” is only meaningful once base_rent_monthly is a Decimal, not the string "$8,450/mo". That typed-candidate contract is the same one enforced at the canonical boundary by metadata normalization standards.

Where confidence actually comes from

“Confidence” is not one number the extractor hands you; it is a fusion of several independent signals, each measuring a different way a field can be wrong. Treating them as interchangeable is the first mistake. The scoring layer’s job is to collect the ones that are available for a given field and combine them into a single per-field score.

• OCR character confidence. Every OCR engine emits a per-glyph or per-token confidence. For a scanned lease, the minimum character confidence across the tokens that make up a field is a hard ceiling on how much you can trust that field — if the rent figure sits on a page where OCR was unsure about the “4” versus “9”, nothing downstream can recover it. The OCR preprocessing workflows that produce this signal also decide whether a page is legible enough to attempt extraction at all.
• Model logits / softmax probability. A named-entity or clause classifier emits a probability distribution; the softmax mass on the chosen label is a confidence signal. It must be read with caution — modern neural models are frequently overconfident, reporting 0.99 on wrong answers — which is exactly why calibration against a holdout, discussed below, is non-negotiable before you trust a logit as a probability.
• Regex match strength. A deterministic match is not automatically certain. An anchored, fully-specified pattern (Base Rent: \$([\d,]+\.\d{2}) per month) that consumes a labeled line is strong; a loose numeric grab that matched the first dollar amount on the page is weak, even though both “succeeded”. Encoding that distinction as a score keeps the regex & NLP clause extraction layer honest about how it found a value.
• Cross-field consistency checks. Independent fields constrain each other. If base_rent_monthly × 12 is within a small tolerance of a separately extracted base_rent_annual, both figures gain confidence; if they disagree by an order of magnitude, both lose it. Lease term ordering (expiration after commencement), square-footage-to-rate reconciliation, and escalation-step monotonicity are all consistency signals, not just validation errors.
• Validation-rule pass/fail. A hard rule — a rent that is negative, a term that runs backward, a required field that is null — is a binary signal that should collapse the affected field’s confidence toward zero and, for a hard failure, short-circuit the whole gate to dead-letter regardless of the aggregate.

The design principle is that these signals floor each other rather than average within a field: a field is only as trustworthy as its weakest applicable signal. A rent value with a strong regex match but low OCR character confidence underneath is a low-confidence field, because the regex matched whatever the OCR mis-read.

Aggregating field scores into a document score

Once every field carries a score, the gate needs one number for the document. The naive choice is the arithmetic mean, and it is dangerous: a lease with nine cleanly-read descriptive fields and one garbled base rent averages out to something comfortably above any reasonable threshold, and the garbled rent auto-commits. The two defensible strategies are the minimum and a risk-weighted mean, and production systems use a hybrid of both.

The minimum is maximally conservative — the document is only as good as its worst field — and it is the right rule when every field matters equally. But leases do not weight fields equally. A misread suite number is an annoyance; a misread base rent or commencement date moves money and triggers the wrong billing schedule. So the aggregate is computed as a weighted mean in which financial and temporal fields carry heavier weights, combined with a hard per-field floor on those same high-risk fields so a weighted-away low score can never be silently absorbed. The decision table below fixes the field classes, their aggregation weights, and their individual review floors:

The weighted mean sets the document score c; the per-field floors are an override that no weighting can drown out. This is the concrete answer to the design rule that a pipeline is only as trustworthy as its weakest money-bearing field — the weighting decides how much the document as a whole is trusted, and the floors guarantee a single high-risk field can independently veto an auto-commit.

Primary implementation

Below is a production-grade scorer: a pydantic FieldConfidence that fuses the raw signals into one per-field score, an ExtractionConfidence that holds the whole document, and a gate() function that computes the risk-weighted aggregate, enforces the per-field floors, and returns a deterministic routing decision. It is pure and side-effect-free by design so the thresholds can be unit-tested and calibrated in isolation.

from __future__ import annotations

from decimal import Decimal
from enum import Enum
from typing import Optional

from pydantic import BaseModel, Field, field_validator, model_validator


class FieldClass(str, Enum):
    FINANCIAL = "financial"
    TEMPORAL = "temporal"
    IDENTITY = "identity"
    DESCRIPTIVE = "descriptive"


class Routing(str, Enum):
    AUTO_COMMIT = "auto_commit"
    MANUAL_REVIEW = "manual_review"
    DEAD_LETTER = "dead_letter"


# Risk weights and per-field floors, keyed by field class. Financial and
# temporal fields dominate the aggregate and can independently veto a commit.
CLASS_WEIGHT: dict[FieldClass, float] = {
    FieldClass.FINANCIAL: 3.0,
    FieldClass.TEMPORAL: 3.0,
    FieldClass.IDENTITY: 1.5,
    FieldClass.DESCRIPTIVE: 1.0,
}
CLASS_FLOOR: dict[FieldClass, float] = {
    FieldClass.FINANCIAL: 0.90,
    FieldClass.TEMPORAL: 0.90,
    FieldClass.IDENTITY: 0.75,
    FieldClass.DESCRIPTIVE: 0.50,
}


class FieldConfidence(BaseModel):
    """Fuses the independent signals for one field into a single score.

    Each signal lives on [0, 1]; the fused score is the *minimum* of the
    signals present, because a field is only as trustworthy as its weakest
    applicable evidence. A hard validation failure zeroes the field.
    """

    name: str
    field_class: FieldClass
    ocr_char: Optional[float] = Field(default=None, ge=0.0, le=1.0)
    model_prob: Optional[float] = Field(default=None, ge=0.0, le=1.0)
    regex_strength: Optional[float] = Field(default=None, ge=0.0, le=1.0)
    consistency: Optional[float] = Field(default=None, ge=0.0, le=1.0)
    hard_rule_failed: bool = False

    @property
    def score(self) -> float:
        if self.hard_rule_failed:
            return 0.0
        signals = [
            s
            for s in (self.ocr_char, self.model_prob, self.regex_strength, self.consistency)
            if s is not None
        ]
        return min(signals) if signals else 0.0

    @property
    def below_floor(self) -> bool:
        return self.score < CLASS_FLOOR[self.field_class]


class ExtractionConfidence(BaseModel):
    """Document-level confidence: a risk-weighted aggregate over its fields."""

    document_hash: str = Field(min_length=12)
    fields: list[FieldConfidence]

    @field_validator("document_hash")
    @classmethod
    def hash_is_hex(cls, v: str) -> str:
        int(v, 16)  # raises ValueError on non-hex content
        return v

    @model_validator(mode="after")
    def has_money_and_dates(self) -> "ExtractionConfidence":
        classes = {f.field_class for f in self.fields}
        missing = {FieldClass.FINANCIAL, FieldClass.TEMPORAL} - classes
        if missing:
            raise ValueError(f"no {', '.join(c.value for c in missing)} field scored")
        return self

    @property
    def aggregate(self) -> float:
        """Risk-weighted mean of field scores; high-risk classes dominate."""
        num = sum(f.score * CLASS_WEIGHT[f.field_class] for f in self.fields)
        den = sum(CLASS_WEIGHT[f.field_class] for f in self.fields)
        return num / den if den else 0.0

    @property
    def any_hard_failure(self) -> bool:
        return any(f.hard_rule_failed for f in self.fields)

    def high_risk_below_floor(self) -> list[str]:
        return [
            f.name
            for f in self.fields
            if f.field_class in (FieldClass.FINANCIAL, FieldClass.TEMPORAL)
            and f.below_floor
        ]


def gate(
    doc: ExtractionConfidence,
    accept: float = 0.92,
    review_floor: float = 0.70,
) -> tuple[Routing, str]:
    """Deterministic routing decision. Order matters: hard failures and
    high-risk floor breaches are checked BEFORE the aggregate, so a single
    bad money field can never be averaged into an auto-commit."""
    if doc.any_hard_failure:
        return Routing.DEAD_LETTER, "hard validation rule failed"

    c = doc.aggregate
    breached = doc.high_risk_below_floor()

    if c < review_floor:
        return Routing.DEAD_LETTER, f"aggregate {c:.3f} below review floor {review_floor}"
    if breached:
        return Routing.MANUAL_REVIEW, f"high-risk field(s) below floor: {', '.join(breached)}"
    if c >= accept:
        return Routing.AUTO_COMMIT, f"aggregate {c:.3f} at/above accept {accept}"
    return Routing.MANUAL_REVIEW, f"aggregate {c:.3f} in review band"


if __name__ == "__main__":
    doc = ExtractionConfidence(
        document_hash="a1b2c3d4e5f6",
        fields=[
            FieldConfidence(name="base_rent_monthly", field_class=FieldClass.FINANCIAL,
                            ocr_char=0.97, regex_strength=0.95, consistency=0.99),
            FieldConfidence(name="commencement_date", field_class=FieldClass.TEMPORAL,
                            ocr_char=0.96, model_prob=0.94, consistency=0.98),
            FieldConfidence(name="tenant_legal_name", field_class=FieldClass.IDENTITY,
                            ocr_char=0.88, model_prob=0.91),
            FieldConfidence(name="parking_count", field_class=FieldClass.DESCRIPTIVE,
                            regex_strength=0.60),
        ],
    )
    decision, reason = gate(doc)
    print(f"{decision.value}: {reason} (c={doc.aggregate:.3f})")

The gate() function is the single place the thresholds live, and its ordering is the load-bearing detail: hard failures and high-risk floor breaches are evaluated before the aggregate comparison, so no amount of clean descriptive data can lift a document past a garbled rent. Swapping min() for the weighted mean between the field level and the document level is exactly the design decision from the section above, made concrete — within a field the weakest signal wins, across fields the risk weights decide, and the floors act as a veto.

Validation and quality gates

The scorer itself needs guardrails, because a confidence gate that is wrong is more dangerous than no gate at all — it launders bad data with a number that looks like assurance. Four checks keep it honest.

• Calibration against a labeled holdout. Thresholds are not chosen by intuition; they are chosen by measuring the accept decision against a hand-labeled holdout set. For a candidate accept threshold, treat “gate said auto-commit” as the positive prediction and “the field values were actually all correct” as ground truth, then compute the precision (of everything auto-committed, what fraction was fully correct) and recall (of everything that was actually correct, what fraction got auto-committed) of that decision. Push accept up until precision on money-bearing fields clears your tolerance — a common target is 0.99+ precision on the auto-commit set — accepting that recall drops and more documents go to review. The full sweep, including per-asset-class curves, is the subject of the calibrating confidence thresholds for lease field extraction guide.
• Per-field overrides. Global thresholds are a starting point, not the end state. A jurisdiction whose OCR runs consistently noisy, or an asset class whose leases use unusual rent structures, may warrant a tighter financial floor. Keep the overrides in config keyed on field class and asset class, never hard-coded in gate(), so they can be re-derived from the holdout without a code change.
• No silent auto-commit of a weighted-away high-risk field. This is the check the per-field floors exist to enforce, verified explicitly: assert that no document reaches AUTO_COMMIT while any financial or temporal field sits below its floor. A unit test that constructs a document with pristine descriptive fields and one 0.60 base rent, then asserts the gate returns MANUAL_REVIEW, is the single most important test in this module.
• Hard-rule short-circuit. Validation failures — negative rent, inverted term, missing required field — must route to dead-letter before the aggregate is even computed, because a document that violates a structural invariant is not “low confidence”, it is broken, and averaging it against good fields is meaningless.

Documents that the gate sends to dead-letter carry their full signal breakdown, the failing field names, and the computed aggregate, so an operator triaging the queue sees why the gate refused rather than a bare rejection. Recoverable review items flow into the human-in-the-loop queue via fallback routing logic; the backoff and dead-letter mechanics for genuinely non-recoverable failures live in error handling & retry logic.

Troubleshooting

Performance and scale notes

The scorer is cheap by construction — it is arithmetic over a handful of fields with no I/O — so it adds negligible latency and can run inline on every document without a dedicated worker pool. The cost that matters is not runtime but calibration freshness: thresholds that were correct for last quarter’s document mix silently decay as new landlords, new templates, and new scanning hardware shift the score distribution. Treat the holdout evaluation as a scheduled job, not a one-time setup, and version the threshold config alongside the model versions written into each field’s provenance so an audit can reconstruct which accept value governed a given commit.

At portfolio scale the scoring layer’s real leverage is on the review budget. Every percentage point you move the accept threshold trades machine precision for human hours, and the two are quantifiable from the holdout: precision on the auto-commit set versus the volume routed to review. Tune that trade-off per asset class rather than globally, because a rent-controlled residential portfolio and a triple-net industrial portfolio have very different costs of a mis-billing and very different base extraction accuracy. When document volume is high enough that scoring runs inside the concurrent extraction workers, it inherits their idempotency guarantees automatically — the gate is a pure function of the candidate, so re-scoring a replayed document always yields the same routing decision, which is what keeps async batch processing reruns from producing divergent commits.

Frequently asked questions

What confidence threshold should trigger manual review? Start with an accept threshold around 0.92 and a dead-letter floor around 0.70, then calibrate both against a labeled holdout rather than trusting the defaults. The threshold that matters is the one that gives you acceptable precision on the auto-commit set for money-bearing fields — often 0.99 or higher — even at the cost of sending more documents to review. Set the band per asset class, because a rent-controlled portfolio warrants a tighter floor than a standard commercial one.

Should the document score be the minimum field score or a weighted mean? Use both at different levels. Within a single field, take the minimum of the available signals, because a field is only as trustworthy as its weakest evidence. Across fields, take a risk-weighted mean so financial and temporal fields dominate the document score, and pair it with a hard per-field floor on those high-risk fields so a weighted-away low score can still independently force review. A plain mean over all fields is the classic mistake that lets one garbled rent hide behind clean descriptive data.

Why can’t I trust the model’s softmax probability directly as confidence? Because neural extractors are frequently overconfident — they report 0.99 on answers that are wrong far more than one percent of the time. A raw softmax is not a calibrated probability. Fit a calibration map (temperature scaling or isotonic regression) on a holdout set and store the calibrated value as the field’s model signal, so the number the gate reads actually reflects observed accuracy.

How do I stop a single bad high-risk field from auto-committing when everything else is clean? Enforce a per-field floor on financial and temporal fields and check it before the aggregate. In the reference gate(), a base rent below its 0.90 floor forces manual review regardless of how high the weighted document score climbs. The unit test that guarantees this — pristine descriptive fields plus one low base rent must route to review — is the most important test in the scoring module.

Related

• Calibrating Confidence Thresholds for Lease Field Extraction — sweeping the accept threshold against a labeled holdout and reading the precision/recall trade-off per asset class.
• Error Handling & Retry Logic — backoff, failure classification, and dead-letter mechanics for the documents this gate refuses.
• Field Mapping Strategies — turning the fields this stage scores into canonical schema slots at the ingestion boundary.
• Regex & NLP Clause Extraction — the hybrid extractor whose match strength and model probabilities feed the per-field signals scored here.
• Fallback Routing Logic — where below-threshold and floor-breaching documents divert for human review instead of auto-committing.

← Back to Parsing & Extraction Workflows

← Back to Parsing & Extraction Workflows