Table Extraction Strategies

Table extraction is the part of a lease pipeline that has to read a grid, not a paragraph. It sits inside Parsing & Extraction Workflows, one layer more specialized than free-text clause reading: the rent schedule, the CAM and operating-expense breakdown, the step-up escalation grid, and the exhibit rent roll are all two-dimensional structures where meaning lives in the alignment of a cell to its row and its column header. A number that belongs in the “Annual Rent” column but lands in “Rent PSF” is not a small error — it is an order-of-magnitude mis-billing. This page is about turning those grids into typed, canonical rows without losing the geometry that makes them correct.

The specific problem here is narrow and unforgiving. Prose extraction can tolerate a fuzzy boundary; a rent table cannot. You have to decide whether a table is drawn with ruling lines or held together only by whitespace, reconstruct a cell grid that may span multiple header rows and wrapped multi-line cells, coerce each cell from a display string like $1,204.50 into a Decimal, and then map an unpredictable set of column labels onto a fixed canonical schema. Every one of those steps can fail silently and produce a plausible-looking but wrong schedule, so each carries its own validation and its own confidence score. The PDF/DOCX ingestion pipelines hand this stage clean pages; the field mapping strategies layer consumes the typed rows it produces.

Prerequisites and environment setup

The reference implementation targets Python 3.11+ and leans on two complementary parsers plus a typed model layer. pdfplumber gives you word-level geometry and its own table finder; PyMuPDF (imported as fitz) is faster and exposes both text blocks and a find_tables() API on rendered page structure; pydantic v2 enforces the typed contract at the boundary. Keep the dependency set small so the extractor is cheap to vendor into a worker image.

Install with pip install "pdfplumber>=0.11" "PyMuPDF>=1.24" "pydantic>=2.6" python-dateutil. Two ground rules govern everything below. First, money is never a float. A rent cell enters as text, is stripped of currency symbols and thousands separators, and becomes a Decimal before it is stored — the same typed contract the metadata normalization standards require at the canonical boundary. Second, detection is a decision, not a default. Running the wrong strategy on a table produces confidently wrong cells, so the extractor inspects the page for ruling lines before choosing lattice or stream rather than trusting a library default.

Choosing a detection strategy

Table detection is the fork that governs accuracy. A lattice (or ruled) strategy reconstructs cells from the drawn lines — horizontal and vertical rules become the grid, and any glyph inside a rectangle belongs to that cell. A stream (or whitespace) strategy has no lines to lean on; it infers column boundaries from consistent horizontal gaps between word clusters and row boundaries from vertical spacing. Neither is universally better. Ruled rent schedules from institutional landlords parse cleanly with lattice; the borderless step-up grids that lawyers paste into exhibit pages need stream. Picking wrong is the single largest source of garbled tables.

For a mixed portfolio the right answer is hybrid: probe the page for ruling-line density first, use lattice when rules are present and stream otherwise. Scanned pages with no text layer are out of scope for this stage — they must first pass through the OCR preprocessing workflows, which deskew, denoise, and emit a searchable text layer before any table finder runs. A rotated or skewed page will defeat both lattice and stream, so page geometry is checked and corrected upstream, not here.

Library tradeoffs: pdfplumber, PyMuPDF, and camelot

The three parsers you will actually reach for make different trades. pdfplumber exposes the richest geometry — every character with its bounding box, every drawn line, and configurable table_settings that let you flip between lattice ("lines") and stream ("text") strategies on the same page. That control is exactly what a lease pipeline needs, at the cost of speed. PyMuPDF is dramatically faster and its page.find_tables() returns cell grids with row and column indices already assigned, which makes it the better choice for high-volume first-pass detection; its geometry access is slightly coarser. camelot is purpose-built for ruled tables and its lattice mode is excellent on clean bordered grids, but it depends on a rendering backend and struggles with the borderless schedules that are common in lease exhibits.

The production pattern is to use PyMuPDF for fast detection and a confidence probe, then fall back to pdfplumber with explicit stream settings when the fast path yields a low-confidence grid. The deep comparison — memory footprint, cell-accuracy benchmarks on real CAM tables, and when each library wins — is worked through in pdfplumber vs PyMuPDF for CAM table extraction. What matters at this level is that the strategy (lattice vs stream) is a decision you own, and the library is an implementation detail you can swap behind a stable grid interface.

Primary implementation

The extractor below detects a rent or CAM table on a page, reconstructs the cell grid, resolves a possibly multi-row header, and coerces each data row into a typed RentScheduleRow. The pydantic model is the gate: money arrives as a display string and is coerced to Decimal, dates are normalized to ISO, and any row that fails coercion is recorded rather than silently dropped. The extract_tables call is shown with explicit stream settings; in the hybrid path the same reconstruction runs on a PyMuPDF grid.

import re
from datetime import date
from decimal import Decimal, InvalidOperation
from typing import Optional

import pdfplumber
from dateutil import parser as date_parser
from pydantic import BaseModel, Field, field_validator

# Canonical column vocabulary: header text (lowercased) -> canonical field.
COLUMN_ALIASES = {
    "period": "period_label",
    "lease year": "period_label",
    "term": "period_label",
    "months": "period_label",
    "start": "start_date",
    "from": "start_date",
    "commencement": "start_date",
    "end": "end_date",
    "to": "end_date",
    "through": "end_date",
    "monthly rent": "monthly_rent",
    "base rent": "monthly_rent",
    "rent/mo": "monthly_rent",
    "annual rent": "annual_rent",
    "rent/yr": "annual_rent",
    "rent psf": "rent_psf",
    "rate psf": "rent_psf",
    "cam": "cam_charge",
    "operating expense": "cam_charge",
    "opex": "cam_charge",
}

MONEY_RE = re.compile(r"[^\d.\-]")  # strip $, commas, spaces, non-breaking spaces


def _to_decimal(raw: Optional[str]) -> Optional[Decimal]:
    """Coerce a display money string like '$1,204.50' to Decimal, or None."""
    if raw is None:
        return None
    cleaned = MONEY_RE.sub("", raw.replace(" ", "").strip())
    if cleaned in ("", "-", "."):
        return None
    try:
        return Decimal(cleaned)
    except InvalidOperation:
        return None


def _to_iso_date(raw: Optional[str]) -> Optional[date]:
    if raw is None or not raw.strip():
        return None
    try:
        return date_parser.parse(raw.strip(), dayfirst=False).date()
    except (ValueError, OverflowError):
        return None


class RentScheduleRow(BaseModel):
    """One typed row of a rent or CAM schedule mapped to canonical fields."""
    period_label: Optional[str] = None
    start_date: Optional[date] = None
    end_date: Optional[date] = None
    monthly_rent: Optional[Decimal] = Field(default=None, ge=0)
    annual_rent: Optional[Decimal] = Field(default=None, ge=0)
    rent_psf: Optional[Decimal] = Field(default=None, ge=0)
    cam_charge: Optional[Decimal] = Field(default=None, ge=0)

    @field_validator("monthly_rent", "annual_rent", "rent_psf", "cam_charge", mode="before")
    @classmethod
    def coerce_money(cls, v):
        return _to_decimal(v) if isinstance(v, str) else v


def _map_header(header_cells: list[str]) -> dict[int, str]:
    """Map each column index to a canonical field via alias lookup."""
    mapping: dict[int, str] = {}
    for idx, cell in enumerate(header_cells):
        key = (cell or "").strip().lower()
        if key in COLUMN_ALIASES:
            mapping[idx] = COLUMN_ALIASES[key]
    return mapping


def _flatten_multirow_header(rows: list[list[str]]) -> tuple[list[str], int]:
    """Detect a multi-row header and merge it into one label row.

    Returns the flattened header and the index of the first data row. A header
    spans multiple rows when the top rows carry no parseable money or date.
    """
    header_depth = 0
    for r in rows[:3]:  # a lease header rarely exceeds three stacked rows
        has_value = any(_to_decimal(c) is not None or _to_iso_date(c) for c in r)
        if has_value:
            break
        header_depth += 1
    header_depth = max(header_depth, 1)
    width = max(len(r) for r in rows)
    merged = []
    for col in range(width):
        parts = [rows[r][col] for r in range(header_depth)
                 if col < len(rows[r]) and rows[r][col]]
        merged.append(" ".join(p.strip() for p in parts))
    return merged, header_depth


def extract_rent_schedule(pdf_path: str, page_number: int) -> list[RentScheduleRow]:
    """Detect and extract a rent/CAM table into typed canonical rows."""
    rows: list[RentScheduleRow] = []
    with pdfplumber.open(pdf_path) as pdf:
        page = pdf.pages[page_number]
        ruling_lines = len(page.lines) + len(page.rects)
        strategy = "lines" if ruling_lines >= 4 else "text"
        settings = {"vertical_strategy": strategy, "horizontal_strategy": strategy}
        tables = page.extract_tables(settings)
    if not tables:
        return rows

    grid = [[(c or "").strip() for c in row] for row in tables[0] if any(row)]
    if len(grid) < 2:
        return rows

    header, first_data = _flatten_multirow_header(grid)
    col_map = _map_header(header)
    if not col_map:
        return rows  # header could not be mapped — caller dead-letters

    for raw_row in grid[first_data:]:
        record: dict[str, object] = {}
        for idx, field_name in col_map.items():
            if idx < len(raw_row):
                record[field_name] = raw_row[idx]
        record["start_date"] = _to_iso_date(record.get("start_date"))  # type: ignore[arg-type]
        record["end_date"] = _to_iso_date(record.get("end_date"))  # type: ignore[arg-type]
        rows.append(RentScheduleRow(**record))
    return rows

The design keeps three responsibilities separate. Detection (ruling_lines >= 4) chooses lattice or stream from observed geometry. Reconstruction (_flatten_multirow_header) turns a ragged list of rows into a rectangular grid with one header, so a Monthly / Rent header split across two physical rows collapses into a single mappable label. Coercion happens inside RentScheduleRow, so a $1,204.50 cell becomes Decimal("1204.50") at construction and an unparseable cell becomes None rather than an exception. The COLUMN_ALIASES dictionary is the seam into field mapping strategies: it is deliberately small and data-driven so new landlord vocabularies extend it without touching parsing logic.

Validation and quality gates

A table can parse without error and still be wrong, so the extractor is wrapped in gates that reject a garbled grid before it reaches the canonical schedule. Each gate produces a signal that rolls up into a single per-table confidence score.

• Shape sanity. Every reconstructed data row must have the same column count as the mapped header. A row that is short by one column usually means a merged or dropped cell shifted every value left — exactly the failure that moves a figure from annual_rent into rent_psf. Rows that do not match width are quarantined, not coerced.
• Typed coercion coverage. Count how many money and date cells coerced to a value versus how many were expected non-empty. A schedule where 30% of rent cells landed as None is not a clean table; it is a mis-detected one. Coercion coverage is the strongest single confidence signal.
• Monotonic dates. Rent schedules step forward in time. start_date[n] < start_date[n+1] and start_date <= end_date within each row must hold. A violation means either row order was scrambled during reconstruction or a stream detection merged two periods. This is the same forward-ordering invariant the escalation formula mapping engine relies on when it expands a schedule into billable periods.
• Per-table confidence. Combine the signals into one score — for example the product of coercion coverage and a shape-consistency ratio, penalized for each monotonicity violation. A table below the review threshold auto-commits nothing.
• Dead-letter on a garbled grid. When the header cannot be mapped, coverage is low, or dates are non-monotonic, the raw grid, page number, chosen strategy, and diagnostic counts are written to a dead-letter record so an operator can triage without re-running the batch. The classification and replay machinery for that lives in error handling & retry logic.

from dataclasses import dataclass


@dataclass
class TableQuality:
    confidence: float
    coercion_coverage: float
    shape_consistent: bool
    monotonic_dates: bool


def score_table(rows: list[RentScheduleRow]) -> TableQuality:
    if not rows:
        return TableQuality(0.0, 0.0, False, False)

    money_fields = ("monthly_rent", "annual_rent", "rent_psf", "cam_charge")
    expected = filled = 0
    for r in rows:
        for f in money_fields:
            val = getattr(r, f)
            if val is not None:
                filled += 1
                expected += 1
    coverage = filled / expected if expected else 0.0

    starts = [r.start_date for r in rows if r.start_date]
    monotonic = all(a < b for a, b in zip(starts, starts[1:]))
    within_row = all(
        not (r.start_date and r.end_date) or r.start_date <= r.end_date for r in rows
    )
    shape_ok = len({tuple(f for f in money_fields if getattr(r, f) is not None)
                    for r in rows}) <= 2

    confidence = coverage
    if not (monotonic and within_row):
        confidence *= 0.5
    if not shape_ok:
        confidence *= 0.7
    return TableQuality(round(confidence, 3), round(coverage, 3),
                        shape_ok, monotonic and within_row)

The threshold discipline mirrors the rest of the pipeline: a table at or above the auto-commit floor flows to the canonical schedule, a mid-band table pauses for human review, and anything below the review floor is dead-lettered with full context. Because a wrong rent posts real money, the auto-commit floor for financial tables should sit higher than a pure classification step would tolerate. Calibrating those thresholds against a labeled holdout is the subject of the confidence scoring & quality gates guide.

Troubleshooting

Performance and scale notes

Table extraction is cheaper than transformer inference but not free, and the cost is dominated by page rendering and geometry access. Two numbers drive sizing: pages-per-document and the fraction of pages that actually contain a schedule. Do not run the table finder on every page — most lease pages are prose. A fast pre-filter (regex for a rent or CAM header near a run of aligned numbers) narrows the candidate set so the expensive extract_tables call runs on tens of pages, not hundreds. Use PyMuPDF for that first-pass detection because its find_tables() is markedly faster than pdfplumber’s, and reserve pdfplumber’s richer geometry for the pages where the fast path returns low confidence.

Because extraction is CPU-bound, it belongs off the event loop when it runs inside the broader pipeline. In the async batch processing controller, table extraction is dispatched to a process or thread pool via run_in_executor so a 90-page lease never stalls concurrent workers. Cache by content hash: the same exhibit re-processed after a retry should reuse its extracted grid rather than re-render the page. And keep the per-table confidence with the committed rows — a schedule that auto-committed at 0.82 should be traceable later, so a downstream billing discrepancy can be tied back to the exact table, page, and detection strategy that produced it. The tighter multi-column reconstruction techniques for dense schedules are covered in extracting multi-column rent schedules from PDFs.

Frequently asked questions

When should I use lattice detection instead of stream detection? Use lattice when the table is drawn with visible horizontal and vertical rules — most institutional rent schedules and bordered CAM tables. Use stream when the table has no lines and columns are held together only by whitespace, which is common in step-up escalation grids and exhibit rent rolls. For a mixed portfolio, probe each page for ruling-line density first and choose per page, because applying the wrong strategy produces confidently wrong cells rather than an obvious error.

Why coerce money to Decimal instead of float? Because binary floating point cannot represent most decimal money values exactly, and a rent schedule is summed, escalated, and reconciled downstream where those rounding errors accumulate into real billing discrepancies. Every money cell enters as a display string, is stripped of currency symbols and thousands separators, and becomes a Decimal before it is stored. A cell that cannot be coerced becomes None and lowers the table’s confidence rather than silently becoming a wrong number.

How do I handle a rent table whose header spans two rows? Detect the header depth by scanning the top rows for any cell that parses as money or a date; the rows above the first value-bearing row are all header. Merge those stacked label rows column-by-column into a single label before mapping to canonical fields, so a Monthly / Rent label split across two physical rows collapses into one mappable string. Add the merged label to your column-alias dictionary rather than special-casing it in parsing code.

What confidence score should let a table auto-commit? Combine coercion coverage, shape consistency, and date monotonicity into one per-table score, and hold the auto-commit floor higher for financial tables than you would for a classification step — a wrong rent posts real money. A common starting point is to auto-commit only well above the general pipeline floor and to send any table with low coercion coverage or non-monotonic dates to review. Calibrate the exact threshold against a labeled holdout rather than picking a constant.

Related

• pdfplumber vs PyMuPDF for CAM table extraction — a head-to-head on speed, geometry access, and cell accuracy for operating-expense tables.
• Extracting multi-column rent schedules from PDFs — reconstructing dense, wide step-up grids without column bleed.
• PDF/DOCX Ingestion Pipelines — the upstream stage that normalizes mixed-format documents and preserves table geometry.
• OCR Preprocessing Workflows — deskew, denoise, and text-layer recovery that scanned image tables require before any finder runs.
• Field Mapping Strategies — translating extracted column labels into canonical schema fields at the ingestion boundary.

← Back to Parsing & Extraction Workflows

← Back to Parsing & Extraction Workflows