Deskewing and Denoising Scanned Lease Pages for OCR

A scanned commercial lease that recognizes at 71% character accuracy and the same page that recognizes at 97% are usually separated by four cheap, deterministic image operations applied before a single character is read. This page resolves one narrow question inside the OCR Preprocessing Workflows stage: given a raster that ingestion has already flagged as image-bearing, which pixel-level corrections actually move recognition accuracy on lease pages, in what order, and where does each one quietly destroy the thin CAM-table rules and 6-8pt clause type that money-bearing extraction depends on. The parent workflow frames the full deskew-to-confidence-gate loop; this page drills into the geometry and filtering mechanics — skew-angle detection, rotation, noise removal, thresholding, and DPI normalization — and how to measure the lift so you can prove a change helped rather than assume it.

Architectural context

Preprocessing is a stateless conditioning function wedged between two neighbors. Upstream, the PDF/DOCX ingestion pipelines decide whether a page even needs this treatment: a born-digital PDF carries a text layer and is routed straight to parsing, so preprocessing only ever sees genuine scans, faxes, and photographed pages. Downstream, the cleaned raster feeds OCR and then two consumers with very different tolerances — free-text clause recognition, and table extraction strategies that need the ruled grid of a rent schedule to survive intact. That second consumer is what makes lease preprocessing distinct from generic document OCR: an aggressive denoise that would be harmless on a paragraph of prose can erase the hairline rules that separate a base-rent column from a CAM column. Every correction here is therefore a tradeoff against table fidelity, the residual-quality signal is handed to the confidence scoring and quality gates layer, and pages that preprocessing cannot rescue are surfaced to the error handling and retry logic that owns fallback and manual review.

Defect, technique, and the caveat that bites

Each visual defect on a lease scan has a well-understood correction, and each correction has a failure mode that shows up specifically on lease geometry — dense ruled tables and fine-print riders — rather than on ordinary prose. Start from the defect, not the technique.

Measuring the lift, not assuming it

Any of these steps can hurt as easily as help, so treat preprocessing as a change you must measure against a labeled holdout of representative lease pages — not a switch you flip and trust. Track three numbers before and after, because a single accuracy figure hides table damage:

If a preprocessing change raises body CER but lowers table-cell recall, it is a regression for lease work no matter what the headline accuracy says. Persist the before/after numbers alongside the page hash so an audit can later reconstruct why a 2019 scan produced a billed escalation.

Recommended preprocessing pipeline

The implementation runs the stages in the fixed order the table implies — grayscale first so geometry analysis is not distracted by colored watermarks, deskew before thresholding so binarization sees axis-aligned strokes, denoise with an edge-preserving filter so table rules survive, then normalize DPI last so the threshold operated on real detail. Skew detection here uses a projection-profile search (rotate through candidate angles, pick the one that maximizes row-sum variance, since a correctly aligned page has the sharpest peaks and troughs between text lines); the Hough approach in the parent workflow is the interchangeable alternative.

import cv2
import numpy as np
from typing import Tuple


def to_grayscale(image: np.ndarray) -> np.ndarray:
    """Collapse to a single channel before any geometry or contrast analysis."""
    if image.ndim == 3:
        return cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    return image


def detect_skew_angle(gray: np.ndarray, limit: float = 6.0, step: float = 0.2) -> float:
    """
    Estimate skew by projection profile: the row-sum variance peaks when
    text baselines are horizontal. Search a clamped range so a vertical
    table rule or a signature flourish cannot pull the estimate off true.
    """
    # Binarize a downscaled copy just for the angle search (cheap, robust)
    small = cv2.resize(gray, None, fx=0.5, fy=0.5, interpolation=cv2.INTER_AREA)
    thresh = cv2.threshold(small, 0, 255,
                           cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1]

    best_angle, best_score = 0.0, -1.0
    for angle in np.arange(-limit, limit + step, step):
        m = cv2.getRotationMatrix2D((thresh.shape[1] / 2, thresh.shape[0] / 2),
                                    angle, 1.0)
        rotated = cv2.warpAffine(thresh, m, (thresh.shape[1], thresh.shape[0]),
                                 flags=cv2.INTER_NEAREST, borderMode=cv2.BORDER_CONSTANT)
        row_sums = np.sum(rotated, axis=1)
        score = float(np.var(row_sums))  # sharpest line separation wins
        if score > best_score:
            best_score, best_angle = score, float(angle)
    return best_angle


def deskew(gray: np.ndarray, angle: float) -> np.ndarray:
    """Rotate by the detected angle with a replicate border (no black wedges)."""
    if abs(angle) < 0.1:
        return gray  # already straight; do not resample needlessly
    h, w = gray.shape[:2]
    m = cv2.getRotationMatrix2D((w / 2, h / 2), angle, 1.0)
    return cv2.warpAffine(gray, m, (w, h),
                          flags=cv2.INTER_CUBIC, borderMode=cv2.BORDER_REPLICATE)


def denoise(gray: np.ndarray, has_fax_speckle: bool) -> np.ndarray:
    """
    Edge-preserving denoise. Bilateral protects the thin rules of CAM tables
    while flattening background grain; a 3x3 median is added ONLY for
    salt-and-pepper fax speckle, never larger or the hairline rules smear.
    """
    cleaned = cv2.bilateralFilter(gray, d=5, sigmaColor=45, sigmaSpace=45)
    if has_fax_speckle:
        cleaned = cv2.medianBlur(cleaned, 3)  # kernel 3 is the ceiling here
    return cleaned


def adaptive_binarize(gray: np.ndarray, block_size: int = 25, c: int = 10) -> np.ndarray:
    """Local Gaussian threshold: survives uneven illumination and shaded riders."""
    block_size = block_size if block_size % 2 else block_size + 1
    return cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
                                 cv2.THRESH_BINARY, max(3, block_size), c)


def normalize_dpi(binary: np.ndarray, source_dpi: int, target_dpi: int = 300) -> np.ndarray:
    """Upsample low-DPI faxed pages so OCR sees strokes at least a few pixels wide."""
    if source_dpi >= target_dpi:
        return binary
    scale = target_dpi / source_dpi
    return cv2.resize(binary, None, fx=scale, fy=scale, interpolation=cv2.INTER_CUBIC)


def preprocess_page(image: np.ndarray, *, born_digital: bool,
                    source_dpi: int, has_fax_speckle: bool = False
                    ) -> Tuple[np.ndarray, float]:
    """
    Full conditioning pipeline. Born-digital pages short-circuit untouched;
    every scan is grayscaled, deskewed by its detected angle, denoised without
    eroding table rules, adaptively thresholded, then DPI-normalized.
    Returns the cleaned raster and the applied skew angle for the quality report.
    """
    if born_digital:
        return image, 0.0  # already perfect; preprocessing only degrades it

    gray = to_grayscale(image)
    angle = detect_skew_angle(gray)
    straight = deskew(gray, angle)
    quiet = denoise(straight, has_fax_speckle)
    binary = adaptive_binarize(quiet)
    normalized = normalize_dpi(binary, source_dpi=source_dpi)
    return normalized, angle

The two guards that matter most for lease work are the small median kernel and the born-digital short-circuit. A kernel above 3 removes speckle beautifully and takes the hairline rules of a triple-net CAM table with it, so downstream table extraction loses column boundaries; capping it at 3 keeps the rules. And running any of this on a page that already had clean vector text is pure loss — it resamples perfect glyphs into slightly worse ones — which is why the pipeline exits before touching a born-digital page.

Edge cases specific to commercial leases

• Over-denoising erases thin CAM-table rules. The classic regression: table-cell recall drops even as body-text CER improves. It shows up when denoise is too aggressive or morphology is added. Keep denoise edge-preserving, cap the median kernel at 3, and validate against table-cell recall, not just character accuracy, before shipping the change.
• Double-page scans. A book-scanned lease captures two pages in one raster with a dark central gutter. Deskew estimates one angle for two differently-skewed halves and gets both wrong. Detect the gutter (a tall low-intensity vertical band), split into two pages, and preprocess each independently.
• Coffee stains and handwriting. A ring stain or a handwritten margin note is high-variance content that no threshold cleanly removes and that inflates edge density, faking a healthy quality score. Treat these as content the pipeline cannot fix rather than noise to filter harder; let the residual quality signal route them out rather than scrubbing until the printed text goes with the stain.
• Extreme skew beyond the search range. A page fed in near 90 degrees (landscape scanned as portrait) sits outside the clamped angle search, so projection-profile detection returns a near-zero correction and leaves it sideways. Detect gross orientation separately — text-line aspect ratio or an orientation-detection pass — and rotate in 90-degree steps before the fine deskew runs.
• Already-clean digital PDF slipped through. If a born-digital page reaches this stage by misclassification, the pipeline should still short-circuit on the born_digital flag; running preprocessing on it is a measurable accuracy loss, not a no-op.

When to escalate

Preprocessing is conditioning, not rescue, and some pages cannot be salvaged by any rotation or filter. Escalate out of the auto-process path when:

• The residual quality signal stays low after conditioning — a stained, faint, or handwriting-dense page that still scores poorly belongs in manual review through the confidence scoring and quality gates, not in an OCR pass that will fabricate rent figures.
• A page cleaned fine but the ruled grid did not survive — if table-cell recall collapsed, the page is not a general-OCR problem; route it to the dedicated table extraction strategies that reconstruct columns from geometry rather than trusting the binarized rules.
• Downstream OCR confidence is low despite a clean-looking raster — that is a recognition-time failure, and the retry-and-fallback taxonomy in error handling and retry logic owns whether to reprocess with different parameters, try a higher-tolerance engine, or hold for a human. The related pattern of text drifting between pages of a multi-page scan is covered in handling OCR drift and layout shifts in scanned lease documents.

Frequently Asked Questions

Should I deskew before or after denoising? Deskew first, then denoise, then threshold. Rotation resamples pixels, and resampling reintroduces a little smoothing that a prior denoise would have to fight; more importantly, thresholding must see axis-aligned strokes or it fragments slanted glyphs and merges table rows. The fixed order is grayscale, deskew, denoise, adaptive threshold, DPI normalize.

When is Otsu thresholding acceptable instead of adaptive? Only when illumination is uniform across the page — a clean, evenly-lit scan with no shadowed gutter or shaded riders. Real lease archives rarely meet that bar, so adaptive Gaussian thresholding, which recomputes the cutoff from each local neighborhood, is the safe default. A global Otsu cutoff will black out a shaded amendment rider on the same page it reads a bright header perfectly.

How aggressively can I denoise a scanned rent schedule? Barely. Use an edge-preserving bilateral filter and, only for genuine salt-and-pepper fax speckle, a 3x3 median. Anything larger smears the hairline rules that separate the base-rent column from the CAM column, and you will see it as a drop in table-cell recall even while body-text accuracy looks fine.

Does upsampling a low-DPI fax actually add accuracy? Upsampling adds no real detail, but it stops the loss of what detail exists. At 150 DPI a 6pt clause stroke can be sub-pixel, and adaptive thresholding erodes it to nothing; interpolating to ~300 DPI before thresholding gives that stroke a few pixels to survive on. The lift is real but modest — it rescues fine print, it does not sharpen a blurry capture.

Should I run this pipeline on every incoming PDF? No. Born-digital PDFs carry a usable text layer and are routed straight to parsing by ingestion; preprocessing only ever sees pages classified as image-bearing. Running deskew and denoise on a clean vector page measurably degrades already-perfect text, so the pipeline short-circuits on a born-digital flag before touching the raster.

Related

• OCR Preprocessing Workflows — the parent workflow that frames the full deskew-to-confidence-gate loop these image operations sit inside.
• Handling OCR Drift and Layout Shifts in Scanned Lease Documents — recovery patterns for recognition-time drift once the raster is clean.
• Using pdfplumber for Commercial Lease Text Extraction at Scale — the text-extraction path for born-digital pages that skip preprocessing entirely.
• Table Extraction Strategies — where a page whose ruled grid survived (or didn’t) goes to reconstruct rent and CAM columns.
• Confidence Scoring & Quality Gates — the gate that routes low-quality pages to review instead of auto-processing them.

← Back to OCR Preprocessing Workflows

← Back to OCR Preprocessing Workflows