Multimodal Document AI Pipeline

Intelligent document understanding system using LayoutLMv3 and Donut for extracting structured fields from invoices, receipts, and forms. Goes beyond plain OCR with spatial-aware, multimodal document comprehension.

Overview

Traditional OCR extracts raw text from documents but loses critical spatial relationships between fields. This pipeline leverages multimodal transformers that jointly reason over text content, visual features, and 2D positional information to accurately extract structured data from complex document layouts.

The system supports two complementary extraction backends:

• LayoutLMv3 -- Token classification approach that combines OCR text with bounding box coordinates and document image patches for spatial-aware entity extraction.
• Donut -- End-to-end document understanding transformer that directly generates structured JSON from document images without requiring an external OCR engine.

An ensemble mode fuses predictions from both models, selecting the highest-confidence extraction for each field.

Supported Document Types

Document Type	Extracted Fields
Invoices	Vendor, Invoice Number, Date, Line Items, Tax, Total, Amount
Receipts	Merchant, Date, Items, Subtotal, Tax, Total
Forms	Key-value pairs, Table structures, Signatures

Architecture

Document (PDF/PNG/JPG/TIFF)
        |
        v
  OCR Pipeline (Tesseract)
   |              |
   v              v
LayoutLMv3    Donut Model
Extractor     Extractor
   |              |
   v              v
  Ensemble Merge (confidence-weighted)
        |
        v
  Structured JSON Output

Installation

# Clone the repository
git clone https://github.com/ln-bodicherla/document-ai-extraction.git
cd document-ai-extraction

# Create virtual environment
python -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Install Tesseract OCR engine
# Ubuntu/Debian:
sudo apt-get install tesseract-ocr
# macOS:
brew install tesseract

Quick Start

Extract fields from a document

from src.document_processor import DocumentProcessor

processor = DocumentProcessor(config_path="configs/config.yaml")
results = processor.process_document("path/to/invoice.pdf")

for field in results["entities"]:
    print(f"{field['label']}: {field['value']} (confidence: {field['confidence']:.2f})")

REST API

# Start the API server
uvicorn src.api:app --host 0.0.0.0 --port 8000

# Extract from a document
curl -X POST http://localhost:8000/extract \
  -F "[email protected]" \
  -F "extraction_mode=ensemble"

Batch processing

results = processor.process_batch(["doc1.pdf", "doc2.png", "doc3.jpg"])

Configuration

All model and pipeline parameters are configured via configs/config.yaml:

• LayoutLM settings -- model checkpoint, sequence length, label schema
• Donut settings -- model checkpoint, generation length, task prompts
• OCR settings -- Tesseract page segmentation mode, confidence threshold, language
• Processing settings -- image size limits, DPI for PDF rendering

Entity Types

The pipeline extracts the following entity types using BIO tagging:

Entity	Description
VENDOR	Company or merchant name
AMOUNT	Monetary amounts (line-level)
DATE	Transaction or invoice dates
INVOICE_NUM	Invoice or receipt identifier
LINE_ITEM	Individual product or service entries
TAX	Tax amounts
TOTAL	Total or grand total amounts

Model Details

LayoutLMv3

Pre-trained on IIT-CDIP (11M scanned documents) with three pre-training objectives: masked language modeling, masked image modeling, and word-patch alignment. Fine-tuned for token classification on document entity extraction.

Donut

Document Understanding Transformer that uses a Swin Transformer visual encoder and BART text decoder. Generates structured output directly from document images in an end-to-end fashion without OCR dependency.

Project Structure

document-ai-extraction/
├── configs/
│   └── config.yaml
├── src/
│   ├── api.py
│   ├── document_processor.py
│   ├── donut_extractor.py
│   ├── layoutlm_extractor.py
│   └── ocr_pipeline.py
├── requirements.txt
└── README.md

Evaluation

No benchmark results or evaluation data are included in this repository. The default configuration loads the public microsoft/layoutlmv3-base checkpoint, which needs fine-tuning on labeled documents before its token-classification output is meaningful (no training code is included), and the public naver-clova-ix/donut-base-finetuned-cord-v2 checkpoint, which was fine-tuned on receipts (CORD).

License

MIT License