← Back to Projects

leptons1618@anishgiri.dev:~/projects/menu-ocr cat project.toml

Computer Vision 2023 Stable

Menu OCR

A modular OCR pipeline that turns restaurant menu photos into schema-compliant JSON — sections, groups, items, and prices — with column-aware layout detection and global price-item matching.

Read the case study → Leptons1618/menu-ocr ↗
category
Computer Vision
year
2023
status
Stable
stack
7 tools
highlights
5
write-up
2 min read

01Highlights

  1. Detects multi-column layouts by clustering OCR text x-coordinates with DBSCAN and identifies the price column automatically.
  2. Replaces greedy price assignment with Hungarian-algorithm bipartite matching so no two items claim the same price.
  3. Enforces Section→Group→Item→Price hierarchy with a finite-state machine decoded via Viterbi for the globally optimal label sequence.
  4. Processes an image in about 330ms on GPU, a 3.5× speedup over CPU.
  5. Benchmarks five ML classifiers against the rule-based baseline and shows receipt-trained models (CORD-v2) underperform on menus due to domain mismatch.

02Built with

  • 01Python
  • 02PaddleOCR
  • 03OpenCV
  • 04PyTorch
  • 05spaCy
  • 06FastAPI
  • 07React

03Case study

November 8, 2023 2 min read 459 words

Menu OCR: Turning Restaurant Menu Photos into Structured JSON

How a six-stage pipeline handles multi-column layouts, ambiguous price-item pairing, and hierarchy extraction that off-the-shelf OCR cannot.

The problem

A menu photo is one of the least structured inputs you can hand an OCR system: no declared reading order, prices far from their items, hierarchy implied only by font size. Off-the-shelf OCR returns boxes and strings, not structure, and models trained on receipts (CORD-v2) don't transfer to menus.

The solution

Build the structure explicitly instead of hoping a model learns it. Cluster text x-coordinates into columns, assign prices globally with Hungarian matching, enforce the section-group-item hierarchy with an FSM decoded by Viterbi. Rules do the primary work; ML only steps in when rule confidence drops below 0.7.

Outcomes

  • Cut per-image processing to roughly 330ms on GPU, a 3.5x speedup over CPU.
  • Eliminated conflicting price assignments by replacing greedy matching with Hungarian-algorithm bipartite matching over vertical alignment, horizontal distance, and column consistency.
  • Resolved multi-column reading order through DBSCAN clustering of text coordinates, with automatic identification of the price column.

+2 more in the write-up

Inside the write-up

  1. The pipeline
  2. Why the rules beat the models
  3. Performance
Read the full case study →