Menu OCR: Turning Restaurant Menu Photos into Structured JSON

The pipeline

1. Image normalization Menus arrive as phone photos: uneven lighting, skew from handheld shots, shadows down the page fold. OpenCV handles contrast stretching, deskew via the dominant text angle, and denoising before anything touches a model. Skipping this stage costs more downstream than it costs here — OCR confidence on a skewed image degrades enough that later stages inherit garbage coordinates.

2. OCR extraction PaddleOCR produces text candidates with bounding boxes and confidence scores. At this point the output is still just lines: “Butter Chicken”, “380”, “Starters”, “Paneer Tikka”, with no knowledge of what any of it means. The only useful structure so far is geometric — where each box sits on the page.

3. Layout reconstruction This is where columns get discovered. All text x-coordinates go through DBSCAN clustering, which separates a two- or three-column menu without any hardcoded layout template. The price column falls out automatically: it’s the rightmost cluster with a high digit ratio. Menus almost always right-align prices, so this heuristic holds even when descriptions wrap and push their boxes leftward.

4. Semantic correction Each line gets classified as Section, Group, Item, or Price. Rule-based classification leads: font-scale clustering catches sections (they’re bigger), spaCy-based lexical priors catch obvious item names, position does most of the rest. ML classifiers only adjust predictions when rule confidence drops below 0.7 — more on why below.

5. Price-item assignment Greedy matching — “nearest price to each item, first come first served” — breaks on real menus. Two items stacked vertically can both claim the same price while a third goes unassigned. Replacing greedy with Hungarian-algorithm bipartite matching over vertical alignment, horizontal distance, and column consistency guarantees every item owns exactly one price and no price is claimed twice.

6. Hierarchy decoding Per-line classification alone produces sequences that violate the schema — a Group appearing after its Items, two Sections interleaved. So transitions run through a finite-state machine (Section→Group→Item→Price) and Viterbi decodes the globally optimal label sequence instead of taking each line’s best label independently. One wrong local classification can be outvoted by the sequence around it.

Why the rules beat the models

I benchmarked five classifiers against the rule-based baseline. Ensemble price accuracy came out at 47.2% versus 13.6–15.9% for individual receipt-trained models. The reason is domain mismatch: CORD-v2 is tuned for receipts — linear layouts, different label semantics, different price patterns. On menus those models actively hurt.

That result settled the architecture: rules are the primary signal, ML is a fallback for low-confidence lines only.

Performance

End-to-end, an image processes in about 330ms on GPU — a 3.5× speedup over CPU. Most of that budget is OCR; the layout and matching stages are cheap because they operate on dozens of boxes, not pixels.