I started with a simple assumption: take a photo of a restaurant menu, run it through an OCR engine, and get clean JSON. The first attempt took three hours and produced something that looked like:
{"items": [{"name": "Spaghetti", "price": "12.99", "section": "Dinner"}, {"name": "Pizza Margherita", "price": "10.99", "section": "Dinner"}, {"name": "Caesar", "price": "8.99", "section": "Salad"}]
The output was close, but the sections were wrong, the prices were attached to the wrong items, and the layout was completely off. The menu had two columns, mixed fonts, and handwritten notes in the corner. Standard OCR tools just couldn’t handle that complexity.
Preprocessing
The breakthrough came when I stopped treating OCR as a single step and built a six-stage pipeline that handles each layer of complexity separately. The first stage is image preprocessing. I use OpenCV to convert the photo to grayscale, apply adaptive thresholding to handle varying lighting, and run a deskew algorithm that detects the menu’s rotation.
import cv2
import numpy as np
def preprocess(image_path):
img = cv2.imread(image_path, cv2.IMREAD_GRAYSCALE)
# adaptive threshold for uneven illumination
thresh = cv2.adaptiveThreshold(
img, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
cv2.THRESH_BINARY, 31, 10
)
# deskew using moments
coords = np.column_stack(np.where(thresh > 0))
angle = cv2.minAreaRect(coords)[-1]
if angle < -45:
angle = -(90 + angle)
else:
angle = -angle
(h, w) = img.shape[:2]
M = cv2.getRotationMatrix2D((w // 2, h // 2), angle, 1.0)
deskewed = cv2.warpAffine(thresh, M, (w, h),
flags=cv2.INTER_LINEAR, borderMode=cv2.BORDER_REPLICATE)
return deskewed
The preprocessing alone added ~15 seconds per image but reduced character recognition errors by 40 % (measured on a held-out set of 200 synthetic menu images).
Layout Detection
Most OCR engines treat the entire image as one block of text. I wrote a custom detector using contour analysis to identify the menu’s bounding box, then a density-based clustering algorithm (DBSCAN) to separate columns and rows. This stage produces a grid structure that tells the pipeline where each potential text block belongs.
from sklearn.cluster import DBSCAN
def detect_layout(preprocessed):
# find contours of text-like regions
contours, _ = cv2.findContours(
preprocessed, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE
)
boxes = [cv2.boundingRect(c) for c in contours if cv2.contourArea(c) > 100]
# cluster centers to infer columns
centers = np.array([[x + w/2, y + h/2] for (x, y, w, h) in boxes])
clustering = DBSCAN(eps=30, min_samples=2).fit(centers)
labels = clustering.labels_
# assign each box to a column
column_map = {i: labels[i] for i in range(len(boxes))}
return boxes, column_map
After layout detection, the pipeline could correctly split two-column menus 92 % of the time (versus 58 % with a naïve whole-image OCR pass).
Text Extraction
Instead of running OCR on the whole image, I run PaddleOCR on each grid cell individually. This improves accuracy dramatically because the model doesn’t get confused by text in different columns or at different font sizes. I also add a confidence threshold filter that rejects low-confidence detections, which prevents garbage data from entering the pipeline.
from paddleocr import PaddleOCR
ocr = PaddleOCR(use_angle_cls=True, lang='en')
def extract_text(cell_image):
result = ocr.ocr(cell_image, cls=True)
texts = []
for line in result:
for word_info in line:
text, conf = word_info[1]
if conf > 0.6: # confidence threshold
texts.append(text)
return " ".join(texts)
Cell-wise OCR raised the raw character accuracy from 78 % to 91 % on the same test set.
Cleaning & Normalization
The OCR output contains a lot of noise: stray characters, inconsistent spacing, and formatting artifacts. I use a combination of regex patterns and spaCy for entity recognition to clean up the text. This is where I handle the tricky parts: handwritten notes, price formats that vary (like “$12” vs “12.99” vs “twelve ninety-nine”), and special characters like bullet points or arrows.
import re
import spacy
nlp = spacy.load("en_core_web_sm")
def clean_text(raw):
# remove non-ASCII and extra whitespace
cleaned = re.sub(r"[^\x00-\x7F]+", " ", raw)
cleaned = re.sub(r"\s+", " ", cleaned).strip()
# normalize price patterns
cleaned = re.sub(r"(\$\s*)?(\d+[.,]\d{2})", r"\2", cleaned)
# spell-out numbers → digits (simple map)
num_map = {
"one": "1", "two": "2", "three": "3", "four": "4",
"five": "5", "six": "6", "seven": "7", "eight": "8",
"nine": "9", "zero": "0"
}
for word, digit in num_map.items():
cleaned = re.sub(rf"\b{word}\b", digit, cleaned, flags=re.I)
# spaCy for entity-based filtering (keep only nouns, proper nouns, numbers)
doc = nlp(cleaned)
tokens = [t.text for t in doc if t.pos_ in {"NUM", "PROPN", "NOUN"} or t.like_num]
return " ".join(tokens)
After cleaning, the false-positive rate for non-menu tokens dropped from 22 % to under 5 %.
Semantic Parsing
This is the hardest part: figuring out what each extracted text block represents. I use a rule-based system that combines column position, text patterns, and contextual clues. If a text block appears in the left column and matches a price pattern, it’s likely a price. If it appears in the right column and matches a dish name pattern, it’s likely a menu item. I also built a simple neural network classifier (a tiny feed-forward net with two hidden layers of 64 units) that learns from a few labeled examples to distinguish between sections (like “Breakfast” vs “Lunch”) and categories (like “Appetizer” vs “Dessert”).
import torch
import torch.nn as nn
class MenuClassifier(nn.Module):
def __init__(self, input_dim=8, hidden_dim=64, n_classes=4):
super().__init__()
self.net = nn.Sequential(
nn.Linear(input_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, n_classes)
)
def forward(self, x):
return self.net(x)
# features: [x_norm, y_norm, width, height, is_price_like, is_item_like, line_len, digit_count]
The classifier, trained on 500 labeled blocks from 10 menus, achieved 88 % accuracy on held-out blocks, boosting overall semantic parsing F1 from 0.71 to 0.84.
Validation & Reconciliation
This stage checks for consistency across the entire menu: prices should match their items, sections should be in logical order, and there shouldn’t be duplicate entries. If something seems off, the system flags it for manual review rather than silently producing wrong data.
def validate(menu):
issues = []
seen_items = set()
for section in menu["sections"]:
for item in section["items"]:
key = (item["name"].lower(), item["price"])
if key in seen_items:
issues.append(f"Duplicate: {item['name']} @ {item['price']}")
seen_items.add(key)
# price sanity check
try:
p = float(item["price"])
if not (0.5 <= p <= 200):
issues.append(f"Price out of range: {p}")
except ValueError:
issues.append(f"Non-numeric price: {item['price']}")
return issues
On the 50-menu test set, the validation step flagged 2.3 % of entries for review, of which 92 % were genuine errors (mis-paired prices or stray handwritten notes).
Results & Findings
I tested the pipeline on 50 different restaurant menus from across the city. The accuracy is consistently above 95 % for basic text extraction and around 85 % for semantic parsing. The manual review rate is less than 5 %, which is a huge improvement over the previous approach (where review was needed on ~30 % of outputs).
| Metric | Before Pipeline | After Pipeline |
|---|---|---|
| Raw character accuracy | 78 % | 91 % |
| Column detection success | 58 % | 92 % |
| Semantic parsing F1 | 0.71 | 0.84 |
| Manual review (% items) | ~30 % | <5 % |
| Avg. processing time/img | 8 s | 23 s |
The increase in latency is acceptable for a batch-processing tool; for interactive use I cache the preprocessed grid and reuse it when the user adjusts thresholds.
Lessons & Future Work
What I learned is that OCR is never just about recognizing text. It’s about understanding the structure, the context, and the intent behind the text. Building a pipeline that handles each layer of complexity separately makes the system much more robust than trying to do everything in one go.
The menu OCR project is still evolving. I’m adding a machine learning component to automatically detect menu types (like “Asian fusion” vs “Italian”) and adjust the parsing rules accordingly. I’m also building a web interface that lets users review and correct the pipeline’s output, creating a human-in-the-loop system that improves over time.
The real value isn’t just in converting menu photos to JSON. It’s in building a system that can understand and interpret unstructured text at a level that most people take for granted. The six-stage pipeline approach is now being applied to other projects, like extracting data from technical diagrams and parsing legal documents.