Why OCR fails at handwritten math structure

A few months ago I ran into a frustrating problem while studying.

Most of my math, physics, and engineering notes are handwritten. On paper everything looks clean — equations aligned, derivations grouped step-by-step, tables structured clearly.

But the moment I tried digitizing them, everything broke.

I tested a lot of OCR tools, including newer AI-based ones. They could usually extract the characters correctly, but the structure was destroyed.

Typical problems looked like this:

• aligned equations lose alignment

• multi-step derivations collapse into paragraphs

• numbered problems merge together

• tables flatten into plain text

Technically the text was “recognized”, but mathematically it became almost unusable unless I manually rebuilt everything in LaTeX.

So I started building Axiom to solve that.

Instead of focusing purely on transcription accuracy, the main goal became structural preservation — keeping the mathematical layout intact during digitization.

The current pipeline looks roughly like this:

1. OCR from image or PDF

2. Structural prompting focused on:

   • math alignment

   • derivation grouping

   • table detection

   • numbered block preservation

3. Post-processing layer that normalizes LaTeX/Markdown and stabilizes layout

4. Export as compile-ready LaTeX, Markdown, or searchable PDF

One surprising thing while building this:
most of the engineering effort ended up being in post-processing and layout normalization, not the OCR itself.

Handling things like:

• aligned equation environments

• multi-line derivations as atomic blocks

• table column stabilization

• pagination that doesn't break math blocks

turned out to be much harder than simply recognizing characters.

Right now Axiom runs directly in the browser and converts handwritten STEM notes into structured output.

I'm still improving things like diagram detection and complex chemistry layouts, but the core system is working surprisingly well for math-heavy notes.

Would love to hear from others working on OCR, document AI, or math tooling — it's a surprisingly deep problem space.