Reading Taiwanese receipts straight into the database

The problem

A receipt in Taiwan is not one thing.

E-invoices have QR codes and fixed fields. Traditional triplicate invoices are printed grids. Handwritten receipts are a different world entirely — handwriting, stamps, highlighter, creases, camera angle. Every one of those breaks generic OCR.

The failure mode is the nasty kind: it does not know it is wrong. A missing line item or an extra zero in the tax field looks perfectly normal until month-end reconciliation.

What we built

1. Fix the image before talking about models

Pre-processing matters more than the model. Deskew, correct perspective, normalise contrast, crop to the document — so the model sees a clean, upright page. The gap in recognition quality after this step is large.

2. Extract fields with a local vision model

The model outputs structured fields directly: date, vendor, tax ID, line items, quantity, unit price, tax, total.

Running on-premise is not about API cost. Receipts contain a customer's purchase details, and that is not data you hand to a third party casually.

3. Write to the database, not to a spreadsheet

The output does not become an Excel file that someone re-types later. It goes through an automated write step that puts the values into the right fields — the accountant opens the system and the data is already there.

Under the hood, that step is a capability (a skill) we wrote for the client's AI agent: it validates the tax ID and date, then writes into the matching tables. From the user's side it is simply "no more manual entry".

4. Confidence decides who reviews what

Low-confidence fields are flagged. The accountant only looks at those; the rest flow through.

The outcome

The human role changes from data entry to review. That is not only faster — it means errors are actually seen, instead of surfacing at reconciliation.

We do not claim "fully automatic". The reality of receipt recognition in Taiwan is that a small fraction will always need a human. Designing for "the human reviews that fraction" is what makes it operable.

What is not done

• No dedicated model for handwritten amounts; receipts with heavy handwriting remain manual
• No multi-page stitching (one receipt photographed across two images)
• Accuracy numbers have not been measured systematically, so we do not publish them