Building an accurate OCR Receipt Engine is an interesting engineering challenge because the problem cannot be solved by a deterministic solution. In reality, there are too many uncertainties (i.e., receipt format, language, origin country, picture quality, receipt angle, etc…) in what a receipt scanner API receives. However, like most engineering problems, the path you take ultimately affects the outcome of the solution in the end. This article outlines what I have found to be one of the most effective ways to build a receipt scanner API with optimized features such as being , , , , and . accurate automatic real-time multi-lingual adaptive I will explain this solution based on our experience in building TAGGUN, a Receipt and Invoice Scanning API powered by machine learning. Firstly, why build a Receipt Scanner API? There are many ways that businesses capitalize off digitizing receipt processing. To name a few: Streamlining business processes by minimizing hardcopies Digital marketing Accounting and financial operations IT optimizations Setting the Scene for NLP in Receipt OCR Tech Giants like AWS, Microsoft, Google and IBM are actively competing with each other to offer the best machine learning and computer vision on the market. So, instead of reinventing the wheel and training a model for ourselves, we take advantage of this healthy competition in the capitalist market and select the best computer vision OCR solutions to convert the image of the receipt to raw texts. tesseract OCR So, the true crux of a modern OCR receipt engine is its ability to . This function is more affiliated with (Natural Language Processing). convert syntactical data into semantic information NLP NLP is the field of Machine Learning that allows computers to digest and understand written and spoken texts ( ). ref. 1 Both NLP and OCR are therefore the rudiments fuelling TAGGUN’s engine. I will now paint the picture of what we’ve used these to build the scanner. 5 Essential Layers of the Receipt Scanner 1) Taking Advantage of Multiple OCR Providers Based on our testing, Microsoft Cognitive and Google Vision are 2 of the best OCR providers on the market. And the latest version from Microsoft Cognitive actually outperforms Google Vision. So, we recently switched to Microsoft as the main computer vision provider, after 3 years with Google.  They each have their own benefits and trade-offs, and we set up our engine to optimize the result from both providers. After the file is processed by OCR provider, the output is naturally the classic computer vision OCR result: raw text with coordinates and bounding box. 2) Classifying the Data To improve the data extraction, a contextual awareness should be built around the file and request to predict the meta of the file. E.g., Predicting the: Type or format of the file (i.e., is it a receipt? An invoice? Screenshot? Email?) Language Geolocation (IP address or near param) Range of the amount 3) Named Entity Recognition This stage from the text. detects and extracts the most basic information For example: The decimals and amounts The locations (city, state, and country) The dates Other numbers Specialized Entities Extract 4) Specialised Entities Extraction This phase of the Scan Receipt API is where the more complex information is identified and extracted. A scenario to think about: If there are five distinct amounts, how do you know which is the total amount? Or the tax amount? As you can imagine, it becomes increasingly tricky as the content, format, and language of invoices and receipts become more variable. Several different algorithms are run to determine the best result for each of the entities. For Merchant Verification (VAT ID) and ABN: The official sum method is followed to validate each number and improve accuracy. For more complicated entities, such as Multi Tax Line items: The recognition of patterns in the text is required. This is so grouped information can be accurately extracted (such as tax rate, gross tax amount, net tax amount). Merchant Name Entity: This can be trained (or fed back) for each account. So, accuracy can be improved, especially for each individual account over time. To summarise some ideas of the specialized entities that could be processed in this stage: Total Amount, Tax Amount, ABN, Multi Tax Line Items Merchant Verification, Merchant Name Receipt Number, Invoice Number IBAN Payment Type (i.e., credit card, cash, visa, MC, etc.) Fapiao Invoice Number and Code. 5) Data Enrichment Public and helpful APIs are called as needed, to acquire supplementary information. Examples of these are: VAT information (for European clients) ABN information (for Australian clients) Location information and verification (using Google Places) Normalization of merchant names. 6) JSON Format The result outputted is a JSON format. Because the JSON format is a common data format, developers can simply integrate the receipt OCR library into any software, with any programming language. What’s also recommended, is building the engine to instantly return the result following the API request. has this feature to avoid developers needing to make additional requests or building additional webhook endpoints. TAGGUN

Google

Instantly

Microsoft

Visa

Python Environment Set-Up with M1 Chip

Nominated for 2022 - HackerNoon Contributor of the Year - Natural Language Processing

Too Long; Didn't Read

Make resilience your competitive advantage

Behind the Scenes of an OCR Receipt and Invoice API Engine

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Python Environment Set-Up with M1 Chip

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

100+ Free Pluralsight Courses to learn Python, Java, and Spring Boot

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

10 Ways AI Has Changed Our Lives

Python Environment Set-Up with M1 Chip

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

100+ Free Pluralsight Courses to learn Python, Java, and Spring Boot

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

10 Ways AI Has Changed Our Lives

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps