OmniAI

Today, we’re officially launching the Omni OCR Benchmark! Are LLMs a total replacement for traditional OCR models? It's been an increasingly hot topic, especially with models like Gemini 2.0 becoming cost competitive with traditional OCR. And we wanted to put some numbers behind it! It's been a huge team effort to collect and manually annotate the real world document data for this evaluation. We've spent more time reading PDFs than I ever thought possible. And we're making that work open source! Our goal with this benchmark is to provide the most comprehensive, open-source evaluation of OCR / document extraction accuracy across both traditional OCR providers and multimodal LLMs. We’ve compared the top providers on 1,000 documents. The three big metrics we measured: - Accuracy (how well can the model extract structured data) - Cost per 1,000 pages - Latency per page A link to the full report + data explorer is below. Check it out!

Today, we’re officially launching the Omni OCR Benchmark! Read the full full benchmark here! https://lnkd.in/eH-fbtqE This benchmark and evaluation data are fully Open Source. - Github: https://lnkd.in/eW9iapmr - Huggingface: https://lnkd.in/einC9Yn7 Our goal with this benchmark is to provide the most comprehensive, open-source evaluation of OCR / document extraction accuracy across both traditional OCR providers and multimodal LLMs. We’ve tested 10 popular providers on 1,000 documents, measuring JSON accuracy, cost per 1,000 pages, and latency per page. Evaluating document parsing is difficult, especially with documents containing charts, handwriting, tables, etc. We hope this benchmark provides an easy way to compare providers for your use case.

What do you do when someone sends you a PDF with 12,640 rows you've got to extract? Get ready for some serious copy pasting... This happens WAY more often than you'd think. And clearly this came from a database that someone exported as a PDF. So shouldn't there be a better way? Unfortunately that's just how a ton of enterprise software systems work (EHRs, Inventory systems, ERPs, etc.). And if you need to use that data, you're stuck with whatever format they give you. So companies are spending hundreds of hours of engineering time figuring out how to parse that data. PDF table extraction is pretty hard normally. But there's an entirely different set of problems when you're in the 500+ page range. But good news! This whole process takes about 10 minutes to set up on Omni. And zero engineering required. Just specify what fields you're looking for, and we'll go page by page aggregating data into a single clean table. We can't stop the world from saving databases as PDFs, but at least we can turn it back into useful data!

We know LLMs can read a pdf, but are they any good at it? Turns out you need a lot of PDFs to answer that question! Let's talk document extraction. Everyone's building some variant of this (ourselves includes), but how do you know that it actually works? Is the LLM 70% accurate, 90% accurate, better than a human? Turns out it's a pretty tough thing to benchmark. Mostly because you need a LOT of correctly annotated documents, and those aren't easy to find. Especially when it comes to the types of real world document problems that LLMs are supposed to be solving. Traditional OCR benchmarks are looking at text similarity on things like textbooks or receipts. But that falls short in a couple ways. 1. The data sets aren't representative of real world data 2. Those benchmarks are more focused on character recognition than structured extraction. And just having all the letters off of page doesn't get you very far. So lets built a better benchmark! Next week we'll launch our open source VML benchmark. This will include a full validation dataset consisting of: - Document images - Markdown - JSON Schema - Validated JSON response As I mentioned above, our main goal is LLM based document extraction. To benchmark this, we’ll be validating the two most common patterns. ?????????? ? ???????? ? ???????????????????? This is the most common workflow. You run OCR on a document, and pass the resulting text to an LLM along with the JSON schema for extraction. We will be only be evaluating providers on their OCR accuracy, and using GPT-4o structured output for the extraction. ?????????? ? ???????????????????? For multimodal LLM providers, we will run a separate test of direct extraction without the OCR step (i.e. GPT 4o & Anthropic PDF). So far we've added the following providers: - OmniAI (of course!) - Azure Document Intelligence - AWS Textract - Google Document AI - Unstructured - GPT 4o Vision - Claude Sonnet 3.5 - Llama 3.3 Vision - Deepseek R1 Let me know in the comments if you want someone else added to the list:

wow 7 day from my last post and we've gone from 8,000 to 9,000 stars on Zerox! Didn't have time to ship the extra features I promised last week ?? But coming soon! - Structured schema extraction - Edge detection & cropping (improves cell phone pictures of documents) - More model options (including Deepseek and Qwen!) - Dockerized deployment for Zerox

We hit 8,000 stars on our OCR library! It's been awesome seeing the community traction on Zerox. Turns out everyone's got documents! And traditional OCR models just don't cover the real world complexity that people need. There's a lot to come on the roadmap. In the next month we plan to roll out: - Structured schema extraction - Llama 3.2 support (plus more custom model options) - Dockerized deployment for Zerox I'll drop a link to the demo page below if you want to check it out!

We hit 7000 stars on our OCR library! Only 3k to go till the double digits. It's been awesome seeing the community traction on Zerox. Turns out everyone's got documents! There's a lot to come on the roadmap. In the next month we plan to roll out: - Structured schema extraction - Llama 3.2 support (plus more custom model options) - Bounding boxes for elements - Dockerized deployment for Zerox I'll drop a link to the demo page below if you want to check it out!

Omni's open source OCR tool has just crossed 6,000 stars on github! We're thankful for all the community contributions, and super excited to keep shipping new features. So far this month we have added: - Intelligent page chunking for RAG. - Support for auto correcting page orientation (quality goes way up when all the pages are facing the right way). - The ability to pass in fine tuned models. - A lot more image preprocessing options (trim whitespace, improve contrast, etc.) Give us a star on github to follow along with the journey.

?? 1 month into my journey at OmniAI! Crazy growth in a month: ???Zerox, our open-sourced vision-based OCR: 1k → 6k stars ???new features are shipped every 2 days ???migrated to k8s because of large data volumes ???launched our pricing page ??10x inbound leads (and closing more deals!) This is just the beginning! And yes, we are HIRING! We're looking for founding engineers and growth/content designers. DM me if you like building an unicorn together ??

We're hiring a Designer + Content engineer at OmniAI! You may have noticed, we're on Linkedin all the time! It's one of our strongest channels, and we'd really love to level up some of our content / branding. Plus expand to new channels. Types of things you'll be working on: - Landing page + blog + testimonials + success stories - Linkedin content (banners, product videos, etc.) - Ad content - Swag, conference banners - Monthly newsletter + changelog - and a lot more! We're looking for people with an engineering mindset, but slightly better design skills than my microsoft paint drawings (which I legitimately use every day ??). Especially interested if you've done some technical content writing before. I'll drop the full details below. Please leave comment if you're interested, or if you know someone who would fit this role!