OmniAI的动态

We know LLMs can read a pdf, but are they any good at it? Turns out you need a lot of PDFs to answer that question! Let's talk document extraction. Everyone's building some variant of this (ourselves includes), but how do you know that it actually works? Is the LLM 70% accurate, 90% accurate, better than a human? Turns out it's a pretty tough thing to benchmark. Mostly because you need a LOT of correctly annotated documents, and those aren't easy to find. Especially when it comes to the types of real world document problems that LLMs are supposed to be solving. Traditional OCR benchmarks are looking at text similarity on things like textbooks or receipts. But that falls short in a couple ways. 1. The data sets aren't representative of real world data 2. Those benchmarks are more focused on character recognition than structured extraction. And just having all the letters off of page doesn't get you very far. So lets built a better benchmark! Next week we'll launch our open source VML benchmark. This will include a full validation dataset consisting of: - Document images - Markdown - JSON Schema - Validated JSON response As I mentioned above, our main goal is LLM based document extraction. To benchmark this, we’ll be validating the two most common patterns. ?????????? ? ???????? ? ???????????????????? This is the most common workflow. You run OCR on a document, and pass the resulting text to an LLM along with the JSON schema for extraction. We will be only be evaluating providers on their OCR accuracy, and using GPT-4o structured output for the extraction. ?????????? ? ???????????????????? For multimodal LLM providers, we will run a separate test of direct extraction without the OCR step (i.e. GPT 4o & Anthropic PDF). So far we've added the following providers: - OmniAI (of course!) - Azure Document Intelligence - AWS Textract - Google Document AI - Unstructured - GPT 4o Vision - Claude Sonnet 3.5 - Llama 3.3 Vision - Deepseek R1 Let me know in the comments if you want someone else added to the list: