I work in a health insurance adjacent field. I can see my work going the way of the dodo as soon as VLLs take off in interpreting historical health records with physicians’ handwriting.

That’s assuming that confidence intervals are even independently comparable. Anecdotally major OCR services with specific languages have average confidence intervals that are wildly divergent from similar services with different languages for the same relative quality of result. Acting as if confidence interval is in any way absolute or otherwise able to reliably and consistently indicate the relative quality of results is a mischaracterization at best. In the worst case CI is as good as an RNG. The value of the CI is in the ability to tune usage of the results based on observations of the users and characteristics of the request, sometimes it is meaningful but not always. In this case “good” code essentially hardcodes handling for all the idiosyncrasies of the common usage and the OCR service.

This is the main focus of VLM Run and typed extraction more generally. If you provide proper type constraints (e.g. with Pydantic) you can dramatically reduce the surface area for hallucination. Then there's actually fine-tuning on your dataset (we're working on this) to push accuracy beyond what you get from an unspecialized frontier model.

An effective way that usually increases accuracy is to use an ensemble of capable models that are trained independently (e.g., gemini, gpt-4o, qwen). If >x% of them have the same output, accept it, otherwise reject and manually review

There’s a very low chance that three separate models will come up with the same result. There are always going to be errors, small or large. Even if you find a way around that, running the process three times on every page is going to be prohibitively expensive, especially if you want to finetune.

And by using two different systems (say Gem plus ChatGPT) you essentially reduce chances of hallucination to zero, no? You would need to be VERY unlucky to find to LLMs hallucinating the exact same response.

I think it would be pretty reliable in controlled circumstances. If I take a picture of a book with my cell phone- google Gemini pro is much better at recognizing the text than Samsung's built in OCR.

I would think the same, the cause for hesitation is that we only think this, but cannot know it without thorough testing. Right now the scope of problems where things behave reliably and as expected and scope of problems where things get whacky are unknown. The borders are known to some rather fuzzy extent at best, by people who work with these things as a full-time job. This means we are just blindly gambling on it. For important things, archiving, etc. where truth matters, I will continue using traditional OCR until we can define the reliable use-case scope of LLM based OCR better. I am extremely enthusiastic about LLM's and the things these offer, but i am also a realist. LLM's are an infant technology, and no-where near the level of maturity that companies like openAI claim.

How about calculating confidence in terms of which output regions are stable across the same input on multiple tries. Expensive, but the hallucinations should have more variable output and be fuzzier than higher confidence regions in averages.

More broadly, it’s not trained to have any self awareness and this is a factor in other “hallucinations”. If you ask, for example, to describe the “marathon crater”, it doesn’t recognize that there’s no such thing in its corpus, but will instead start by writing an answer (“sure! The marathon crater is..”) and freestyle from there. Same if you ask it why it did something, or details about itself, etc. You should access one directly (not through an app like chatGPT) and build a careful suite of tests to learn more. Really fascinating.

Logits are not probabilities... at least not in the way you understand probability. Probabilities mathematically are anything that broadly behaves like a probability, whereas colloquially probabilities represent the likelihood or the preponderance of a particular phenomenon. Logits are not either of those.

It doesn’t. Asking for confidence doesn’t prompt it to make multiple passes, and there’s no real concept of “passes” when you’re talking about non-reasoning models. The model takes in text and image tokens and spits out the text tokens that logically follow them. You can try asking it to think step by step, or you can use a reasoning model that essentially bakes that behavior into the training data, but I haven’t found that to be very useful for OCR tasks. If the encoded version of your image doesn’t resolve to text in the model’s latent space, it never will, no matter how much the model “reasons” (spits out intermediate text tokens) before giving a final answer.

With most Machine Learning algorithms I used to get shapley values or other 'explainable AI' metrics (for a large cost compared to simple inference, yes), it's very unsettling and frustrating to work without them now on LLMs.

Kind of. Tesseract's confidence is just a raw model probability output. You could easily use the entropy associated with each token coming out of an LLM to do the same thing.

Why not though? Both kinds of models jumble around the data and spit out a probability distribution. Why is the tesseract distribution inherently more explainable (aside from the UI/UX problem of the uncertainty being per-token instead of per-character)?

I was thinking ABBYY FineReader, but those, too. Instead of using VLMs or any sort of generative AI, they're build on good old-fashioned feature extraction and nearest neighbor classifiers such as the k-nearest neighbors algorithm. It's possible to build a working prototype of this technique using basic ML algorithms.

I think OCR tools are good at what they say on the box, recognizing characters on a piece of paper etc. If I understand this right, the advantage of using a vision language model is the added logic that you can say things like: "Clearly this is a string, but does it look like a timestamp or something else?"

VLMs are able to take context into account when filling in fields, following either a global or field specific prompt. This is great for e.g. unlabeled axes, checking a legend for units to be suffixed after a number, etc. Also, you catch lots of really simple errors with type hints (e.g. dates, addresses, country codes etc.).

You can also use it for robustness. Looking at e.g. historical censuses, it's amazing how many ways people found to not follow the written instructions for filling them out. Often the information you want is still there, but woe to you if you look at the columns one by one and assume the information in them to be accurate and neatly within its bounding box.

What I want: take scan/photo of a document (including a full book), pass it to the language model, and then get out a Latex document that matches the original document exactly (minus the copier/camera glitches and angles). I feel like some kind of reinforcement learning model would be possible for this. It should be able to learn to generate Latex that reproduces the exact image, pixel for pixel (learning which pixels are just noise).

A big difficulty there is typeface detection, some of these were never digital fonts. But, even if it could detect them, you likely don't have those fonts on your computer to be able to put it back together as a digital typesetting for any but the most trivial fonts.

maybe it was my prompt, but there seems to be far too much interpretation after the image embedding. In my examples it implicitly started to summarize parts of the text, unfortunately incorrectly. On an invoice with typed lettering it summarized that payments submitted would not post for 2-3 business days, when in reality the text said if you submitted after 2p on a friday, the payment would not post until the following monday. Which is significantly different. I'd be curious if you could ablate those layers in some way, because the one-shot structured text detection recognition was much better than vanilla ocr.

Good to see more work being done here, but I don't understand why this is tied to someone's proprietary API. Swapping model providers and adding some basic logging is not remotely painful enough to justify onboarding yet another vendor. Especially one that's handling something as sensitive as LLM prompts.

Cool to see, may use this locally for OCR in some cases. But I think the "handwriting" example is a little misleading. Thats a font, not a scan of hand written material

The AI OCR build into snipping tool in windows is better than tesseract, albeit more inconvenient than something like powertoys or Capture2Text, which use a quick shortcut.

I've been looking for a solution to translate a dictionary for me. It is a Shipibo-Conibo (indigenous Peruvian language) to Spanish dictionary- I'd like to translate the Spanish to English (and leave the Shipibo intact). Curious for any thoughts here. I have the dictionary as a PDF (already searchable so I don't think it would need to be re-OCR'd...though that's possible too, it's not clearest scan).

Service doesnt inspire confidence. Openai-compatible api doesnt work (expects `content.str` in message to be a string - ???). Getting 500s on non-openai compatible endpoint - seems like timeouts(?). When it did work it missed a lot, and hallucinated a lot too on custom documents/schemas.

I wonder what the speed of this approach vs traditional ocr techniques. Also, curious if this could be used for text detection (find a bounding box containing text within an image).

Was just coming here to say this, there does not yet exist a multimodal vision LLM approach that is capable of identifying bounding boxes of where the text occurs. I suppose you could manually cut the image up and send each part separately to the LLM but that feels like an kludge and it's still in-exact.

Wait what? That's pretty neat. I'm on my phone right now, so I can't really view the notebook very easily. How does this work? Are you using some kind of continual partitioning of the image and refeeding that back into the LLM to sort of pseudo-zoom in/out on the parts that contain non-cut off text until you can resolve that into rough coordinates?

We've seen so many different schemas and ways of prompting the VLMs. We're just standardizing it here, and making it dead-simple to try it out across model providers.

Basically there is no model schema combination. IF you go ahead and prompt a open source model with the schema it doesn't produce the results in the expected format. The main contribution is how to make these model conform to your specific needs and in a structured format.

Wait, but we're doing that already, and it works well (Qwen 2.5 VL)? If need be, you can always resort to structured generation to enforce schema conformity?

Existing solutions like Tesseract already can embed text into the image, but I'm wondering if there's a way to combine LLM with Tesseract, so that LLMs can help correcting results and finding unidentified text, and finally still embed text back to the image

Maybe I’m being greedy but is it possible to have a vLLM detect when a portion is an image? I want to convert some handwritten notes into markdown but some portion are diagrams. I want the vLLM to extract the diagrams to embed into the markdown output

VLM's can't replace ocr one to one.. most hosted multimodal models seem to have a classical OCR (tesseract-based) step in their inference loop

A lot worse. But, higher quality OCR will reduce the amount of human post-processing needed, and, in turn will allow us to reduce the number of humans. Since humans are relatively expensive in energy use, this can be expected to save a lot of energy.

The PUE of humans for that 80 watts is terrible, though. Ridiculous multiples of additional energy needed to convert solar power to a form of a energy that they can use, and even the manufacturing lifecycle and transport of humans to the datacenter is energy inefficient.

People really only started talking about the cost of running things when LLMs came out. Most everything before that was too cheap to be a serious consideration.

We convert to a JSON schema, but it would be trivial to convert this to yaml. There are some minor differences in e.g. tokens required to output JSON vs yaml which is why we've opted for our strategy.