Delivering an exceptional customer experience requires a strong data catalog. This is behind-the-scenes work, and it is fundamental to building a great product.
Instacart offers one of the more grounded examples of applied machine learning in production. It starts from a deceptively boring place: the product catalog where grocery product data is stored. From there emerges a set of hard truths about running vision language models when millions of dollars and real customer trust are on the line. The lessons apply well beyond groceries.
The Catalog Is Not Metadata, It Is the Product
In online grocery ordering, the catalog is the substrate for every shopping decision. When a customer searches for something, filters for gluten free options, or trusts that a product is a good source of fiber, they are not really interacting with a shelf and cannot see the physical product. Instead, they are interacting with structured catalog attributes presented through a digital interface. If those attributes are missing or wrong, the customer either cannot find what they want or, worse, is served a confidently incorrect answer and may end up ordering the wrong box of cereals or something they don’t really want.
Catalog data is easy to treat as plumbing, the unglamorous layer beneath the interesting product work that focuses on UX and features. That framing is wrong. The catalog is not beneath the product. For Instacart, it is the product. Without the data catalog, there is no product.
Every ranking model, every recommendation, every dietary filter inherits the quality of the underlying data saved in the catalog. If any data is missing or wrong, those garbage attributes do not stay contained and they propagate into every downstream experience a customer touches. What makes this genuinely hard is that a lot of the most important information may be visual and not live in text at all. For example, in a grocery store you can look at a package and immediately see that it is high in fiber, gluten free, and on sale. All this information is presented on the box in different ways, but none of that is written in the structured data.
The catalog may have photos of the product’s package, and that data exists only on the physical packaging, captured in the image. To be useful for searching or filtering, that data needs to be extracted from the image. That is what makes this a multimodal problem rather than a text problem. To fill in the catalog, you have to actually look at the picture.
Two Jobs for the Same Model: Extraction and Matching
Instacart focuses on two use cases. They look different but share a spine.
The first is attribute extraction. You take a product image, run it through a vision language model, and pull out structured attributes like fiber content, allergen status, or claims printed on the label. This is largely an offline batch process, which buys you some latency headroom.
The second is product matching, which happens when a shopper in a store cannot find the exact item a customer ordered and needs a substitute. This one is real time, and it is not an exact match problem. It is fuzzy. For example, if a customer ordered a twenty-ounce shampoo bottle and the shopper can find only a nineteen-ounce bottle of the same thing, getting the slightly smaller size is usually far better than getting nothing at all. The goal is not perfection. It is a good-enough decision made quickly.
There is a validation wrinkle here. Normally you would confirm a product using its barcode, but for a wide range of reasons the barcode is not always usable in the store. So the system leans on shopper photos taken in the aisle instead. And those photos are noisy in every way you can imagine: bad lighting, cluttered backgrounds, odd angles, partial views. Reasoning reliably over that kind of image is a real engineering problem, not a demo.
The system needs to understand the photo supplied by the shopper, extract meaningful data, compare it to what the customer actually ordered, and provide a go/no-go decision to the shopper. This needs to be done in near-real time while the shopper shops in the grocery store, and it needs to be reliable enough to provide a satisfactory shopping experience to the customer.
What unites both jobs is that they are irreducibly multimodal. You cannot solve them from text alone and you cannot solve them from images alone. You have to fuse both to make the call. And the stakes are asymmetric. A single wrong decision at scale, repeated across millions of orders, can cost real money and erode the trust that keeps people on the platform.
Why Vision Language Models Changed the Calculus
For years, the way you would have attacked these problems is with narrow, purpose built models. One classifier for this attribute, another pipeline for that matching task, each trained and maintained separately. A single vision language model can now cover a surprising amount of this surface. One model reasons over both images and text, handles both extraction and matching, and generalizes to the long tail of products that no bespoke classifier was ever trained to see.
That generalization to the long tail is the quiet superpower. Grocery catalogs are enormous and constantly changing. The old approach struggled precisely where the world is messiest, on the rare and the new. A model that can reason rather than merely classify has a fighting chance on items it has never encountered or with the messy reality of photos taken in the store.
(Canvas from the 3 Benefits of AI)
The Four Problems
Beyond the capability, there are four concrete failure modes that show up once you try to ship this. The structure matters.
Problem 1: Hallucination. If the model labels a non-vegetarian product as vegetarian, that is not a rounding error. That is a disaster, both for the customer who trusted the label and for the platform that published it. In a food context, a confident wrong answer can cause real harm.
Problem 2: Cost. Running frontier vision models across a catalog of this size is expensive, and images are far more costly to process than text. The most capable models are also the priciest, so naive use does not scale economically.
Problem 3: Non-determinism. This one is subtle. Model outputs carry randomness, and that randomness creates jarring user experiences. Imagine you block a particular substitution, and then the same replacement quietly reappears because the model made a different call on a different run. Inconsistency, even when each individual answer is defensible, feels broken to users.
Problem 4: Latency. For real time matching, speed is a hard constraint. If a decision requires reasoning over multiple images, it can take several seconds, which is too slow when a shopper is standing in an aisle waiting.
These are not reasons to avoid the technology. They are the actual work. Anyone can get a vision model to produce an impressive answer once. The engineering is in taming hallucination, cost, variance, and latency all at the same time.
The Techniques That Make It Work
A set of practical techniques pushes back on those four problems. None of them are exotic. All of them are the kind of thing you only learn by shipping.
The first is a verification pass, essentially using the model to check its own work. After the model extracts an attribute value, you feed that answer back and ask it to confirm whether the extraction actually satisfies the guideline before you trust it in production. It is the machine equivalent of asking a person, “are you really sure?” and it measurably improves reliability on the attributes that matter most.
The second is fine tuning smaller models on high quality data. Once you have curated a strong dataset, you can distill the behavior of a large model into a much smaller one. You win on two fronts at once. The smaller model is cheaper and faster, and because it has internalized the guidelines during training, you no longer have to stuff those long instructions into every single prompt. The prompt itself becomes simpler because the model has already learned the rules. You get lower latency and lower cost, and the surrounding system gets less brittle.
The third is decomposition. For an ambiguous matching problem, rather than asking one model to weigh size and brand and flavor and everything else in a single tangled judgment, you break the problem apart. Match the size, then match the brand, then match the remaining attributes, and aggregate those separate decisions into a final answer. Simplify the problem as much as possible, then reassemble it. This is classic engineering wisdom applied to a probabilistic system, and it makes failures far easier to diagnose.
The Assumption Hiding Underneath All of It
Every one of those techniques quietly depends on the same thing: you need a good evaluation dataset. Fine tuning needs it. Verification thresholds need it. Knowing whether your decomposition helped or hurt needs it. Without evaluation data, you end up spending your time debugging guidelines and rewriting prompts rather than actually improving the system. Once you have it, everything else unlocks.
This assumption carries more risk than a single sentence lets on. A vision system in grocery operates in an open, drifting world. New products appear constantly. Packaging gets redesigned. Store lighting and shopper photo habits vary. A fixed evaluation set is a snapshot of a moving target. The danger is not that the dataset is wrong on day one. The danger is that it silently stops representing reality while all your downstream confidence, your fine tuned models and your verification thresholds, keeps assuming it still does. When that gap opens, your metrics can look healthy while customer experience quietly degrades.
There is a second, related trap worth naming: self verification assumes the model’s judgment is somewhat independent from its own error. But a model that hallucinates a fact may also cheerfully confirm that hallucination when asked to check it. The critic and the author share a brain. Verification helps, and the results clearly show it helps, but it is not a guarantee, and treating it as one is how quiet failures slip through. The practical implication for anyone building on this pattern is that your evaluation data and your human review loop are not scaffolding you remove once the model is good. They are permanent infrastructure. The moment you stop measuring against fresh reality, you have started flying blind, no matter how sophisticated the model is.
Why This Matters Beyond Groceries
Most businesses do not run a grocery catalog. But the shape of this problem is everywhere now. Any business turning messy real world inputs, whether photos, documents, or scans, into structured decisions is facing the same four problems and reaching for the same four tools. The specifics change, but the discipline does not:
Reason over multiple modalities, refuse to trust a single confident answer, distill capability into something cheap and fast enough to actually deploy, decompose ambiguity into pieces you can measure, and never stop grading yourself against the real world.
The most valuable lesson here is the refusal to treat AI as magic. Vision language models are powerful, imperfect components that have to be engineered into a reliable system by people who understand exactly where they break. That is the mindset worth carrying into any similar work.
Key Takeaways
- The catalog is the substrate, not the plumbing. Data quality at the foundation propagates into every customer facing experience, so treat it as a first class product surface.
- The important information is often only in the image. Claims like gluten free or high in fiber live on packaging, which makes catalog enrichment an inherently multimodal problem.
- Matching is fuzzy, and that is a feature. A close substitute delivered fast beats a perfect match that never arrives. Design for good enough, not for exact.
- The four real problems are hallucination, cost, non-determinism, and latency. Capability is easy to demo. Shipping means solving all four at once.
- Self verification, distillation, and decomposition are the workhorses. Ask the model to check itself, fine tune small models on strong data to cut cost and latency, and break ambiguous decisions into simple, measurable steps.
- Evaluation data is permanent infrastructure. It underpins every other technique, and because the world drifts, a stale evaluation set will let quality decay while your metrics still look fine.
- Reasoning models generalize to the long tail. The biggest advantage of modern vision language models is handling the rare and the new, exactly where narrow classifiers used to fail.