A model that has read every document your company ever produced still cannot tell you whether the shelf in aisle four is empty. Most of what happens in the world is not written down; it is seen, judged, acted on, and only sometimes summarized afterward by a person deciding what mattered. Giving an agent eyes is not a feature addition. It changes what the system is permitted to know.
Most of the world was never written down
Text is the residue of events, not the events themselves. Someone looked at a damaged pallet and typed damage noted on arrival. Someone stood in a hospital room, read three numbers off a monitor, and entered them into a chart an hour later. Someone walked a construction site, took eleven photographs, and wrote a paragraph that mentioned two of them. In every case the informative artifact existed first in a visual form, and the written record is a lossy, delayed, human-authored compression of it, filtered through whatever the author believed was worth preserving. A language model trained purely on text inherits that compression and nothing underneath it. It is extraordinarily capable at reasoning over what people bothered to write, and structurally blind to everything that was seen, judged unimportant, and never transcribed.
The asymmetry in raw volume is easy to underestimate. A mid-sized logistics operation generates a few thousand text records a day and several hundred thousand camera frames. A hospital produces orders of magnitude more pixels than prose. A retailer's shelf cameras, a utility's drone imagery, an insurer's claim photographs, a bank's scanned account-opening packets, a software team's bug-report screenshots: all of these are primary records that were, until recently, accessible to software only through the narrow keyhole of whatever metadata a human attached to them. Search over that corpus meant searching the filenames. Reasoning over it meant hiring people to look.
What changes when an agent can see is not that it gains a party trick. It gains the ability to verify rather than merely retrieve. A text-only agent asked whether the installation was completed correctly can only report what the completion form says. An agent that can look at the photograph attached to the form can notice that the form says completed and the photograph shows an unconnected conduit. That gap between what was recorded and what is actually the case is where an enormous amount of organizational cost lives, and it is not addressable at all by a system that can only read.
What multimodal actually means now
The word has been recycled through at least three distinct technical eras, which is why it carries so little information on its own. The first era, roughly through the mid-2010s, meant a pipeline: an object detector produced labels from a fixed taxonomy, a classifier produced a scene category, and a template or a small language model stitched the labels into a sentence. These systems were genuinely useful within their taxonomy and completely helpless outside it. A detector trained on eighty common object classes has no concept of a cracked insulator or a mislabeled reagent bottle, and adding one meant collecting thousands of examples and retraining.
The second era introduced a shared embedding space. Contrastive models trained on hundreds of millions of image and caption pairs learned to project pictures and text into the same vector space, so that the vector for a photograph of a dog landed near the vector for the phrase a photograph of a dog. This was a real advance, and it made two things possible that had not been: zero-shot classification against arbitrary labels invented at query time, and retrieval across modalities, searching a photo library with a sentence. What it did not provide was reasoning. A contrastive embedding can tell you that an image is close to a description. It cannot tell you why the third valve from the left is the one that is leaking.
The current era, and the one that matters for agents, is the vision-language model: a single network in which a visual encoder projects images directly into the token space of a language model, and the whole thing is trained jointly on interleaved sequences of images and text. The consequence is that an image stops being a separate subsystem with its own API and becomes just another part of the context window. The same prompting techniques apply. The same tool-calling machinery applies. The same instruction-following, the same structured output constraints, the same context engineering discipline. And, importantly, the same failure modes, including the one that matters most: a model that is fluent about images will produce a fluent description of an image it has fundamentally misread.
It is worth naming what multimodal does not mean, because the distinction is load-bearing. A pipeline that captions an image with one model and then feeds the caption to a text model is not a multimodal agent; it is a text agent behind a lossy sensor, and every question it can answer is limited to what the captioner happened to mention. If the caption says a photograph of an electrical panel and the actual question is about a scorch mark in the lower right corner, no amount of reasoning downstream recovers the information. Where the image enters the model determines what the model can be asked.
How a model actually sees an image
The mechanism is worth understanding precisely, because almost every practical failure of a vision-language agent follows directly from it. An image arriving at the model is not analyzed as a picture. It is cut into a grid of small square patches, typically fourteen or sixteen pixels on a side, and each patch is flattened and passed through a learned projection into a vector, the same way a word piece is turned into an embedding. Position information is added so the model knows where each patch sat in the grid. The result is a sequence of vectors that occupies exactly the same representational space as text tokens, which is why attention can range freely across words and pixels in a single pass.
That is the entire trick, and its simplicity is the point. There is no separate visual reasoning module, no symbolic scene graph, no geometry engine. The model reasons about an image using the same attention machinery it uses to reason about a paragraph, which is why it inherits both the strengths, flexible open-ended reasoning over arbitrary content, and the weaknesses, notably that it has no reliable notion of precise measurement, exact counting, or geometric relationship except insofar as those properties survived being encoded into patch vectors.
Resolution is where the abstraction starts to bite. Vision encoders are trained at a fixed input size, historically something like 336 or 448 pixels on the long edge, because the number of patches, and therefore the compute, grows with the square of the dimension. An image larger than that gets downscaled to fit, and downscaling is not a neutral operation: it is a deletion. A twelve-megapixel photograph of an equipment rack, squashed to fit a 448-pixel grid, has lost every serial number, every gauge reading, and every warning label on it. The model will still answer questions about them, fluently, because nothing in its architecture signals that the information was destroyed rather than merely absent.
Modern systems mitigate this with tiling. The image is split into several overlapping crops, each encoded at native resolution, plus one downscaled thumbnail that preserves global layout, and all of those tiles are concatenated into the context. This genuinely works, and it is why current models can read a dense document page that models two years ago could not. It also multiplies the token cost by the number of tiles, which is where the second half of the problem begins.
The resolution problem and the bill that comes with it
Do the arithmetic once and it reorganizes your architecture. A single high-resolution image processed with tiling commonly costs somewhere between one thousand and three thousand tokens, depending on the model and the dimensions. A dense page of prose is around seven hundred. An image is therefore not a cheap attachment to a prompt; it is frequently the most expensive object in the context window, and a conversation that accumulates a dozen photographs has spent more of its budget on pixels than on everything else combined. Teams that price a text agent per turn and then add vision are routinely surprised by a bill that grew by a factor rather than a percentage.
The instinct to send the highest quality image available is therefore exactly wrong, and so is the opposite instinct to always downscale. What works is deciding, before the image is sent, what question is being asked of it. If the question is does this photograph show a person wearing a hard hat, a small image is sufficient and a large one is waste. If the question is what is the serial number on the nameplate, the full image is useless and a tight crop around the nameplate is both cheaper and dramatically more accurate. The most effective single optimization in most production vision pipelines is not a better model; it is a cheap detector or a simple heuristic that finds the region of interest and sends only that.
Latency follows the same shape. Encoding images adds meaningfully to time-to-first-token, and unlike text, it cannot be streamed away, because the model cannot begin responding until the visual context is fully encoded. For interactive products this shows up as a pause before anything happens, which users read as the system being stuck. Prompt caching helps enormously when the same image is referenced across multiple turns, which is common in inspection and review workflows, and is worth wiring up early rather than as an optimization pass later.
There is also a quieter cost that only appears at scale: storage and transfer of the images themselves, retention obligations attached to them, and the egress charges for moving them to an inference provider. A pipeline that sends every frame of every camera to a frontier model is not primarily a machine learning system; it is a bandwidth bill with a model attached. Almost every viable deployment puts something cheap in front of the expensive model, and most of the engineering effort goes into deciding what that filter should be.
An image is not a picture to the model. It is a fixed budget of tokens spread across a grid, and anything smaller than one cell of that grid effectively does not exist.
OCR did not die, it changed jobs
For roughly two decades, extracting information from documents meant optical character recognition followed by layout analysis followed by rules or a trained extractor: find the text, find its coordinates, figure out which coordinates correspond to the invoice total. Vision-language models can skip all of it. Hand a model a photograph of an invoice and ask for the total, the vendor, and the line items as structured data, and it will frequently produce them correctly with no OCR stage at all, no layout templates, and no per-vendor configuration. For documents that vary in format, which is most of them, this is a genuine step change in what is buildable by a small team.
The mistake is concluding that dedicated OCR is now obsolete. It is not, and the reasons are specific. Purpose-built recognizers remain more accurate on dense pages of small type, cost a fraction as much per page, run in milliseconds rather than seconds, and, critically, return bounding boxes. That last property is not a nicety. When an extracted figure has to be defended, in an audit, a dispute, or a regulatory review, being able to point at the exact rectangle on the exact page where the number was read is the difference between a system of record and a system of assertions.
The strongest production pattern is therefore not either-or. OCR runs first and produces text plus coordinates. The vision-language model receives both the image and the OCR output, and is asked to reason over them together, which lets it use layout and visual context that flat OCR text loses, while anchoring its reading of individual characters in a deterministic extraction. Where the two disagree, that disagreement is one of the most useful confidence signals available in the entire pipeline, and routing those cases to a human catches a disproportionate share of real errors for a small fraction of the review budget.
The failure mode that makes this worth the trouble deserves stating bluntly. When an OCR engine fails on a character it produces garbage: a substitution that looks wrong, a string of symbols, a low confidence score. When a vision-language model fails on a character it produces something plausible. It will return an account number of the right length and the right shape with two digits invented, formatted perfectly, offered with the same fluent confidence as a correct one. There is no visual tell. For any field where a wrong value is worse than no value, and account numbers, dosages, part numbers, and monetary amounts all qualify, the architecture has to include something other than the model's own assurance.
Video: the modality that breaks every assumption
Video is not a harder image problem; it is a different problem wearing the same clothes. To a current model, a video is a sequence of sampled frames, and the sampling rate is an architectural decision with brutal consequences. Sample at one frame per second, a common default, and a ten-minute clip becomes six hundred images, which at even a modest per-image token cost lands somewhere between six hundred thousand and one and a half million tokens. No context window absorbs that, and no budget survives it. Every practical video system is therefore, first and foremost, a system for deciding which frames not to look at.
What sampling costs you is precisely the events that are shorter than the interval. A person slipping, a spark, a package changing hands, a vehicle running a light: these occupy a fraction of a second, and a one-frame-per-second sampler will miss most of them, silently, while returning a confident summary of the minute in which they happened. Better systems make sampling adaptive: a cheap motion detector or shot-boundary detector runs across every frame at negligible cost, and the expensive model is invoked only on segments where something changed. The architecture that scales is one where the vision-language model is the adjudicator rather than the watchman.
Temporal reasoning is the second and less appreciated weakness. Models that describe individual frames beautifully are markedly worse at questions that depend on time: what happened before what, how many distinct times did this occur, how long did the state persist, did the person return. Frames arrive as a sequence of images with weak temporal encoding, and the model has no persistent notion of object identity across them, which is why it will happily count the same forklift four times as four forklifts. Where identity across frames matters, a dedicated tracker feeding stable object identifiers into the context outperforms asking the model to work it out.
The practical assembly, then, looks like a funnel. Cheap perception runs continuously and produces candidate segments. A tracker maintains identity. The vision-language model receives a small number of well-chosen frames, plus structured metadata about what the cheap layer observed, and answers the question that actually requires judgment: is this a safety violation, is this the same package, does this warrant an alert. Teams that skip the funnel and stream everything to a frontier model build systems that work impressively in a demo with a two-minute clip and collapse on the first day of real footage.
Multimodal retrieval: finding things that were never words
Retrieval-augmented generation, as it is usually built, assumes the corpus is text. Documents are chunked, embedded, and searched, and anything that was not text is either dropped or reduced to a filename. This is a larger loss than most teams measure. In a typical enterprise repository, a substantial fraction of the actual information lives in slide decks whose content is diagrams, scanned contracts with signatures and stamps, engineering drawings, dashboard screenshots pasted into reports, and photographs attached to tickets. A retrieval system blind to all of it will confidently report that the corpus contains no information on a topic that is documented six times over, in pictures.
There are two workable approaches and they trade off cleanly. The first is caption-then-embed: run a vision-language model over every image at ingestion time, generate a rich description, and embed that text alongside everything else. It is cheap at query time, works with existing text infrastructure, and has the underrated property of being debuggable, because you can read what the system thinks each image contains. Its weakness is that the caption is a bottleneck fixed at ingestion, so a query about a detail the captioner did not mention will not retrieve the image that contains it.
The second approach embeds the visual content directly, producing vectors for page images or regions without an intermediate description, so that a query embedding can match visual structure the captioner would never have articulated. Page-level visual retrieval of this kind has become notably strong for documents, sidestepping OCR entirely and handling layout, tables, and figures as first-class content. The cost is opacity, a heavier ingestion pipeline, and infrastructure that does not reuse the text stack. In practice, hybrid systems that run both and re-rank the union outperform either alone, particularly on corpora that mix scanned pages with born-digital text.
Whichever path is taken, the grounding requirement is the same and is frequently forgotten: when the agent answers from an image, it must be able to show the image, and ideally the specific region of it. An answer that says the maintenance log shows a coolant leak on the fourteenth is unverifiable unless the user can see the page it came from. Multimodal citation is harder than text citation because the unit is a rectangle rather than a sentence, but the crop the model was shown is exactly the right artifact to display, and storing it is cheaper than defending an unsourced claim later.
Generating pixels: when the image is the output
Most discussion of multimodal agents concerns input, but agents increasingly produce visual output too: charts, diagrams, mockups, annotated screenshots, marketing imagery. The single most useful distinction here is between rendering and generation, and it is a distinction about facts. Rendering means the agent writes code, a plotting call, a diagram specification, a layout template, and a deterministic tool turns that code into pixels. Generation means a diffusion model produces pixels directly from a description.
The rule that follows is worth adopting without exception: if the image carries facts, render it; if it carries a feeling, generate it. A generated bar chart is not a chart. It is a drawing of a chart, with bars at plausible heights and axis labels that look like numbers, and it will be read by every viewer as data. That is a hallucination with a legend on it, and it is far more dangerous than a hallucinated sentence because visual output carries an unearned presumption of measurement. Every chart, table image, diagram of a real system, and annotated screenshot should come out of a deterministic renderer with the underlying values available for inspection.
Generation earns its place where there is no ground truth to violate. Concept illustration, editorial imagery, design exploration, product mockups, variations on a visual theme, placeholder assets: these are tasks where the model's job is to produce something plausible and evocative, and where iteration through editing, inpainting a region, adjusting a style, extending a canvas, is the actual workflow. Treating image generation as a creative tool rather than an information channel resolves most of the confusion about where it belongs.
The obligations around generated imagery are also more concrete than around generated text. Provenance metadata and content credentials are becoming an expectation rather than a courtesy, particularly for anything published. Likeness and trademark are live legal questions in a way that paraphrased prose usually is not. And an agent that publishes images without a human looking at them first is taking a risk whose downside is disproportionate to the convenience, because a bad sentence is embarrassing and a bad image is a screenshot that circulates.
If the picture carries facts, render it from data. If it carries a feeling, generate it. Confusing the two is how a hallucination ends up with axis labels.
Domain images are not internet images
A vision-language model's competence is a map of its training distribution, and that distribution is the visible internet: product photographs, snapshots, memes, stock imagery, screenshots, charts, and a large but uneven sample of documents. It is extremely good at the visual world that people photograph and post. It is much weaker on the visual worlds that are captured by instruments and looked at by specialists. Radiographs, ultrasound, thermal imagery, electron micrographs, synthetic aperture radar, printed circuit board layouts, seismic sections, histology slides: these appear rarely in web-scale data, are interpreted according to conventions the model has seen described more often than demonstrated, and carry diagnostic signal in exactly the fine detail that patch encoding tends to discard.
_chest_radiograph_(X-ray).jpg)
What makes this dangerous rather than merely limiting is that fluency does not degrade alongside competence. Shown a chest film, a general model will produce a paragraph in correct radiological register, using the right anatomical vocabulary and the right hedging conventions, and it may miss the finding entirely. The output reads exactly like the output would read if it were right. Outside its distribution the model's confidence becomes uninformative, and any workflow that relies on a human noticing that an answer sounds uncertain will not notice anything at all.
Two responses work. The first is to adapt the model: fine-tuning the visual encoder and the projection layer on domain-specific image and text pairs, which is often surprisingly data-efficient because the language half already knows the vocabulary and only the perceptual half needs teaching. The second, and usually the better starting point, is to keep specialist models as tools and use the general model as an orchestrator: a purpose-built classifier or segmentation model produces the measurement, and the vision-language model handles routing, explanation, and structured reporting around it. In regulated fields, the second pattern is frequently the only one available, because the specialist model is the component that carries the clearance and the general model is explicitly not the thing making the call.
A worked example: the inspection agent
Consider a utility that flies drones over its distribution network and produces forty thousand images a day. Today a team of inspectors triages them, spending most of their time confirming that the overwhelming majority show nothing wrong. The task is high-volume, mostly routine, occasionally consequential, and almost entirely visual, which makes it a fair representative of where multimodal agents earn their keep.
A well-built pipeline barely resembles send the photos to a model. Images arrive with metadata: GPS coordinates, altitude, heading, timestamp, and the flight plan that generated them, which is joined against the asset database to establish which pole, span, or panel is actually in frame. A cheap detector trained on the specific asset class runs over every image and proposes candidate regions, discarding the large majority that contain nothing of interest. Each surviving candidate is cropped tightly, which both preserves the detail that downscaling would have destroyed and cuts the token cost by an order of magnitude. Only then does the vision-language model see anything.
What the model is asked for is narrow and structured: given this crop, this asset record, and this list of defect categories, return the category, a severity, a short justification, and a confidence. Constraining the output to an enumerated set matters as much here as anywhere in agent design, because free-form defect descriptions cannot be aggregated, compared across inspections, or fed into a work-order system. The justification field is not decoration either; it is the artifact a human reviewer reads first, and a justification that describes something not visible in the crop is the cheapest available detector of a confabulated classification.
Everything consequential then routes through people. High-severity findings go to an inspector with the crop, the full image, and the asset history attached. Low-confidence findings go to the same queue. A random sample of the confident negatives goes there too, because the failure that will actually hurt this system is not a false alarm, which costs a few minutes, but a missed defect, which is invisible by construction and will never appear in a metric that only counts what the system flagged. The exact crop, prompt, and model version are stored alongside every decision, because in twelve months someone will ask why the system did not flag a component that failed, and the only acceptable answer is a reproducible one.
How multimodal agents fail
The failure catalogue is consistent enough across deployments to be worth memorizing. Confident misreading of small text tops it, for the reasons already described: a number that survived downscaling into six ambiguous pixels comes back as a specific, well-formed, wrong number. Anything read off an image and written into a system of record needs either a deterministic second reader or an explicit confirmation step, and treating model-transcribed identifiers as verified data is the single most common way these systems cause real damage.
The second family is expectation bias, and it is the most conceptually interesting. Models report what usually appears in a scene of that type, which means they are far better at confirming presence than establishing absence. Ask whether the photograph shows a guard rail and the answer is reliable; ask whether the guard rail is missing and the model, primed by ten thousand training images of walkways that had one, will frequently say it is there. Negation and absence are the weakest capability in current vision-language systems by a wide margin, which is unfortunate given that a large share of inspection, compliance, and safety questions are precisely absence questions. Where they matter, the reliable construction is to ask the model to enumerate what it sees and check the list for the expected item, rather than to ask directly whether something is missing.
The third family is quantitative: counting beyond a handful, judging spatial relations, reading analog gauges and dials, and extracting values from charts. Models will produce a number for all of these, and the number will be approximately right often enough to be trusted and wrong often enough to be dangerous. Precise counting in particular should be delegated to a detector that returns instances, with the model used to interpret rather than to enumerate.
The fourth family is prosaic and accounts for more production incidents than the rest combined: an image that arrived rotated because its orientation lived in an EXIF tag nobody honored, a crop that removed the context needed to interpret what remained, aggressive compression that erased the artifact under investigation, a screenshot captured at a scale that rendered the interface text unreadable, and, most quietly, an image that failed to load at all while the agent answered the question anyway from the surrounding text. Validating that the pixels the model received are the pixels you intended, and failing loudly when they are not, is unglamorous work that pays for itself immediately.
Every image is untrusted input
Any text visible in an image is text the model reads, and a model that reads text is a model that can be instructed by it. This is prompt injection with a new delivery mechanism, and the mechanism is unusually convenient for an attacker because images pass through pipelines that were designed to treat them as inert. A screenshot pasted into a ticket, a PDF emailed to a shared inbox, a photograph uploaded by a customer, a web page an agent captures while browsing: each is a channel through which an instruction can reach a model that has tools attached to it.
The instruction does not need to be legible to a person. Pale grey text on a white background, a line of characters in the last few pixels of a margin, a caption embedded in an image at a size a human eye skips over: all of these are read by a model that processes the pixels rather than glancing at the page. There is no visual review process that reliably catches this, because the entire point is that the payload is designed to be invisible to the reviewer and salient to the encoder.
A screenshot is a message from whoever made the screenshot. Treating it as data your system produced is how an attacker gets a turn in your context window.
The defenses are the same structural ones that apply to text injection, applied with more discipline because the channel is easier to overlook. Content extracted from an image is data and never instruction, and the system prompt should say so explicitly and be reinforced by the way the content is delimited in the context. Tool access must be scoped so that no combination of instructions arriving through an image can authorize a consequential action; if reading a customer's uploaded receipt can trigger a refund, the architecture is wrong regardless of how good the prompt is. Sanitization helps at the margins, downscaling to defeat micro-text, running OCR separately and inspecting it for imperative language, flagging images whose extracted text contains instruction-shaped content, but none of it substitutes for capability limits.
Privacy is the other half of the story, and images are uniquely leaky. A photograph submitted to document a damaged product also contains the room it was taken in, the faces of whoever was present, the documents on the desk, the screen in the background, and, in the metadata, the precise coordinates where it was taken and the device that took it. A screenshot submitted to report a bug contains whatever else was on the screen. None of this was intended, all of it is now in your storage and possibly in a third party's inference logs, and the obligations attached to it are the same as if you had collected it deliberately. Stripping metadata on ingest, blurring faces and screens where the use case does not require them, and applying the same retention and access discipline to images as to any other sensitive store are baseline requirements, not enhancements.
Evaluating a system whose output is a description
Evaluation is where multimodal projects most often go quietly wrong, because the artifacts that make text evaluation tractable are missing. There is no reference answer for describe this photograph, similarity metrics between a generated description and a human one reward vocabulary overlap rather than correctness, and a model that misses the one defect in an image will still produce a description that scores well against a caption written by someone who also missed it. The way out is to stop evaluating descriptions and start evaluating decisions.
That means building a labeled set where each image is paired not with a caption but with the outcome the system exists to produce: this pallet is damaged, this form is missing a signature, this invoice total is this number, this frame contains a safety violation. A few hundred such examples, drawn from the actual capture conditions rather than from a clean sample, is enough to make real decisions about prompts, crops, resolutions, and models, and is worth more than any published benchmark score. The metric that deserves the most attention is the false negative rate on absence and defect cases, because that is both the weakest capability and the failure that does not generate a complaint.
Perturbation testing is the second pillar and has no real equivalent in text evaluation. Take the labeled set and degrade it in the ways reality degrades images: rotate, compress, dim, blur, add glare, resize down, add the color cast of a specific camera. Measure how far accuracy falls. A system that is ninety-four percent accurate on clean images and sixty percent accurate on the compressed, badly-lit images your field technicians actually produce is a sixty percent system, and knowing that before launch is the difference between a phased rollout and an emergency.
Slice everything. Accuracy differs by camera model, by site, by time of day, by whether the photographer held the phone level. It also differs across the people in the images, and vision systems have a long, well-documented history of performing worse on subjects underrepresented in training data, which means an aggregate number can conceal a system that works well for most users and poorly for a specific group. Aggregate metrics on vision systems are not merely uninformative; they are actively misleading, and the slices are where the deployable truth lives.
Choosing an architecture
The recurring decision is whether to route everything through one general model or to assemble a pipeline of specialists with a general model on top. The honest answer depends on a few properties of the task, and teams get into trouble mainly by choosing on aesthetics rather than on these.
A single general vision-language model is the right default when the visual inputs are varied, the questions are open-ended, the volume is modest, and there is no labeled dataset to train anything on. This describes most internal tools, most document workflows, and every prototype. It is also the fastest path from nothing to something working, and the flexibility is real: adding a new question requires editing a prompt rather than collecting data.
Specialist models earn their place when the task is fixed and the volume is large, when latency budgets are measured in milliseconds, when the output needs to be a bounding box or a calibrated probability rather than a sentence, or when inference has to happen on a device at the edge of the network. A camera pipeline processing thirty frames a second cannot call a frontier model per frame, and would not want to: a small detector does that job better, cheaper, and more predictably. The general model then sits behind it, invoked on the small subset of frames where judgment rather than detection is required.
Where the images live shapes the choice as much as the task does. Cameras generate far more data than most networks can move, which pushes the first stage of perception onto the device, and privacy constraints in healthcare, retail, and workplace monitoring frequently make on-device processing the only lawful design. The general pattern that survives contact with production is a funnel: cheap and local at the wide end, expensive and central at the narrow end, with the expensive layer touching a small and deliberately chosen fraction of the pixels.
The case against reaching for vision
A great deal of visual AI work exists because someone could not get access to a database. The screenshot being parsed was rendered from an API that would have returned the same values as JSON. The scanned form was printed from a system that still holds the structured original. The dashboard being photographed has an export button. Vision applied to a problem that has an upstream structured source is not clever engineering; it is an expensive, fragile reconstruction of information that was deliberately destroyed by printing it, and it will break the first time someone changes a layout. The first question in any multimodal project should be whether the pixels are the actual source of truth or merely the most accessible copy.
Where the information genuinely originates as an image, there is still often a deterministic tool that beats a model. Barcodes, QR codes, and datamatrix labels exist precisely so that machines do not have to read printed characters, and a scanner library will read them correctly at a rate no vision model approaches, for no cost, in a millisecond. Fiducial markers, structured forms with anchor points, and machine-readable zones on identity documents all belong to the same family. Reaching for a language model to read a barcode is a recognizable symptom of a team that has one tool.
Vision earns its complexity when the visual channel is the primary record, when the content varies faster than templates can be maintained, and when the question being asked requires judgment rather than extraction. Is this damage consistent with the claimed cause. Does this installation match the specification. Is this the same product as the one in the catalogue photograph. Those are questions with no structured upstream answer, no deterministic parser, and no template, and they are where multimodal systems do something that was previously not possible at all rather than something that was previously possible with a script.
Where this is heading
The clearest trajectory is toward video as a native modality rather than a stack of images. Models with genuine temporal encoding, longer visual context, and persistent object identity across frames would remove most of the scaffolding that current video pipelines are made of, in the same way that native document understanding removed most of the layout-analysis stacks that preceded it. Streaming perception, where a model consumes a live feed and maintains state rather than answering questions about clips, is the version of this that changes what products are possible.
The second direction is unification. Models that accept and emit any combination of text, images, and audio in a single pass are already appearing, and the interesting consequence is not multimodal output for its own sake but images as intermediate reasoning steps: a model that can sketch a diagram to work out a spatial problem, annotate an image to isolate a region before answering about it, or generate a visualization as part of its own analysis. Reasoning that passes through pixels rather than only through words is a genuinely different capability, and early results suggest it helps most on exactly the spatial and geometric tasks where current models are weakest.
What will not change is worth stating too, because it constrains anything built on top. Acuity will remain a budget: more detail will always cost more compute, and the decision about what to look at closely will remain an engineering decision rather than something the model makes for free. Pixels will remain untrusted input. Absence will remain harder than presence. And provenance, being able to say which pixels produced which claim, will become more important rather than less as these systems move into contexts where their outputs are contested.
Getting started without drowning in pixels
For a team adding vision to an existing agent, the productive path is narrower than the survey above suggests. Pick one decision rather than one capability: not let the agent understand photos but determine whether a submitted meter reading photograph matches the reported value. Collect two hundred real examples with real labels before writing a single prompt, because those examples will settle in an afternoon the questions that would otherwise generate weeks of architectural debate about models and resolutions.
Then build the unglamorous parts first. Crop before you send, because it is simultaneously the largest accuracy improvement and the largest cost reduction available. Constrain the output to a structure with enumerated categories and a confidence, so that results can be aggregated and gated. Route low confidence and high severity to people, and sample the confident negatives, because that is the only way absence failures ever surface. Store the exact crop, prompt, and model version with every decision. Validate that the image loaded and was oriented correctly, and fail loudly when it did not. None of this is research; all of it is the difference between a demo and a system.
The deeper shift is worth keeping in view while doing the small work. For as long as software has existed, the boundary of what it could reason about was the boundary of what someone had typed in. Vision moves that boundary out to whatever a camera can see, which is most of the physical world and most of the work that happens in it. That is an enormous expansion of what an agent can be asked, and it arrives with a matching expansion of what an agent can be confidently wrong about. The systems that hold up are the ones that treat seeing as an act of measurement, with all the calibration, verification, and provenance that measurement has always required, rather than as a superpower that arrived in an API.