The interface was never built for a model
Most of the software running the world today has no API at all, and the software that does have one usually exposes only a fraction of what its interface actually lets a human do. A hospital's scheduling system, a supplier's ordering portal, an internal tool built by a contractor who left the company years ago, a government form that only exists as a PDF rendered inside a browser tab, these are not edge cases, they are the overwhelming majority of the software an agent might reasonably be asked to operate on someone's behalf. For years, the practical answer to this gap was to simply not automate those systems, or to pay someone to write a brittle, purpose-built script against one specific site that broke the next time a button moved three pixels to the left.
Computer-use agents are the industry's answer to that gap: instead of asking every piece of software in the world to expose an API a model can call, teach the model to do what a human already does, look at the screen, decide what to click, and click it. This reframes an enormous integration problem, one API at a time, forever, into a single capability problem, can a model reliably operate an arbitrary interface, that once solved reasonably well generalizes to almost everything a human currently does by hand. That is a genuinely different bet than the tool-calling paradigm most agent systems still rely on, and it comes with a genuinely different set of engineering problems attached to it.
It is worth being precise about what changed to make this feel achievable now rather than five years ago. Vision-capable language models reached a level of screen understanding, reading small text, distinguishing a disabled button from an enabled one, recognizing a dropdown versus a text field, good enough to act as the perception layer for a real interface rather than a toy demo. That capability, combined with models that can hold a multi-step plan across many actions without losing track of the goal, is what turned a model that can describe what's on a screen into a model that can get a real task done by operating one, and it is the reason nearly every major lab shipped some version of this capability within the same narrow window of time rather than one of them getting there years ahead of the rest.
From function calls to pixels: how we got here
The earliest production agents avoided the screen entirely. If a task involved a web application, the practical approach was to talk to whatever the application exposed underneath its interface: a REST API if one existed, or the page's own DOM if it didn't, parsed and queried the way a script written by a human engineer would query it. This worked well for a narrow set of applications and broke constantly for everything else, because the DOM of a real, actively maintained web application is not a stable contract, it is an implementation detail that changes with every redesign, every A/B test, and every framework migration, none of which are announced to whatever is depending on the old structure.
The next step, still short of full screen operation, was structured accessibility data: the same tree of roles, labels, and states that a screen reader relies on to describe a page to someone who cannot see it. This is a meaningfully better foundation than scraping raw DOM, because it is a standardized, semantic description that browser vendors and application developers already have some incentive to keep accurate, precisely because real screen reader users depend on it. An agent that can ask what interactive elements exist on this page, and what are their roles and current states, through the accessibility tree, gets a clean, structured answer without ever needing to interpret a pixel.
What accessibility trees cannot do is cover everything, and that gap is what pushed the field toward vision. Canvas-rendered applications, custom-drawn widgets, legacy desktop software with no modern accessibility layer, and plenty of ordinary web pages where developers simply never bothered to label their elements correctly all leave holes in the accessibility tree, sometimes small, sometimes covering the entire interface. A model that can only act through accessibility metadata inherits every one of those gaps as a hard failure. A model that can also fall back to looking at the screen the way a human would, reading pixels, recognizing shapes, inferring what a button does from its icon and position, does not have that ceiling, which is exactly why the current generation of computer-use agents leans on vision as the universal fallback even when a cleaner structured signal is available and preferred when it exists.
This is why the state of the art today is not vision instead of structured data, but a layered approach: reach for the accessibility tree or DOM first when it is available and trustworthy, because it is faster, cheaper, and less ambiguous, and fall back to vision, or combine both, for everything that structured layer cannot see. Treating this as an either-or choice, which a surprising number of early implementations did, throws away real reliability in either direction: pure vision is slower and more error-prone than it needs to be when good structured data already exists, and pure structured data is blind to the large share of real software that never exposes it properly.
A model that can only act through accessibility metadata inherits every gap in that metadata as a hard failure. A model that can also look at the screen does not have that ceiling.
What operating a screen actually requires
It helps to break operating a screen into the separate capabilities it is actually made of, because treating it as one monolithic skill obscures where systems actually succeed or fail. Four capabilities function fairly independently in practice, and a system worth trusting needs a real answer for each one, not just an overall demo that looked convincing.
Perception: turning whatever the interface currently shows, a screenshot, an accessibility tree, or both, into a representation the model can reason over.
Grounding: mapping a decision like click the submit button onto an actual, precise, actionable target, a pixel coordinate, an element handle, an accessibility node, because a model that correctly decides what to do next is useless if it cannot translate that decision into a specific, executable action.
Action: actually executing the click, keystroke, scroll, or drag in a way that the underlying application receives and processes correctly, which sounds trivial and is not, because synthetic input events do not always behave identically to input generated by real hardware.
Verification: confirming, after acting, that the action actually produced the intended effect, rather than assuming it did because no error was thrown, since a silently failed click looks identical to a successfully executed one until you specifically check.
Most of what distinguishes a genuinely reliable computer-use system from an impressive demo comes down to how seriously it takes the second and fourth items on that list. Perception has gotten good enough, fast enough, in current-generation multimodal models that it is rarely the bottleneck anymore. Grounding and verification are where the remaining failures concentrate, and they are also the two capabilities that get the least attention in a quick demo, because a demo only needs to work once, on one interface, under conditions the person recording it controlled.
The threat model, stated plainly
A computer-use agent's threat model is broader than it might click the wrong thing, although that alone deserves to be taken seriously, because unlike a malformed API call, which usually fails loudly with an error response, a misdirected click on a real interface often succeeds, just against the wrong target, and the system has no built-in way to know the difference between success against the intended element and success against a lookalike one sitting a few pixels away.
Prompt injection takes on a distinct and arguably more dangerous shape in a computer-use context than it does in a text-only agent pipeline. A webpage, a PDF rendered in a viewer, or a document open in an editor can contain content specifically crafted to be read by the model doing the screen understanding, rather than by the human user, instructing it to navigate somewhere else, submit a form it was never asked to submit, or copy sensitive information somewhere it should not go. Because the agent's entire perception of the world is the screen, and the screen is rendering content from whatever source the task involves, there is no clean separation between instructions from the user and content the agent happens to be looking at, the same distinction that matters in every other agentic system, made harder here because the injection can be visual as well as textual, hidden in an image, disguised as an ordinary-looking button, or placed in a location a human would never think to read carefully before an agent clicks past it.
Credential and data exposure is a more mundane but equally real concern. An agent operating a real screen is, definitionally, looking at whatever is on that screen, which may include other open tabs, notification banners, autofilled form fields, or a password manager's popover, none of which are relevant to its current task but all of which are technically visible to a system built to read the whole screen rather than a narrow, scoped input. And when the agent itself needs to type into a field, whether that is a search box or a login form, the same operational discipline that applies to any automated system handling credentials applies here too, arguably more urgently, since the action is being decided by a model in real time rather than executed from a pre-written script a human reviewed in advance.
The last piece of the threat model is the one that is easiest to underweight because it does not sound like a security problem at all: plain ambiguity. Real interfaces are full of elements that look similar, are labeled unclearly, or require context a human would bring automatically, which button is Submit versus Submit and notify, which of two nearly identical rows in a table is the one the task actually referred to, whether a modal that just appeared is the expected next step or an unrelated interruption. None of this is adversarial, and all of it produces exactly the same category of outcome as a deliberate attack: an action taken against the wrong target, with real consequences, that nobody intended.
A misdirected click doesn't fail loudly like a malformed API call. It usually succeeds, just against the wrong target.
Two philosophies of grounding: pixels versus the accessibility tree
Even once perception is solved, precisely translating click the blue submit button into an executable action is its own hard problem, called grounding, and there are three broad approaches in active use today. The first, coordinate prediction, asks a vision model to output the literal x and y pixel position it believes corresponds to the target element, directly from a screenshot. This is the most general approach, since it works on absolutely anything rendered on a screen regardless of what technology built it, but it is also the least precise, because a model predicting continuous coordinates from a raster image is prone to small errors, and a small error against a densely packed toolbar or a small checkbox is often exactly the difference between the intended target and its neighbor.
The second approach, generally called set-of-mark prompting, sidesteps continuous coordinate prediction almost entirely. Instead of asking the model to guess a pixel position, the system first detects every interactive element on the screen, overlays a visible numbered label on each one, and asks the model to choose a number rather than a coordinate. This turns an error-prone regression problem, predict the exact pixel, into a much more reliable classification problem, pick the right label from a fixed, visible set, and it tends to produce a meaningful jump in accuracy specifically because models are far better at discrete selection than continuous spatial prediction, especially on cluttered or densely packed interfaces where two plausible targets sit close together.
The third approach skips pixels almost entirely and grounds directly against structured elements: rather than reasoning about the screen as an image, the system exposes a tree of the actual interactive elements present, each carrying a stable reference, and asks the model to act against a specific reference rather than a location. This is the fastest and most reliable option when it is available, since there is no coordinate error to make in the first place, an element reference either exists and can be clicked directly or it does not, but it depends entirely on the underlying application exposing a usable structured representation, which, as covered above, a meaningful share of real software simply does not do well.
None of these three approaches is universally superior, and the strongest systems in production today do not commit to just one. A reasonable default is to prefer structured element references wherever the application exposes them reliably, fall back to set-of-mark labeling when a structured tree exists but is incomplete or partially unreliable, and reserve raw coordinate prediction for the residual cases, canvas-drawn interfaces, custom widgets, legacy software, where nothing else is available. Choosing the grounding strategy per element, rather than per application, tends to outperform any single strategy applied uniformly, because most real interfaces are a patchwork of well-labeled and poorly-labeled elements sitting side by side on the same page.
The verification problem
Executing a click is not the same as confirming it did what it was supposed to do, and the gap between those two things is where a large share of real-world computer-use failures live, quietly, because a failed action that raises no error looks indistinguishable from a successful one to a system that never checks. A click can land on the correct pixel and still fail to register, because the application was mid-transition, because a synthetic input event was handled differently than the application expected, or because an overlay nobody accounted for briefly intercepted it, and in all three cases the agent's own action log shows exactly the same click executed successfully outcome as it would for a click that actually worked.
The practical answer is to treat every action as unverified until the resulting state confirms it, rather than trusting the fact that a click event fired at all. That confirmation can come from re-reading the accessibility tree or DOM to check that the expected state changed, a form field now shows the entered value, a modal that should have closed did, a new row appeared in a table, or from a follow-up screenshot compared against what was expected to change. Either way, this means a well-built computer-use loop budgets for observing the world again after acting, not just before, and treats the action executed and the action worked as two separate claims that each need their own evidence.
A closely related failure is what amounts to a race condition between what the agent observed and what it then acts on: the screenshot or accessibility snapshot the model reasoned over can be stale by the time the resulting action actually reaches the application, because the page changed in the interval, an animation completed, a background request resolved, a notification appeared. This is the same time-of-check-to-time-of-use pattern that shows up constantly in security-sensitive systems generally, applied here to UI state rather than a filesystem path, and the mitigation is structurally similar: re-verify the state immediately before acting whenever the gap between observation and action is long enough for the world to have plausibly changed, rather than trusting a snapshot that might already be out of date.
Verification also has to account for actions that appear to succeed but accomplish the wrong thing entirely, which is a distinct failure mode from an action simply not registering. An agent can correctly click a checkout button and have the purchase go through cleanly, while having selected the wrong shipping option two screens earlier because the interface's default silently reset between steps, something no amount of watching the final click succeed would ever catch. The lesson generalizes: verification needs to check the specific state the task actually cares about, not merely that some action fired without error, and the two are a lot further apart than they first appear on interfaces designed for humans who fill in the gaps automatically.
Latency, retries, and the screenshot loop
The basic architecture underneath almost every computer-use agent is a loop: observe the current state of the screen, decide on one action, execute it, wait for the interface to settle, observe again. That last step, observing again after every single action rather than planning several actions ahead against a single snapshot, is not a stylistic choice, it follows directly from the verification problem above: a model that plans five clicks against one observation is operating on an assumption that nothing in between those five clicks changes, an assumption that fails constantly on real, dynamic interfaces where a single action routinely triggers a loading state, a validation message, or a layout shift that invalidates everything the model assumed about the following four steps.
This tight loop is expensive in a way that plain text-generating agents are not, because every iteration costs a full round trip: capture a screenshot or refresh an accessibility snapshot, send it to a vision-capable model, wait for a decision, execute the resulting action, wait for the interface to finish reacting before the next capture is even meaningful. A task that a human completes in fifteen seconds of rapid, intuitive clicking can take a genuinely long time for an agent working through the same task one carefully verified step at a time, and that latency is not a rough edge to be optimized away later, it is close to the central engineering cost of the entire approach, and it shapes which tasks are actually worth automating this way in the first place.
Some of that cost is reducible without giving up the safety the loop provides. Caching a page's structural layout between actions that do not plausibly change it, batching genuinely independent reads, such as checking several unrelated pieces of on-screen state at once instead of one call at a time, and using a cheaper, faster model for the routine bulk of navigation while reserving a stronger model specifically for ambiguous or consequential decisions all cut real cost without reintroducing the blind multi-step planning that makes the loop unreliable in the first place. What does not hold up well is skipping the re-observation step outright to save latency, because the failures that step exists to catch, stale state, unexpected popups, actions that silently failed, are exactly the failures that erase whatever time was saved, usually while producing a much larger cleanup cost than the latency it avoided.
Timeouts deserve the same explicit, first-class treatment here that they get in any system executing untrusted or unpredictable work: a hard wall-clock limit on how long the agent waits for an interface to settle after an action, and a hard cap on how many iterations of the loop a single task is allowed to run before it is treated as stuck rather than simply slow. Without both, an agent can end up in a state where it waits indefinitely for a page that will never finish loading, or cycles through the same failed click dozens of times because nothing ever tells it to stop and escalate instead.
Guardrails built for a mouse, not a shell
The guardrail thinking that applies to agents generally, classify actions by how reversible and how consequential they are, and gate the risky ones behind confirmation rather than treating every action identically, applies to computer-use agents with one added complication: on a real interface, the confirmation step the system is supposed to respect is often itself just another button on the screen, and an agent capable of clicking anything is, by construction, also capable of clicking through the very dialog meant to stop it.
This means an are you sure you want to delete this modal cannot be treated by the agent as simply the next element to interact with in pursuit of its goal. The system built around the agent needs its own, separate notion of which actions are consequential, tied to what the action actually does rather than to whatever confirmation UI a given application happens to render, because relying on the target application's own confirmation dialog as the safety mechanism means the safety mechanism only exists when that particular application's developers happened to build one, and disappears entirely on the interfaces that do not.
In practice this argues for an explicit action classification maintained outside the model's own judgment: reading a page, navigating between screens, and filling in form fields that have not yet been submitted are typically safe to execute without pausing, since a mistake there is cheap to notice and cheap to undo before anything leaves the sandboxed session. Submitting a form, sending a message, completing a purchase, deleting a record, or granting a permission are a different category entirely, and deserve a genuine pause for explicit confirmation from the person who authorized the task in the first place, independent of whether the interface being operated happens to also show its own dialog at that moment.
The same discipline extends to scope: an agent operating a screen should be constrained, at the session level, to the smallest set of applications, tabs, and accounts the task actually requires, rather than given a general-purpose desktop or browser session with everything the operator happens to have open or logged into. A task that only needs to book a conference room has no legitimate reason to be running in a browser profile that also has an email inbox, a payment method saved to autofill, and an open banking tab, because every one of those is a target the agent does not need and should not be one accidental click away from reaching.
An agent capable of clicking anything is, by construction, also capable of clicking through the dialog meant to stop it.
A worked example: booking a conference room
Consider a fairly ordinary task: find a free thirty-minute slot next Tuesday afternoon, book a specific conference room for it, and invite two colleagues. Nothing about this task involves writing code or touching a filesystem, which makes it a useful example of computer-use specifically, since the only way to accomplish it is by operating whatever calendar and room-booking software the organization actually uses, most of which was never built with any external API a script could call directly.
A well-built agent starts by reading the calendar's current view through whichever grounding method that particular application supports best, ideally its accessibility tree if the calendar is a modern web application that exposes one reliably, falling back to a labeled screenshot if it does not. It identifies the open slots in the relevant afternoon window, cross-references room availability the same way, and settles on a specific room and time before touching anything, keeping the actual interaction with the interface to the minimum number of actions once a decision has been made, rather than clicking around exploratorily while reasoning, which is both slower and more likely to leave the interface in a half-modified state if the task is interrupted partway through.
When it comes time to act, the agent selects the time slot, opens the booking dialog, and fills in the room and meeting details, verifying after each step that the field it just set actually reflects what it intended, the selected room shows the correct name, the time picker reflects the correct half-hour window, rather than assuming the fill succeeded because no error appeared. Adding the two colleagues as invitees is treated as routine, reversible input, since nothing has actually been sent yet at this stage and a wrong entry here costs nothing to correct.
Actually sending the invitation, the point at which two other people receive a calendar notification and the room becomes reserved against everyone else's ability to book it, is exactly the kind of consequential, semi-reversible action that deserves the pause for confirmation described above, independent of the fact that the booking application will also, on its own, show a final send invitations button as part of its normal flow. Once confirmed, the agent verifies the booking actually appears, that the calendar now shows the meeting at the intended time in the intended room, rather than trusting the absence of an error message on the submission screen, and reports back with what was booked, not just that an action was taken.
How these systems actually fail
A handful of failure patterns show up again and again across computer-use deployments, and most of them trace back to one of the four capabilities described earlier not being taken seriously enough. Misclick cascades are the most visible: a single wrong click early in a task lands the agent on an unexpected screen, and rather than recognizing that its mental model of where it is no longer matches reality, the system keeps executing its original plan against a page it is now misreading, compounding one small error into a sequence of increasingly nonsensical actions that only make sense if you assume the first click worked as intended.
Hallucinated elements are a specific, sneaky version of the same problem: a vision-based model describing what it believes is on the screen, a button, a checkbox, a menu item, that is not actually present, whether because it half-remembers similar interfaces from training or because ambiguous visual content nudged it toward a confident but wrong reading. This is precisely why grounding against structured element references, when available, beats pure vision even when vision alone might have gotten the right answer most of the time, because a reference either points at a real element or the action simply fails to resolve, whereas a hallucinated coordinate can silently click on whatever happens to occupy that pixel instead.
Popups and modals are an underrated source of failure precisely because they are so mundane: a cookie consent banner, a rate this app prompt, a session-timeout warning, appearing exactly when the agent expected a different screen entirely. A system that only knows how to execute its planned next action, rather than first checking whether the screen actually matches what it expected before acting, will frequently click through, around, or straight into these interruptions in ways that either silently fail or, worse, silently accept a modal's default option, granting a permission or dismissing a warning nobody meant to dismiss.
Infinite retry loops happen when an action fails in a way that looks, from the agent's limited vantage point, like it might succeed if just attempted slightly differently, and nothing in the surrounding system enforces a hard ceiling on how many attempts a single step gets before the whole task is treated as stuck and escalated to a human rather than retried yet again. Layout shift from ads, dynamically loaded content, or asynchronous data fetches invalidates coordinate-based grounding specifically, since an element's pixel position from a moment ago may no longer correspond to anything by the time the click actually fires, which is one more argument for grounding against stable references rather than raw coordinates wherever that option exists. And authentication friction, expired sessions, unexpected two-factor prompts, and CAPTCHAs remain a hard boundary computer-use agents are not meant to push through on their own, both because bypassing bot-detection mechanisms is exactly the kind of action that should never be automated without a human directly involved, and because a CAPTCHA appearing mid-task is frequently itself a signal that something about the automated pattern of behavior has already been noticed.
Choosing the right approach for your situation
The question worth asking before reaching for a general-purpose, vision-driven computer-use agent is not can this be done through the screen but does it actually need to be. If the target application exposes a real, documented API, using it directly is almost always faster, cheaper, and more reliable than operating the same functionality through a screen, because an API call either succeeds or returns a specific, actionable error, with none of the grounding or verification uncertainty a screen interaction carries by construction. Computer-use agents are a tool for the gap an API does not cover, not a universal replacement for using one where it already exists.
Within the set of tasks that genuinely have no better option, the right technical approach still depends heavily on specifics. A task performed occasionally, where a few extra seconds of latency per action is a non-issue, tolerates a heavier, more thorough verification loop far better than a task that needs to run at real volume, where the added engineering cost of getting grounding and caching right pays for itself quickly. A task operating on an application with a solid, well-maintained accessibility tree should lean into structured grounding as the default, reserving vision as a fallback rather than the primary strategy, while a task stuck operating legacy or canvas-rendered software has little choice but to lean on vision and set-of-mark techniques more heavily from the start.
The reversibility of what the task actually does should drive how much of the guardrail machinery described earlier gets built before the system ever runs unattended. A read-only task, gathering information displayed across several screens, checking a status, comparing values, can reasonably run with fairly light supervision, since a wrong click mostly costs a wasted step rather than a real-world consequence. A task that submits, purchases, deletes, or sends anything on someone else's behalf needs the full treatment: scoped sessions, explicit confirmation gates independent of the target application's own dialogs, and a complete action log, regardless of how infrequently that particular task actually runs, because the cost of one badly executed high-consequence action easily outweighs the convenience saved across a hundred low-stakes ones.
The case against over-engineering this
A team automating one specific, well-understood, rarely-changing interface, an internal tool everyone already knows the layout of, a single vendor's ordering page that has not changed meaningfully in years, often gets more reliability for less cost from a conventional, scripted browser automation tool with fixed selectors than from a general-purpose vision-driven agent making a fresh judgment call about the same screen every single time. A traditional script is faster, cheaper to run, and fails in an obvious, debuggable way the moment the interface actually changes, rather than silently reasoning its way to a plausible-looking but wrong action against a page it was never quite sure it understood correctly in the first place.
The case for a genuinely general, model-driven computer-use agent gets stronger specifically as the interface becomes unpredictable: many different applications the system was never specifically built for, layouts that change without notice, tasks defined at request time rather than hard-coded in advance. That is a meaningfully narrower set of situations than the current wave of enthusiasm around the category might suggest, and a team should be honest with itself about which situation it is actually in before committing to the heavier approach, since the heavier approach costs real latency and real engineering complexity that a fixed script simply does not carry.
The judgment call is the same one that shows up across most of engineering: match the approach to the actual variability of the problem, and revisit that judgment when the underlying assumptions change, an internal tool being replaced, a single-vendor integration turning into a need to support dozens of vendors' differently designed portals, a task that used to run occasionally suddenly needing to run at real volume. The mistake is not picking the simpler, scripted option for a genuinely narrow, stable problem; the mistake is failing to notice when the problem has quietly grown past what that simpler option was ever meant to handle.
Where this is heading
Screen-understanding accuracy has climbed sharply in a short window, and the trajectory shows no sign of flattening, which means the perception half of this problem is likely to keep becoming less of a bottleneck relative to grounding, verification, and guardrail design, the parts of the stack that depend on careful systems engineering rather than raw model capability. That shift matters because it changes where teams building on this technology should actually spend their effort: betting on the underlying model to keep getting better at reading a screen is a reasonably safe assumption at this point, while betting that grounding and safety infrastructure will take care of themselves is not.
Operating system vendors moving to build agent hooks directly into the platform, rather than leaving every implementation to reconstruct screen understanding entirely from raw pixels and synthetic input, is a meaningful direction of travel, because native support could offer exactly the kind of structured, reliable access that accessibility trees already provide for the applications that implement them well, extended to cover far more of the operating system than any one vendor's accessibility layer currently reaches on its own. This would not eliminate the vision fallback, plenty of software will always sit outside whatever a platform natively supports, but it would shrink the share of tasks that have to rely on the least precise grounding method as their only option.
The piece of infrastructure still most obviously missing is a shared, standardized way to describe what a given computer-use session is allowed to touch, which applications, which actions, which data, in a form portable across implementations rather than reinvented bespoke by every team building one of these systems. Tool-calling agents gained something like this through emerging shared protocols for connecting models to external systems; screen-operating agents do not yet have an equivalent for describing session scope and action permissions, and building one would do for this category roughly what those protocols did for tool connectivity, turning a decision every team currently makes inconsistently into a shared, well-scrutinized default most teams could simply adopt.
Getting started without overbuilding it
For a team adding computer-use capability to an agent today, the pragmatic starting point is narrower than the full survey above might suggest. Check for a real API before reaching for the screen at all, and use one wherever it exists, reserving screen operation specifically for the gap it does not cover. Where screen operation is genuinely necessary, prefer structured grounding, an accessibility tree or a labeled element reference, over raw coordinate prediction wherever the target application supports it well enough to trust, and treat vision-based coordinate guessing as the fallback for the residual cases rather than the default strategy for everything.
Build the observe-act-verify loop as a hard structural requirement from the first version rather than an enhancement to add once something goes wrong: every action gets checked against the resulting state before the agent moves on, not assumed successful because no error was thrown. Apply a wall-clock timeout and a maximum iteration count to every task before the first real one ever runs, and treat hitting either limit as a signal to stop and escalate rather than a bug to silently work around.
Classify actions by consequence before the agent runs unsupervised, and gate anything that submits, sends, deletes, purchases, or grants a permission behind a confirmation step that lives outside the target application's own dialogs, since those dialogs are not a safety mechanism the surrounding system can rely on. Scope every session narrowly to exactly what a task needs, a specific browser profile, a specific set of applications, rather than a general-purpose desktop the agent happens to have full access to, and log the complete trace of every action taken, including the observation that justified it, so that a failure afterward can be reconstructed rather than guessed at.
Operating a screen is one of the most broadly useful capabilities an agent can have, precisely because so much of the software worth automating was never going to get a real API, and one of the categories where the gap between an impressive demo and a system worth trusting unattended is unusually wide. The underlying models are already good enough to make this genuinely useful today; what determines whether a given deployment is reliable is almost entirely the discipline applied around them, grounding that does not rely on guesswork, verification that treats every action as unproven until checked, and guardrails that do not assume the interface itself will catch the mistakes the agent is capable of making.