Guide · Computer use
What is computer use?
Computer use means an AI model operating a computer the way a person does: it looks at the screen, then clicks, types and scrolls to get a job done. Here is how it works, what can go wrong, and how Solen for Windows does it.
What it means
A chat assistant answers in words, and you still do the clicking. With computer use, the AI takes the steps itself: it opens the app, finds the button, fills in the box and presses Send.
Because it works through the same screens you do, it can work in apps that have no plug-in or connection for AI. That is the appeal, and also why it needs care.
How it works
Most computer-use assistants run the same loop:
- Look. The assistant gives an AI model your request and what is on the screen: a picture of the window, and often the text and controls Windows can report about it.
- Decide. The model picks the next step: click here, type this, press a key, scroll.
- Act and look again. The assistant does that step, looks at the result, and the loop repeats until the job is done or the model needs you.
Each turn of the loop is a request to an AI model. That is why a job with many steps takes longer, and costs more, than one with a few.
What can go wrong
- Wrong step. The model can click the wrong button, type into the wrong box or misread a screen.
- Wrong place. Without limits, it could act in a window that has nothing to do with your job.
- Instructions hidden in content. A web page or document can contain text written to steer the AI. Microsoft calls this cross-prompt injection and warns that such content “can override agent instructions”.
- Saying Done too soon. A model can report success when the last step did not work.
- Privacy. When the model runs online, pictures and text from your screen are sent to it.
Safeguards worth looking for
- Limits on which windows it may act in.
- A question before anything costly or hard to undo, such as sending, paying or deleting.
- A way to stop it at once.
- An ending that says plainly what was done and what was not.
- A clear statement of what leaves your computer.
How Solen for Windows does it
Solen for Windows is a computer-use assistant for Windows. Here is how the released Beta, 0.26.1 beta 2, handles each of those points.
Where it works
Only in the window you were in when you asked, windows the job opens, and windows you agree to. For any other window it asks: “May Solen work in it? Only that window, and only for this task.” A no lasts for the whole job.
How it sees and acts
It reads the window’s text and controls, takes a picture when it needs to see, then clicks, types, scrolls, drags or presses keys, and looks again. Clicks on a control often go through Windows accessibility without moving the pointer; scrolling, dragging, keys and some clicks use your real mouse and keyboard.
When it asks
Sending to other people, paying, deleting for good, and security or access changes wait for your yes, unless your request asked for exactly that step. No answer within two minutes counts as no.
Stopping
Press Ctrl + Alt + Esc or the Stop button, and Solen says what it had already done. If you start typing or clicking while Solen is typing, its typing stops.
How it ends
Each request ends with a plain ending, such as Done, Answer, Have a look or Didn’t finish, followed by the answer or a count of what it did. If nothing changed or you stopped it, it says so. If it pressed something risky, such as Publish or Send, and hasn’t looked at that window since, it says Have a look, not Done. These checks can miss mistakes, so review important results.
What leaves your PC
Solen runs on online AI models. To do a job it sends the model your request, the text it reads from the job’s windows and, when it needs to see a window, a picture of that window. It won’t take a picture that would show a password field Windows knows about, or of a private or password-manager window it recognizes; these checks can miss some apps. See what stays on your computer.
The full step-by-step is on How it works.
Where it stands today
Computer use is still early. In one research test at three times our long-job length, using a stand-in AI connection with tighter limits than the released app, none of 10 runs finished, and some were stopped at a cost limit. The same research recorded long jobs that did finish through a different route, so this is not the released Beta’s success rate. The lab notes give the numbers and how each was measured.
The Solen Beta is free and invitation-only, for 64-bit Windows 11; Windows 10 and ARM PCs are untested.