In Checking the assistant’s work, we started measuring how an assistant handles everyday tasks. An assistant that can use your PC’s screen is part of what we want a Disanchor PC to offer, so we need an honest way to tell assistants apart. We’re now building a fairer test: three different assistant setups will take the same tasks on the same kind of desktop, and we’ll compare them.
The same conditions for everyone
A comparison only means something if nobody gets an easier version. Every assistant will get the same screenshot, at the same size and in the same format, and the same set of mouse and keyboard actions. We’re making sure no screenshot is quietly shrunk on its way to one of them.
The tasks come in two stages, from simple to harder. Each assistant will try each task more than once, starting every time from a freshly restored desktop, so a leftover from one try can’t help the next.
The assistants have to work through the screen, the way a person would. Opening a terminal at all during a run is against the rules, and that run counts as a failure.
Answers the assistant can’t reach
An assistant saying “done” won’t count. A separate check will look at the result after the task is over, and the expected answers stay out of the assistants’ reach the whole time. If something breaks on our side, that run will be set aside and repeated. If the assistant breaks something, it counts against the assistant.
Every run will keep its record: the screenshots, each action and how long each step took. We’ll describe exactly how each setup was configured, because one assistant’s result can depend as much on its surrounding tools as on the model inside it.
Reviewed before it runs
Each part of the test is written by one AI coding tool and reviewed by another, round after round, before anything runs for real. The reviews have already caught real problems. One was a naming clash that would have made every click fail. Another was a pre-task check that could have let an assistant find the answers.
What this is, and what it isn’t
This is the test itself, not results, and not a feature you can use. Nothing has run on a real PC yet. The next step is a short check on a real desktop, and it starts only with our approval. When results exist, we’ll publish them together with the method, including the parts that went wrong.
Which everyday tasks would you trust an assistant with on your PC? Tell us in #feedback on our Discord.
Where we are: building a fair, independently checked test that compares assistants on the same desktop tasks. It hasn’t run on a real PC, there are no results yet, and early access is closed.