Benchmarks
Solen Bench
How well different AI models do real tasks on a real computer when they work through Solen. Every result is checked in the app's own saved data, never taken from the model's word.
Solen Bench — independent analysis of AI models operating a real computer
Solen Index i
How much of the work each model actually got done on a real computer, checked in each app's own stored state after the run. Higher is better.
Solen over builds i
The same planner model on different Solen builds: what the runtime itself changed. Each build is scored only on the tasks every build of that model ran.
Highlights
Six numbers per model. Each card is sorted best first.
False completions i
Runs where Solen told the person the task was done while the app's own data says it was not. Lower is better; zero is the goal. This is the number Solen Bench exists for.
Suites
Share of runs done in each live benchmark suite, hardest first. A run is done only when the app's store shows the asked change and nothing else moved. Suites every model already passes are retired and replaced by a harder version: they are listed at the bottom of the page.
Quality vs speed and cost
The shaded corner is the most attractive quadrant: above the median index and below the median time or cost.
Solen Index vs seconds per task i
Median wall-clock seconds per run (x) against the Solen Index (y).
Solen Index vs cost per task i
Estimated US$ per run at API list price (x) against the Solen Index (y).
Full table
Every model and metric. Click a column to sort.
Tasks
Each task, what Solen is asked, how the result is checked, and how many of each model's runs were done.
Models
One model at a time: its suite profile and every run it made.
Methodology
How the numbers are made, and what they do not show.
Retired suites
Every model that ran these scored 90% or more, or a harder version now replaces them. They are left out of the Solen Index and the Suites above; their old results stay in the tables.