Results
What egin scored on each benchmark, against eve and a plain bash loop, including the switches that did not help.
Every number here came from one binary, one microVM per task, in November 2026: Qwen4 Max and the other router models on an EC2 m7g.metal through the You.com router, and the Claude and GPT rows on Amazon Bedrock. WeirdML is the exception: its tasks ran through bench/weirdml/run.py on a GPU host. Benchmarks has the command for each suite. A run of 20 tasks moves by two or three passes between repeats, so read small gaps as ties.
Against eve, full suites
All 165 GAIA validation tasks and all 1,000 SimpleQA Verified questions, egin and eve on the same model with the same web tools. GAIA ran with a $1 limit per task, a 300-turn cap (in effect no limit) and exclude_echo on, and is audited: a pass is dropped when the model was shown agent or benchmark writing that quotes the question next to its answer. The page a question is about still counts. That dropped at most one pass per run.
| Model | GAIA, egin | GAIA, eve | GAIA cost, egin | GAIA cost, eve | SimpleQA, egin | SimpleQA, eve |
|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | 136 | 130 | $16.46 | $23.76 | 963 | 960 |
| Qwen4 Max | 136 | 119 | $19.10 | $27.77 | 949 | 944 |
| GPT-6.1 Sol | 129 | 124 | $12.58 | $22.35 | 965 | 964 |
| Claude Haiku 5.5 | 114 | 108 | $1.19 | $1.74 | 947 | 950 |
| GPT-6 Luna | 103 | 98 | $0.80 | $1.89 | 963 | 961 |
egin passes more GAIA tasks on every model, at 31 to 58% less cost. SimpleQA is a tie: both harnesses find short facts equally well, within a few questions in 1,000.
Against eve, all router models, no spend limit
All 165 GAIA validation tasks on 12 router models and all 1,000 SimpleQA Verified questions on 15, egin against eve 1.0.2, with no spend limit for either. egin keeps its 300-turn cap; eve has no turn limit. GAIA ran with exclude_echo on. Passes, with the total cost across all models at the bottom.
| Model | GAIA, egin | GAIA, eve | SimpleQA, egin | SimpleQA, eve |
|---|---|---|---|---|
| kimi-k3.1-pro | 138 | 133 | 974 | 971 |
| glm-5.3 | 133 | 120 | 968 | 975 |
| qwen3.8-max | 132 | 122 | 966 | 959 |
| muse-wonder-1.0 | 130 | 132 | 959 | 943 |
| kimi-k3 | 129 | 128 | 970 | 966 |
| qwen3.8-flash | 127 | 100 | 961 | 941 |
| deepseek-v4-pro | 126 | 118 | 972 | 970 |
| glm-5.3-flash | 125 | 125 | 959 | 973 |
| beam-501b-a23b | 122 | 115 | 966 | 964 |
| mercury-3 | 120 | 111 | 966 | 971 |
| minimax-m3 | 99 | 86 | 961 | 962 |
| gpt-oss-120b | 76 | 79 | 952 | 946 |
| deepseek-v4.1-flash | 968 | 969 | ||
| muse-spark-1.3 | 946 | 952 | ||
| muse-ember-1.0 | 961 | 952 | ||
| Total | 1,457 (73.6%) | 1,369 (69.1%) | 14,449 (96.3%) | 14,414 (96.1%) |
| Cost | $949 | $2,792 | $278 | $667 |
egin leads GAIA on 9 of 12 models and ties one. SimpleQA is a tie. Most of eve’s extra spend is a few runs that never stop: a typical run costs the same in both ($0.036 on GAIA), but 24 eve GAIA runs cost $20 or more each, $1,755 together, and 11 of them passed. The largest, on kimi-k3.1-pro, read 653 million prompt tokens over three hours for $363.70 and failed. egin had 3 runs over $20.
With a $1 limit per task the same runs score egin 1,427 and eve 1,268 on GAIA (72.1% and 64.0%), and 14,398 and 14,075 on SimpleQA (96.0% and 93.8%). At the limit eve stops with no answer, which is most of that gap; egin makes one last call with no tools.
minimax-m3 fails requests above about 75,000 tokens of context, though the router lists 524,288. It affects both harnesses on long GAIA tasks.
GAIA, against eve, router models
165 validation tasks. Both harnesses ran with the same web tools, a $1 limit per task, a 300-turn cap, and exclude_echo on. “Audited” drops a pass when the model was shown agent or benchmark writing that quotes the question next to its answer.
| Model | eve | eve audited | eve cost | egin | egin audited | egin cost |
|---|---|---|---|---|---|---|
| deepseek-v4.1-flash | 120 | 120 | $19.19 | 130 | 128 | $13.57 |
| muse-ember-1.0 | 121 | 120 | $12.15 | 124 | 121 | $6.09 |
| muse-spark-1.3 | 113 | 112 | $49.22 | 125 | 124 | $39.68 |
egin eval out/gaia --model deepseek-v4.1-flash --jobs 6 --set max_turns=300 --set exclude_echo=true
Most of the gap on muse-spark-1.3 is what each harness does at the $1 limit. 28 eve runs reached it (21 on muse-spark-1.3) and stopped with no answer. 14 egin runs on muse-spark-1.3 reached it too; egin then makes one last call with no tools to get an answer, and 3 of those passed.
Without exclude_echo, raw scores are higher for both and about one pass in six is a leak: eve 138, 133 and 119, egin 144, 136 and 124, in the same model order. Audited, those are eve 110, 110 and 105, and egin 120, 115 and 114.
SimpleQA Verified
100 questions, passes out of 100. client-web is egin’s own search and fetch tools, router-web lets the router search, closed-book has no web.
| Model | client-web | router-web | closed-book | eve |
|---|---|---|---|---|
| deepseek-v4.1-flash | 96 | 97 | 35 | 96 |
| muse-ember-1.0 | 95 | 96 | 19 | 94 |
| muse-spark-1.3 | 96 | 97 | 26 | 93 |
| kimi-k3.1-pro | 98 | 98 | 58 | |
| deepseek-v4-pro | 96 | 96 | 55 | |
| qwen3.8-max | 96 | 98 | 47 | |
| qwen3.8-flash | 96 | 98 | 33 | |
| glm-5.3-flash | 96 | 96 | 36 | |
| beam-501b-a23b | 95 | 97 | 26 | |
| mercury-3 | 95 | 94 | 37 | |
| minimax-m3 | 95 | 99 | 23 | |
| kimi-k3 | 94 | 96 | 42 | |
| muse-wonder-1.0 | 94 | 94 | 28 | |
| gpt-oss-120b | 93 | 95 | 15 | |
| glm-5.3 | 90 | 98 | 31 |
GraphWalks
60 problems. Inline puts the graph in the prompt and measures the model. File puts it in a file the agent reads with tools and measures the harness.
| Model | inline | file | file, no compaction | file, no elision |
|---|---|---|---|---|
| glm-5.3-flash | 54 | 60 | 60 | 60 |
| qwen3.8-flash | 50 | 60 | 60 | 60 |
| kimi-k3.1-pro | 47 | 60 | 60 | 60 |
| mercury-3 | 47 | 60 | 60 | 60 |
| qwen3.8-max | 46 | 60 | 59 | 60 |
| glm-5.3 | 45 | 58 | 60 | 60 |
| kimi-k3 | 42 | 60 | 60 | 60 |
| deepseek-v4-pro | 41 | 60 | 60 | 60 |
| muse-wonder-1.0 | 33 | 60 | 56 | 60 |
| beam-501b-a23b | 27 | 60 | 60 | 59 |
| minimax-m3 | 36 | 60 | 43 | 58 |
| gpt-oss-120b | 22 | 47 | 50 | 50 |
The router’s stream drops a closing ] from minimax-m3 after tool calls, so every one of its answers arrived as Final Answer: [a, b. The grader accepts a list that runs to the end of the reply without its bracket; the minimax-m3 row is its original runs graded that way.
Terminal-Bench 2
All 89 tasks, against eve on the same microVMs and model, with no spend limit. Two task environments failed to build, so passes are out of 87.
| Model | egin | eve |
|---|---|---|
| kimi-k3.1-pro | 68 | 66 |
| qwen4-max | 61 | 62 |
| glm-5.3 | 52 | 56 |
| Total | 181 | 184 |
A tie within run-to-run variation: eve passes 19 tasks egin fails and egin passes 17 eve fails. Eight of eve’s wins were egin runs that ran out of wall-clock time while waiting on one slow command. These runs predate background jobs in the benchmark agent; rerun with it, egin passed 3 of those 8.
FrontierHarness
The 21 Terminal-Bench 2.1 tasks from FrontierHarness Eval v1.1, on kimi-k3 like its leaderboard, one run each. Leaderboard rows are from its published data; our runs used our own router, so compare passes, not cost.
| Passes of 21 | Harness |
|---|---|
| 18 | eve 1.0.2 |
| 17 | egin |
| 16 | claude-code, pi-responses, dsh-ptc, dsh-standard, oh-my-pi, exo |
| 15 | codex, dsh-creator, dsh-minimal, opencode |
| 14 | kimi-code |
| 13 | hermes |
The subset in bench/tb2/subset.txt, passes out of 20.
| Model | egin |
|---|---|
| muse-ember-1.0 | 17 |
| glm-5.3 | 16 |
| glm-5.3-flash | 16 |
| kimi-k3.1-pro | 16 |
| mercury-3 | 16 |
| qwen3.8-flash | 16 |
| deepseek-v4-pro | 15 |
| qwen3.8-max | 15 |
| muse-spark-1.3 | 14 |
| beam-501b-a23b | 13 |
| kimi-k3 | 13 |
| minimax-m3 | 12 |
| muse-wonder-1.0 | 12 |
| gpt-oss-120b | 8 |
Against eve and a plain bash loop (bash only, one call a turn, full output, no context management):
| Model | egin | bash loop | eve |
|---|---|---|---|
| muse-ember-1.0 | 17 | 19 | 13 |
| muse-spark-1.3 | 14 | 13 | 6 |
SWE-bench Verified
Ten Django instances, sealed so the upstream fix is out of reach. Passes out of 10. All fifteen router models scored 8 or 9 with egin, so this subset is too easy to separate them.
| Model | egin | bash loop | eve |
|---|---|---|---|
| deepseek-v4.1-flash | 8 | 8 | 8 |
| muse-ember-1.0 | 8 | 4 | 9 |
| muse-spark-1.3 | 9 | 7 | 8 |
WeirdML-style tasks
Three small machine-learning tasks on a shared H100, three runs each, scored by test accuracy averaged over runs. The official data is private, so the shapes are regenerated to the published spec and the chess games come from the open Lichess database. This measures the model more than the harness and is not comparable to the WeirdML leaderboard. A run that ended in an error counts as zero, which is most of what pulls down the last three rows.
| Model | chess winners | shapes easy | shapes hard | mean |
|---|---|---|---|---|
| muse-wonder-1.0 | 0.85 | 1.00 | 0.97 | 0.94 |
| glm-5.3-flash | 0.84 | 1.00 | 0.93 | 0.92 |
| kimi-k3 | 0.80 | 1.00 | 0.97 | 0.92 |
| muse-ember-1.0 | 0.84 | 1.00 | 0.88 | 0.90 |
| glm-5.3 | 0.84 | 1.00 | 0.84 | 0.89 |
| muse-spark-1.3 | 0.66 | 1.00 | 0.92 | 0.86 |
| minimax-m3 | 0.74 | 1.00 | 0.75 | 0.83 |
| mercury-3 | 0.55 | 1.00 | 0.88 | 0.81 |
| deepseek-v4-pro | 0.55 | 1.00 | 0.79 | 0.78 |
| kimi-k3.1-pro | 0.47 | 1.00 | 0.79 | 0.75 |
| deepseek-v4.1-flash | 0.74 | 1.00 | 0.49 | 0.74 |
| gpt-oss-120b | 0.66 | 0.98 | 0.30 | 0.64 |
| qwen3.8-max | 0.62 | 0.60 | 0.43 | 0.55 |
| qwen3.8-flash | 0.39 | 0.50 | 0.41 | 0.43 |
| beam-501b-a23b | 0.00 | 0.12 | 0.12 | 0.08 |
Switches that did not help
Each arm changes one feature from the default on the Terminal-Bench subset (no elision also turns off summarisation; background jobs also changes bash_wait). Passes out of 20 unless a run was cut short.
| Model | default | no compaction | no elision | no batching | planning | bash only | read before write | background jobs |
|---|---|---|---|---|---|---|---|---|
| kimi-k3.1-pro | 19 | 19 | 16 | 17 | 19 | 16 | 15 | 17 |
| muse-ember-1.0 | 17 | 18 | 17 | 19 | 18 | 19 | 16 | 15 |
| qwen3.8-flash | 15 | 12 | 13 | 12 | 11 | 13 | 13 | 12 |
| mercury-3 | 14 | 12 | 13 | 17 | 13 | 15 | 10 | 14 |
| glm-5.3-flash | 13 | 12 | 13 | 15 | 14 | 14 | 12 | 11 |
| deepseek-v4-pro | 12 | 15 | 14 | 15 | 15 | 15 | 15 | 15 |
| minimax-m3 | 12 | 10 | 10 | 10 | 9 | 12 | 10 | 12 |
| muse-wonder-1.0 | 10 | 13 | 10 | 9 | 9 | 11 | 13 | 11 |
No switch helps or hurts across models by more than run-to-run variation on this subset. Background jobs is now the default in the benchmark agent too, as it already was in egin, because on the full 89 tasks it recovered runs that otherwise timed out waiting on one command. The one model-specific setting kept is bench/tb2/profiles/muse-ember-1.0.toml, where bash only with no batching tied the default over two repeat runs, 35 of 40 each (counting as a fail one run that fetched the reference solution), at 21% lower cost and about 15% faster. It is kept for the cost, not the score.
Five more models (beam-501b-a23b, glm-5.3, gpt-oss-120b, kimi-k3, qwen3.8-max) ran the same arms but their runs hung one or two tasks short. Their partial counts show no arm beating the default by more than two passes; the largest drops are glm-5.3 without elision (10 passes against 15) and gpt-oss-120b with bash only (7 against 11), both on runs a task or two short.
Two other ideas were measured and dropped:
- A review step, where the model checks its final answer once: across 495 GAIA runs it fixed 4 answers and broke 7, at about 1.6 times the cost.
- Majority vote over three runs: estimated from existing GAIA runs at 141, 141 and 124, no better than one run.
What moved the score
On GAIA, in order of size: showing long pages and search results a part at a time instead of cutting them (89% of searches and 31% of fetches were being cut), a final answer call with no tools when a limit is hit, and transcribing audio files.
Benchmark lookups
In the Terminal-Bench ablation runs, a scan of the traces found three runs out of about 2,000 that fetched the Terminal-Bench repository itself (two on count-dataset-tokens, one on headless-terminal). They are left in the counts above.