Results

What egin scored on each benchmark, against eve and a plain bash loop, including the switches that did not help.

Every number here came from one binary, one microVM per task, in November 2026: Qwen4 Max and the other router models on an EC2 m7g.metal through the You.com router, and the Claude and GPT rows on Amazon Bedrock. WeirdML is the exception: its tasks ran through bench/weirdml/run.py on a GPU host. Benchmarks has the command for each suite. A run of 20 tasks moves by two or three passes between repeats, so read small gaps as ties.

Against eve, full suites

All 165 GAIA validation tasks and all 1,000 SimpleQA Verified questions, egin and eve on the same model with the same web tools. GAIA ran with a $1 limit per task, a 300-turn cap (in effect no limit) and exclude_echo on, and is audited: a pass is dropped when the model was shown agent or benchmark writing that quotes the question next to its answer. The page a question is about still counts. That dropped at most one pass per run.

Model GAIA, egin GAIA, eve GAIA cost, egin GAIA cost, eve SimpleQA, egin SimpleQA, eve
Claude Sonnet 5.5 136 130 $16.46 $23.76 963 960
Qwen4 Max 136 119 $19.10 $27.77 949 944
GPT-6.1 Sol 129 124 $12.58 $22.35 965 964
Claude Haiku 5.5 114 108 $1.19 $1.74 947 950
GPT-6 Luna 103 98 $0.80 $1.89 963 961

egin passes more GAIA tasks on every model, at 31 to 58% less cost. SimpleQA is a tie: both harnesses find short facts equally well, within a few questions in 1,000.

Against eve, all router models, no spend limit

All 165 GAIA validation tasks on 12 router models and all 1,000 SimpleQA Verified questions on 15, egin against eve 1.0.2, with no spend limit for either. egin keeps its 300-turn cap; eve has no turn limit. GAIA ran with exclude_echo on. Passes, with the total cost across all models at the bottom.

Model GAIA, egin GAIA, eve SimpleQA, egin SimpleQA, eve
kimi-k3.1-pro 138 133 974 971
glm-5.3 133 120 968 975
qwen3.8-max 132 122 966 959
muse-wonder-1.0 130 132 959 943
kimi-k3 129 128 970 966
qwen3.8-flash 127 100 961 941
deepseek-v4-pro 126 118 972 970
glm-5.3-flash 125 125 959 973
beam-501b-a23b 122 115 966 964
mercury-3 120 111 966 971
minimax-m3 99 86 961 962
gpt-oss-120b 76 79 952 946
deepseek-v4.1-flash 968 969
muse-spark-1.3 946 952
muse-ember-1.0 961 952
Total 1,457 (73.6%) 1,369 (69.1%) 14,449 (96.3%) 14,414 (96.1%)
Cost $949 $2,792 $278 $667

egin leads GAIA on 9 of 12 models and ties one. SimpleQA is a tie. Most of eve’s extra spend is a few runs that never stop: a typical run costs the same in both ($0.036 on GAIA), but 24 eve GAIA runs cost $20 or more each, $1,755 together, and 11 of them passed. The largest, on kimi-k3.1-pro, read 653 million prompt tokens over three hours for $363.70 and failed. egin had 3 runs over $20.

With a $1 limit per task the same runs score egin 1,427 and eve 1,268 on GAIA (72.1% and 64.0%), and 14,398 and 14,075 on SimpleQA (96.0% and 93.8%). At the limit eve stops with no answer, which is most of that gap; egin makes one last call with no tools.

minimax-m3 fails requests above about 75,000 tokens of context, though the router lists 524,288. It affects both harnesses on long GAIA tasks.

GAIA, against eve, router models

165 validation tasks. Both harnesses ran with the same web tools, a $1 limit per task, a 300-turn cap, and exclude_echo on. “Audited” drops a pass when the model was shown agent or benchmark writing that quotes the question next to its answer.

Model eve eve audited eve cost egin egin audited egin cost
deepseek-v4.1-flash 120 120 $19.19 130 128 $13.57
muse-ember-1.0 121 120 $12.15 124 121 $6.09
muse-spark-1.3 113 112 $49.22 125 124 $39.68
egin eval out/gaia --model deepseek-v4.1-flash --jobs 6 --set max_turns=300 --set exclude_echo=true

Most of the gap on muse-spark-1.3 is what each harness does at the $1 limit. 28 eve runs reached it (21 on muse-spark-1.3) and stopped with no answer. 14 egin runs on muse-spark-1.3 reached it too; egin then makes one last call with no tools to get an answer, and 3 of those passed.

Without exclude_echo, raw scores are higher for both and about one pass in six is a leak: eve 138, 133 and 119, egin 144, 136 and 124, in the same model order. Audited, those are eve 110, 110 and 105, and egin 120, 115 and 114.

SimpleQA Verified

100 questions, passes out of 100. client-web is egin’s own search and fetch tools, router-web lets the router search, closed-book has no web.

Model client-web router-web closed-book eve
deepseek-v4.1-flash 96 97 35 96
muse-ember-1.0 95 96 19 94
muse-spark-1.3 96 97 26 93
kimi-k3.1-pro 98 98 58
deepseek-v4-pro 96 96 55
qwen3.8-max 96 98 47
qwen3.8-flash 96 98 33
glm-5.3-flash 96 96 36
beam-501b-a23b 95 97 26
mercury-3 95 94 37
minimax-m3 95 99 23
kimi-k3 94 96 42
muse-wonder-1.0 94 94 28
gpt-oss-120b 93 95 15
glm-5.3 90 98 31

GraphWalks

60 problems. Inline puts the graph in the prompt and measures the model. File puts it in a file the agent reads with tools and measures the harness.

Model inline file file, no compaction file, no elision
glm-5.3-flash 54 60 60 60
qwen3.8-flash 50 60 60 60
kimi-k3.1-pro 47 60 60 60
mercury-3 47 60 60 60
qwen3.8-max 46 60 59 60
glm-5.3 45 58 60 60
kimi-k3 42 60 60 60
deepseek-v4-pro 41 60 60 60
muse-wonder-1.0 33 60 56 60
beam-501b-a23b 27 60 60 59
minimax-m3 36 60 43 58
gpt-oss-120b 22 47 50 50

The router’s stream drops a closing ] from minimax-m3 after tool calls, so every one of its answers arrived as Final Answer: [a, b. The grader accepts a list that runs to the end of the reply without its bracket; the minimax-m3 row is its original runs graded that way.

Terminal-Bench 2

All 89 tasks, against eve on the same microVMs and model, with no spend limit. Two task environments failed to build, so passes are out of 87.

Model egin eve
kimi-k3.1-pro 68 66
qwen4-max 61 62
glm-5.3 52 56
Total 181 184

A tie within run-to-run variation: eve passes 19 tasks egin fails and egin passes 17 eve fails. Eight of eve’s wins were egin runs that ran out of wall-clock time while waiting on one slow command. These runs predate background jobs in the benchmark agent; rerun with it, egin passed 3 of those 8.

FrontierHarness

The 21 Terminal-Bench 2.1 tasks from FrontierHarness Eval v1.1, on kimi-k3 like its leaderboard, one run each. Leaderboard rows are from its published data; our runs used our own router, so compare passes, not cost.

Passes of 21 Harness
18 eve 1.0.2
17 egin
16 claude-code, pi-responses, dsh-ptc, dsh-standard, oh-my-pi, exo
15 codex, dsh-creator, dsh-minimal, opencode
14 kimi-code
13 hermes

The subset in bench/tb2/subset.txt, passes out of 20.

Model egin
muse-ember-1.0 17
glm-5.3 16
glm-5.3-flash 16
kimi-k3.1-pro 16
mercury-3 16
qwen3.8-flash 16
deepseek-v4-pro 15
qwen3.8-max 15
muse-spark-1.3 14
beam-501b-a23b 13
kimi-k3 13
minimax-m3 12
muse-wonder-1.0 12
gpt-oss-120b 8

Against eve and a plain bash loop (bash only, one call a turn, full output, no context management):

Model egin bash loop eve
muse-ember-1.0 17 19 13
muse-spark-1.3 14 13 6

SWE-bench Verified

Ten Django instances, sealed so the upstream fix is out of reach. Passes out of 10. All fifteen router models scored 8 or 9 with egin, so this subset is too easy to separate them.

Model egin bash loop eve
deepseek-v4.1-flash 8 8 8
muse-ember-1.0 8 4 9
muse-spark-1.3 9 7 8

WeirdML-style tasks

Three small machine-learning tasks on a shared H100, three runs each, scored by test accuracy averaged over runs. The official data is private, so the shapes are regenerated to the published spec and the chess games come from the open Lichess database. This measures the model more than the harness and is not comparable to the WeirdML leaderboard. A run that ended in an error counts as zero, which is most of what pulls down the last three rows.

Model chess winners shapes easy shapes hard mean
muse-wonder-1.0 0.85 1.00 0.97 0.94
glm-5.3-flash 0.84 1.00 0.93 0.92
kimi-k3 0.80 1.00 0.97 0.92
muse-ember-1.0 0.84 1.00 0.88 0.90
glm-5.3 0.84 1.00 0.84 0.89
muse-spark-1.3 0.66 1.00 0.92 0.86
minimax-m3 0.74 1.00 0.75 0.83
mercury-3 0.55 1.00 0.88 0.81
deepseek-v4-pro 0.55 1.00 0.79 0.78
kimi-k3.1-pro 0.47 1.00 0.79 0.75
deepseek-v4.1-flash 0.74 1.00 0.49 0.74
gpt-oss-120b 0.66 0.98 0.30 0.64
qwen3.8-max 0.62 0.60 0.43 0.55
qwen3.8-flash 0.39 0.50 0.41 0.43
beam-501b-a23b 0.00 0.12 0.12 0.08

Switches that did not help

Each arm changes one feature from the default on the Terminal-Bench subset (no elision also turns off summarisation; background jobs also changes bash_wait). Passes out of 20 unless a run was cut short.

Model default no compaction no elision no batching planning bash only read before write background jobs
kimi-k3.1-pro 19 19 16 17 19 16 15 17
muse-ember-1.0 17 18 17 19 18 19 16 15
qwen3.8-flash 15 12 13 12 11 13 13 12
mercury-3 14 12 13 17 13 15 10 14
glm-5.3-flash 13 12 13 15 14 14 12 11
deepseek-v4-pro 12 15 14 15 15 15 15 15
minimax-m3 12 10 10 10 9 12 10 12
muse-wonder-1.0 10 13 10 9 9 11 13 11

No switch helps or hurts across models by more than run-to-run variation on this subset. Background jobs is now the default in the benchmark agent too, as it already was in egin, because on the full 89 tasks it recovered runs that otherwise timed out waiting on one command. The one model-specific setting kept is bench/tb2/profiles/muse-ember-1.0.toml, where bash only with no batching tied the default over two repeat runs, 35 of 40 each (counting as a fail one run that fetched the reference solution), at 21% lower cost and about 15% faster. It is kept for the cost, not the score.

Five more models (beam-501b-a23b, glm-5.3, gpt-oss-120b, kimi-k3, qwen3.8-max) ran the same arms but their runs hung one or two tasks short. Their partial counts show no arm beating the default by more than two passes; the largest drops are glm-5.3 without elision (10 passes against 15) and gpt-oss-120b with bash only (7 against 11), both on runs a task or two short.

Two other ideas were measured and dropped:

  • A review step, where the model checks its final answer once: across 495 GAIA runs it fixed 4 answers and broke 7, at about 1.6 times the cost.
  • Majority vote over three runs: estimated from existing GAIA runs at 141, 141 and 124, no better than one run.

What moved the score

On GAIA, in order of size: showing long pages and search results a part at a time instead of cutting them (89% of searches and 31% of fetches were being cut), a final answer call with no tools when a limit is hit, and transcribing audio files.

Benchmark lookups

In the Terminal-Bench ablation runs, a scan of the traces found three runs out of about 2,000 that fetched the Terminal-Bench repository itself (two on count-dataset-tokens, one on headless-terminal). They are left in the counts above.