Researchers from Microsoft and City University of Hong Kong measured how AI agents choose between paths in long tasks, and the strongest system reached only 59.7 percent accuracy. Their Taste-Bench contains 502 questions built from recorded agent runs on coding and AI research work. Each question offers a binary choice between two directions at a branching point. The result matters for business because agents now run research and software projects alone, while a wrong early decision burns budget before its cost becomes visible.
How Taste-Bench tests decisions at forks
The benchmark freezes a trajectory at a decision fork, hides later steps, and asks a model to select the better path. Trajectories come from thousands of scored runs, including SWE-bench Pro and METR public transcript release MALT. A generator model selected suitable forks, removed trivial and undecidable cases, and human review confirmed almost all automatic labels. The set includes two fork types: parallel attempts at one task that diverge, and detours where an agent corrects itself inside a single run.
The evaluation covers 14 current models from Claude, GPT, Grok, DeepSeek, GLM, MiniMax, and Mistral families. Each question was presented twice with flipped option order, and credit required correct answers in both presentations, so random guessing yields 25 percent. GPT-5.6 Sol averaged 59.7 percent, followed by GPT-5.5 at 59.5 percent, with all other models scoring well below. Detour forks proved consistently harder than parallel forks. The authors state this skill differs from classic end-to-end tests such as SWE-bench Verified.
The design rests on hindsight rather than immediate grading. A clean and a sloppy implementation can produce the same output and pass the same tests at first, while the difference appears only when later changes become easier or harder. Human grading of such steps would demand deep expertise and would not scale. Because agent systems attempt tasks multiple times, final outcomes reveal which earlier branch was better. That later path supplies the label for the earlier choice without extra manual scoring.
What weak long-term judgment means for business
Companies that assign agents to multi-step coding or research work get completion without quality control of intermediate steps. Standard checks record whether tests pass at the end, yet METR found that SWE-bench Verified overestimates performance because professionals rejected many passing solutions for basic functional bugs. Taste-Bench adds a missing layer by scoring the quality of path selection. For large teams this means agent output needs review at branch points, not only acceptance testing, while small teams without senior reviewers face higher rework risk.
The limits are sharp when evidence arrives late. When the clue is already visible before the fork, average accuracy across models reaches 62.3 percent, but when the clue appears only after extensive later work, accuracy falls to 21 percent. Longer reasoning did not fix this: switching from the lowest to the highest reasoning setting barely changed results, and models spent the most tokens on questions they missed most often. On MirrorCode one model ran 19 days without human help, which shows how long an agent can follow a weak choice before anyone intervenes.
The direction to watch is whether training for judgment transfers into higher task success. Distilling a teacher model into Qwen3.6-27B raised accuracy on unseen tasks from 30 to 47.9 percent. When that student advice was added at each fork, success on 41 held-out SWE-bench Pro tasks rose from 14.6 to 33.7 percent, against 39 percent with perfect advice. If later studies repeat such gains on other task families, buyers will have a concrete marker that taste training is becoming a standard part of agent deployment.
