FDE-Gym: The Last Mile of AI Delivery

The short version

The problem

AI agents can do well when told exactly what task to complete, but customer projects often start with a vague or misleading request. Engineers must find out what the customer really needs, navigate existing systems and people, and avoid changes that make things worse.

The idea

FDE-Gym is a practice and testing environment built from realistic customer projects. An agent can inspect a company’s files and systems, ask scripted customer staff questions, and make changes; its score reflects whether the customer’s real-world measure improved, while accounting for the cost of asking and operational mistakes. For example, a request for a chat tool might conceal a more important problem in an existing business process.

What it got

Across 13 tested AI configurations, the best earned an average score of 0.56 on a scale where 1 matches an expert-built solution. It fully resolved only 14 of the benchmark’s 95 tasks.