- The problem
- When an AI assistant uses tools over several exchanges, a final success or failure does not show which particular response caused it. Later steps can vary even when the current response is the same, so training on the wrong exchange may not help.
- The idea
- Critical-State RL checks which candidate response has a useful connection to task success. It tries different responses from the same situation, then repeats what happens afterward while holding each response fixed, separating the response’s effect from later randomness. It trains the chosen response while using the rest of the interaction as context rather than updating every exchange.
- What it got
- In a benchmark study, training the selected response improved tool-use accuracy, while training alternative responses often did not. The key gain was about 14 percentage points on a missing-function task.