- The problem
- In a deep language model, information from early layers can become harder for later layers to use. Existing fixes either retrieve the past in just one way or keep several costly copies of the model’s running state.
- The idea
- MSAR gives each past layer several views, each with a different amount of nearby word context, and lets each view independently choose which earlier layers matter. For example, one view can focus on very nearby context while another includes a wider window; the model then blends their retrieved information into one layer input. It creates these views when needed instead of storing all copies.
- What it got
- In tests at two large model sizes, MSAR beat the compared residual designs on all 12 benchmarks. It also reached the same training loss using 1.58× less compute than standard residual connections, according to the paper’s scaling-law experiments.