LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketches

The short version

The problem

Training a language model with feedback from its answers can require far more memory than simply running the model, because training normally stores large gradients and optimizer data. This can put reinforcement learning out of reach for teams whose hardware can run a model but cannot train it.

The idea

LoGRA replaces each large gradient with a compact sketch: a smaller record that preserves a useful direction for changing the model. It builds and sends this compact update without first storing the full gradient, then estimates how much the change would affect the model’s next-word predictions and scales it down if needed—like checking a steering adjustment before making the full turn.

What it got

LoGRA cut average training memory by up to 45.7% while maintaining comparable reasoning performance. It also trained a 27-billion-parameter model for more than 1,100 steps on one eight-GPU system, where the standard Adam approach ran out of memory.