GRPO testbed: same training run, three different “improvements“ depending on measurement
In open reinforcement learning from verifiable rewards (RLVR), whether a model actually improved depends on which metric you use — reward signal, evaluation benchmark, or decoding strategy. The same training run can look like success, failure, or even reversal.
Jules Roussel built a small testbed on Qwen 2.5–0.5B to separate reward functions, metrics, and extractors, exposing how most GRPO pipelines conflate all three, making accuracy gains partly an artifact of the instrument, not the model.
Single-seed exploratory study on GSM8K with held-out evals; confident in measurement failures, tentative on rankings.