Reinforcement Learning from Human Feedback (RLHF) has become the standard approach to LLM alignment. However, the generation phase remains a critical bottleneck, consuming up to 80% of total training time due to slow autoregressive decoding — the model produces only one token per forward pass.

Speculative decoding addresses this by using a lightweight draft model to generate multiple candidate tokens in parallel, which the target model verifies in a single pass. This project integrates EAGLE3 speculative decoding into the VeRL training framework to accelerate this generation phase.

However, two challenges arise in a training context that do not exist in static inference: the effective batch size fluctuates dynamically as sequences complete at different rates, and the target model’s continuous updates progressively degrade draft model alignment. The system addresses both through the two contributions below.