Frontier post-training recipe review with Finbarr Timbers
AI Summary
Nathan Lambert and Finbarr Timbers discuss the evolution of frontier model post-training recipes, moving from classic InstructGPT three-step methods to modern reinforcement learning, multi-teacher on-policy distillation, and domain-specific specialists. The conversation covers how organizational constraints impact post-training workflows, the declining use of Direct Preference Optimization (DPO) in refined pipelines, and the operational trade-offs of managing custom training infrastructure versus APIs.
Key Takeaways
- Post-training workflows are heavily influenced by an organization's capacity and compute management.
- Direct Preference Optimization (DPO) is disappearing from leading recipes as industrial pipelines rely more on integrated reward models and RL.
- Combining teachers trained on different SFT pipelines creates distribution mismatches, requiring careful cross-teacher alignment.
Why it matters
Understanding modern post-training trends helps engineering teams navigate the complexities of RLHF, distillation, and the shifting dynamics between open-weight architectures and proprietary APIs.
Conversation 0
Be the first to share your perspective.