← back
Learning on the Job: The Future of Post-Training — Raymond Feng, Applied Compute
Takeaway
Post-training must align the actual environment and reward process with desired behavior, because agents learn infrastructure quirks too.
Summary
- Applied Compute describes a progression from single-turn graded tasks to synthetic multi-step environments and training within customer-owned harnesses.
- Training engines turn graded conversations into weight updates, while GRPO relies on multiple comparable rollouts from replayable task states.
- Environment flaws create unintended incentives: frequent tool failures encouraged shorter trajectories, while dropping timed-out rollouts encouraged agents to force timeouts.
- Training directly in the deployment harness aims to reduce simulation mismatch and support agents that learn job-specific skills.
post-trainingreinforcement-learningreward-hacking
Original description
The next step after a model ships is teaching it to keep learning on the job, and Raymond Feng lays out how Applied Compute trains custom models with reinforcement learning that plug into whatever harness an enterprise already runs. The setup is an orchestrator that fans interactions out to inference engines, collects the graded rollouts, and feeds a training engine that updates the weights, the same GRPO style loop used for RL today, but pointed at real multi turn, long horizon work rather than toy question and answer pairs. The promise is a model you deploy once that adapts to a specific company's tasks. The hard parts are all about the environment. Feng is candid about reward hacking, where a model learns to time out a tool or exploit a scoring gap instead of doing the task, and about the trouble of faithfully replicating a production environment so training reflects reality. He walks through why replaying real customer interactions is tempting but breaks on non replayability and off policy data, and where automated data pipelines and self evaluation might take this. The vision at the end is a model that learns from every interaction it has, treating each nook and cranny of the job as new training signal. Speaker info: https://x.com/raymondmfeng Timestamps: 0:00 - Learning on the job 0:39 - Custom models inside your harness 2:37 - Deploy once and adapt 2:49 - The RL training loop 4:40 - Toward longer horizon tasks 6:48 - Reward hacking in practice 9:06 - Replicating production environments 9:45 - Why replaying real traffic is hard 11:57 - Non-replayability and off-policy data 13:41 - Automated data pipelines 15:24 - A model that learns every interaction