← back
What Lies Beneath the API — Benjamin Cowen, Modal
Takeaway
Consider fine-tuning when product-specific evidence and economics justify specialization, using existing evaluations and interaction data as the foundation.
Summary
- As AI products specialize, frontier APIs can become limiting on domain quality, cost, latency, or throughput, motivating custom models.
- Fine-tuning is most justified when mature evaluations and collected data reveal a persistent bottleneck; an existing agent harness can support reinforcement-learning rollouts.
- Serverless compute and open-source training libraries offer algorithm control without requiring teams to operate a dedicated training cluster.
- Modal provides GPU and sandbox primitives for training and large-scale rollouts, with examples of supervised fine-tuning or reinforcement learning in roughly 300 lines of Python.
fine-tuningmodalreinforcement-learning
Original description
Intercom is beating their frontier API at one tenth the cost. Pinterest claims orders of magnitude. Ben Cowen from Modal argues this pattern is not the exception for maturing AI products. It is the destination. Frontier labs want their models to win at everything. You want to win at your specific business logic. Those are different goals. He offers three signals it is time to fine tune: paying more for the API than customers pay you, evals that have plateaued, and latency requirements no shared endpoint will meet. His practical case: if you have already built an agent harness and are collecting eval data, you have what you need to start RL training. Supervised fine tuning fits in 300 lines of Python. Modal customers have scaled to 50,000 sandboxes just for RL rollout. Speaker info: / benjamincowenmath https://github.com/BenCowen