← back
Compilers in the Age of LLMs — Yusuf Olokoba, Muna
Takeaway
A Python compiler with LLM-assisted verification can turn AI inference code into portable native binaries that run anywhere, sidestepping container-based deployment.
Summary
- Muna built a Python-to-native compiler that takes plain Python inference code and emits self-contained C++/Rust binaries deployable from cloud to Apple Silicon to edge.
- Motivation: hybrid inference future requires small device-side models cooperating with big cloud models — without the Docker/Kubernetes complexity tax.
- Tried PyTorch FX tracing first; abandoned because it required fake inputs and didn't handle numpy/OpenCV/dict types; built an AST-based tracer for an internal IR.
- Type propagation: Python is dynamic, C++/Rust need static types — compiler infers types per IR node and generates concrete C++ overloads for operations like string addition.
- Initial LLM-based trace generation had ~100% accuracy but was too slow; LLMs now used for verification and test generation around the deterministic compiler.
compileredge-inferencedeployment
Original description
Python is where ideas start—but it isn't where portable, low-latency software ends. In this talk, I'll show how we use LLMs inside a constrained, verifiable compiler pipeline to turn plain Python functions into self-contained native binaries that run anywhere (cloud, desktop, mobile & web); and how our customers use this technology to run open-source AI models locally and in the cloud with the familiar OpenAI client experience. Yusuf Olokoba is the founder of Muna, specializing in code generation for AI inference workloads. He previously co-founded a real estate technology startup, backed by First Round Capital and Bessemer, which was later acquired. He holds patents in computer vision and augmented reality, powering augmented reality experiences used by millions of users. Yusuf holds a B.A. in computer science from Dartmouth College and is an alumnus of South Park Commons. --- Socials: - LinkedIn: https://www.linkedin.com/in/olokobayusuf/ - X (Twitter): https://x.com/olokobayusuf - GitHub: https://github.com/olokobayusuf - Company: Muna (https://muna.ai)