← back
Building AI Agents that actually automate Knowledge Work - Jerry Liu, LlamaIndex
Takeaway
Real knowledge-work automation needs a document toolbox (parse, extract, index, manipulate) — not just RAG retrieval.
Summary
- Jerry Liu (LlamaIndex CEO) splits enterprise knowledge-work agents into 'assistive' (chat) and 'automation' (background) types, both built atop a document toolbox over unstructured data (~90% of enterprise data).
- Pitches the 'document MCP server' as a generalization of RAG with semantic search, file lookup, manipulation, and structured querying — not just one-shot vector retrieval.
- LlamaIndex's parsing service interleaves LLMs/LVMs (Sonnet 3.5/4, Gemini 2.5 Pro, GPT-4.1) with traditional parsers plus agentic validation, outperforming open and proprietary benchmarks.
- Announces a new Excel agent that normalizes irregular spreadsheets (gaps in rows/columns) into 2D tables and supports agentic QA — released in early preview.
- Argues neither RAG nor text-to-CSV works on real Excel; deep semantic structure understanding is required.
agentsdocument-parsingllamaindex
Original description
Agents are all the rage in 2025, and every single b2b SaaS startup/incumbent promises AI agents that can "automate work" in some way. But how do you actually build this? The answer is two fold: 1. really really good tools 2. carefully tailored agent reasoning over these tools that range from assistant-to-automation based UXs. The main goal of this talk is to a practical overview of agent architectures that can automate real-world work, with a focus on document-centric tasks. Learn the core building blocks of best-in-class "tools" around processing, manipulating, and indexing/retrieving PDFs to Excel spreadsheets. Also learn the range of agent architectures suited for different tasks, from chat assistant-based UXs with high human-in-the-loop, to automation UXs that rely on encoding a business process into an end-to-end task solver. These architectures have to be generalizable but also highly accurate as agents get increasingly better at reasoning and code-writing. About Jerry Liu Jerry is the co-founder/CEO of LlamaIndex, the most accurate and flexible way to automate your document workflows with AI agents. Before this, he led the ML monitoring team at Robust Intelligence, did self-driving AI research at Uber ATG and worked on recommendation systems at Quora. Recorded at the AI Engineer World's Fair in San Francisco. Stay up to date on our upcoming events and content by joining our newsletter here: https://www.ai.engineer/newsletter