← back
UX Design Principles for Semi Autonomous Multi Agent Systems — Victor Dibia, Microsoft
Takeaway
For agents that act in stateful environments, design the UX around streaming plan visibility and a mix of deterministic tools and autonomous exploration.
Summary
- Dibia (MSR, AutoGen, AutoGen Studio creator) builds 'BlenderLM' from scratch — a multi-agent system that turns natural-language prompts into Blender 3D scenes over WebSocket.
- Background: LIDA (2022) was an early agentic data-visualization workflow; Davinci Codex's 20% error rate dropped to 1.5% with GPT-3.5-turbo three months later.
- Distinguishes deterministic workflows (great when you know the solution) from autonomous LLM-driven exploration needed when actions have side effects (e.g. clicking in Blender).
- Synthesizes UX design principles: transparency of planning, streaming progress, mixing fixed tools with LLM-driven steps, and observability for long-running tasks.
- Live demo creates two glossy silver spheres with a planning agent emitting steps that stream to both UI and Blender.
multi-agentuxautogen
Original description
Autonomous or semi-autonomous multi-agent systems (MAS) involve exponentially complex configurations (system config, agent configs, task management and delegation, etc.). These present unique interface design challenges for both developer tooling and end-user experiences. In this session, I'll explore UX design principles for multi-agent systems, addressing critical questions: What is the true configuration space for autonomous MAS? How can users arrive at the correct mental model of an MAS's capabilities, if at all? How can we improve trust and safety through techniques like cost-aware action delegation? What makes agent actions observable? How do we enable seamless interruptibility? Attendees will gain actionable insights to create more transparent, trustworthy, and user-centered multi-agent applications, illustrated through real-world implementations in AutoGen Studio - a low code developer tool built on AutoGen (44k stars on GitHub, MIT license) and similar tools. ---related links--- https://x.com/vykthur https://www.linkedin.com/in/dibiavictor/ https://newsletter.victordibia.com/ https://victordibia.com/