← back
Shipping Products When You Don't Know What they Can Do — Ben Stein, Teammates
Takeaway
PM for autonomous-agent products requires shipping despite genuine uncertainty about what your own product can do — a new discipline is emerging.
Summary
- Ben Stein, co-founder of Teammates (platform for designing a digital AI workforce), shows how PM discipline breaks when you don't know what your own product can do.
- Concrete trigger: a customer asked whether they could @-tag their AI 'teammate' Stacy in a Google Doc comment, and Stein had no idea what the agent would do — no webhook, but maybe via Gmail notification.
- Argues product management itself is undergoing a paradigm shift toward agentic products with autonomous behavior; engineering's IDE-tools transformation is more visible but PM's is deeper.
- Teammates ship as named identities (avatars, Slack accounts, Gmail accounts, Gen Alpha slang) blurring the line between coworker and software.
product-managementagentsautonomous-agents
Original description
A customer recently asked me: “Hey, can I tag your AI agent in a Google Doc comment?” The honest answer: I have no idea! We never designed our agents to handle Google Doc comments, but we tried it anyway… and it worked! The agent performed beautifully, the customer was thrilled, and I was left bewildered. Welcome to Product Management for AI agents, where roadmaps are fuzzy and we only learn the boundaries of our products after they’re released. When a product doesn’t follow predefined requirements but instead learns and improvises at runtime, PMs must give up control and lean into uncertainty, curiosity, experimentation, and fast feedback loops. This talk is a field guide for Product/Engineering teams navigating this new reality. We’ll cover how to write specs for affordances instead of features, how to use AI evals as a product development tool, and how to perform User Acceptance Testing on undocumented emergent behavior. Most importantly, we’ll explore how to build trust with customers even when the answer is, truthfully, “I don’t know.” If you’re managing AI-native products in 2025 the same way you managed web apps in 2020, you might find yourself A/B testing an agent that decided to go off and do C, D, and E all by themselves! About Ben Stein Ben is a customer-obsessed technology executive and product leader who seamlessly bridges the worlds of business, product, and technology. He has repeated success leading cross-functional teams at multiple lifecycle stages, from 3x startup founder, to scaling through hypergrowth, to managing mature lines of business. In 7 years at Twilio, Ben was GM of multiple business units (Developer Experience, Enterprise), Product Director for text messaging, and Head of R&D for Twilio.org. As CPTO at Arcadia (climate tech unicorn), he led a global team building APIs to decentralize and decarbonize the electrical grid. He cofounded multiple startups including Mobile Commons (acquired by $UPLD), an early platform for SMS marketing; and QuitCarbon, an AI platform to transition 100M homes off fossil fuels. He is currently building Teammates, a platform for designing and managing a virtual workforce of truly autonomous virtual colleagues. Recorded at the AI Engineer World's Fair in San Francisco. Stay up to date on our upcoming events and content by joining our newsletter here: https://www.ai.engineer/newsletter