Creating and scaling your own custom copilots with Azure AI Studio: Hanchi Wang

303 views · Feb 06, 2025 · 24:20 min · Watch on YouTube ↗

Takeaway

Azure AI Studio + Prompt Flow gives enterprises framework-agnostic tracing (OpenTelemetry), evals, and monitoring to ship and maintain LLM copilots end-to-end.

Summary

Azure AI Studio (20k+ monthly enterprise customers as of GA) hosts models, AOAI, search, ML, and end-to-end LLMOps; Prompt Flow ships SDK/VS Code/CLI tools.
Demo chatbot uses Assistants API with two tools: text-to-SQL retrieval and code-interpreter charting; live demo also surfaced an incorrect chart, motivating evals.
Prompt Flow's @trace decorator captures function I/O and LLM interactions as OpenTelemetry traces, viewable in a local UI or synced to Azure AI Studio for sharing.
Three pillars: tracing/instrumentation, evaluation for quality and safety, production monitoring.
Open-source Prompt Flow runs fully local without an Azure account but unlocks team collaboration when paired with the cloud studio.

azureprompt-flowllmops

Original description

Custom copilots are evolving from simple LLM+data retrieval augmented generation (RAG) solutions into more complex solutions with multiple interoperating APIs, models, metaprompts and grounding data. In this session, we’ll talk about these emerging patterns and show how Azure AI Studio and Promptflow empowers developers to build multi-agent copilot solutions. Discover advanced features such as debugging tools and security measures for developing intelligent, scalable, and safe AI solutions

Recorded live in San Francisco at the AI Engineer World's Fair. See the full schedule of talks at https://www.ai.engineer/worldsfair/2024/schedule & join us at the AI Engineer World's Fair in 2025! Get your tickets today at https://ai.engineer/2025

About Hanchi
Hanchi is a seasoned engineer and engineering manager with extensive experience in AI, machine learning, and large-scale data processing. He is currently focused on developing innovative AI tools that empower engineers to efficiently build, evaluate, deploy, and monitor their AI applications.