Azure OnAir Podcast - Integrate Summit: Generative AI and API Management

Guest: Andre Kamenev

• Part of the API management team at Microsoft
• Experience: 8 years at Microsoft, started as a cloud solution architect, now in engineering
• Focuses on different API types and generative AI capabilities

Key Topics Discussed

Role of API Management in Generative AI

• Intelligent Applications: Use large language models like Azure OpenAI
• Tokens: Main currency used in LLMs; represent text chunks
• Payment Models: Pay-as-you-go vs. reserved capacity (Provisioned Throughput Units)
• Scaling Challenges:
  • Tracking token usage across multiple applications
  • Rate limiting to manage token consumption and improve user experience
  • Load balancing across multiple Azure OpenAI endpoints

GenAI Gateway Capabilities

• Token Limit Policy: Specify token limits at subscription level
• Metrics: Cross charges and usage tracking via application insights
• Load Balancer & Circuit-Breaker Capabilities: For managing multiple endpoints

Security and Future Roadmap

• Security Challenges: Prompt hacking, jailbreaking, content safety (PII detection, harmful content detection)
• Integration with Other LLMs: Beyond Azure OpenAI, support for more models

AI within API Management

• Copilot for Azure: Integrated with Microsoft Copilot for Azure
• Skills Introduced:
  • Generate Policy Skill: Helps generate policy configurations
  • Explain Policy Skill: Explains existing policy configurations
• Future Features: Potential to expand scope and functionality

Customer Adoption and Scenarios

• Adoption Rate: Hundreds of customers already using AI with Azure OpenAI
• Customer Scenarios:
  • Chat assistants for internal/external users
  • API management exposes AI capabilities for application developers
• Semantic Caching Policy: Saves tokens and reduces latency

Future Opportunities and Feedback

• What-If Scenarios: Testing and simulating potential changes
• Unified Model Interface: Direction towards a common API for all models

Conclusion

• Exciting growth and future developments in API management and generative AI
• Potential for further discussions and developments in the field