AI Evaluations: The New Product Roadmaps

Xavi Amatriain, Expedia Group's chief AI and data officer, revealed that AI evaluations are now the new product development roadmaps. These evaluations, including red teaming, are integrated into the design process before coding begins. Amatriain, formerly of Google, believes in a compositional AI architecture, using specialized agents rather than a single large model. Expedia's governance strategy is built on three layers: high-level principles, enforcement processes, and automation, all calibrated to risk levels.

Expedia utilizes agent release toll gates, which are checkpoints that scale with an agent's risk. This ensures that security reviews and evaluations align with potential dangers, shifting from recommended to required checks as risk increases. Guardrails are considered a necessary evil, with the goal of minimizing their impact over time to avoid biasing feedback loops. The company emphasizes that AI agents should assist and recommend but never make final booking decisions, a critical step that remains with the user for security and agency.

This systemic design approach, focusing on the composition of specialized agents, enhances security by allowing isolated evaluation and lockdown of individual components. The feedback loop from operational AI systems into evaluation suites is also crucial for rapid improvement. However, a recent survey indicated that 54% of enterprises have experienced an AI agent security incident or near-miss, highlighting the ongoing challenges in AI security, especially with the rise of threats from other AI systems.