PayPal's AI Boost: Context Over New Models

PayPal has dramatically enhanced its internal AI assistant's question-answering capabilities, boosting accuracy from 43% to 95%. This remarkable improvement was achieved not by upgrading the core large language model (LLM), but by integrating proprietary business context. This external knowledge layer ensures that the AI understands PayPal's specific definitions, query patterns, and analytical procedures, which are crucial for generating accurate business answers.

The company deliberately keeps this business expertise separate from the LLM. This model-agnostic design allows PayPal to adopt improved LLMs in the future without losing its hard-earned knowledge. The key innovation is a knowledge layer containing "golden queries" and analytical playbooks, ensuring AI responses are grounded in specific business rules and investigation methodologies, rather than solely on data availability.

This strategy extends to more complex AI agents. PayPal is developing frameworks to control when these agents consult trusted sources or require human approval for actions. For instance, a seven-agent workflow aids in investigating alerts across thousands of services by synthesizing information from various sources, distinguishing signals from noise, and analyzing incidents. This careful approach to AI autonomy prioritizes accuracy and risk mitigation, with proposed decisions consistently vetted against sources of truth.

PayPal is also focused on the crucial gap between testing and production. To address AI agents passing internal tests but failing in real-world scenarios, the company employs broader offline simulations, synthetic data generation, and multi-turn testing. Ongoing monitoring and employee feedback are vital for identifying failures, ensuring the AI's performance aligns with user expectations and business needs, even when AI output distribution differs in production.