Banks Use Smaller AI Models to Manage Costs

Bank of America and Capital One executives said at the Evident AI Symposium in New York that routing requests to smaller AI models can help banks control costs while using more capable models for tasks that need them. Bank of America plans to double its AI budget next year, with a focus on handling more advanced conversations through Erica, its AI-enabled financial assistant. Chief technology and information officer Hari Gopalkrishnan said the bank is also improving Erica’s institutional memory and ability to complete more complex transactions. He made the remarks at the symposium last week.

Bank of America’s Orchestra routing layer uses a small classifier model (a model that sorts requests by type) to direct each query to the most cost-effective model. Gopalkrishnan said managing token spending and routing queries to lower-cost models are effective cost-management tactics. He said the bank wants flexibility to use existing small language models and fine-tuned models, and that simple decisions do not always require a large language model.

Capital One distilled open-weight models fine-tuned with proprietary data into smaller models for Chat Concierge, a multiagent application that helps car buyers navigate the purchasing process. Chief enterprise AI officer and chief scientist Prem Natarajan said the smaller models reduced latency by five times and increased dealership customer engagement by 50%. Snowflake has also added dynamic model routing to Cortex AI Gateway, which automatically selects a model for each query.

Bank of America’s AI strategy heading into 2027 includes guardrails to keep agents within assigned runtime limits and prevent them from completing multiple tasks autonomously. Gopalkrishnan said the bank is developing instrumentation, telemetry and feedback loops. Capital One executive Prem Natarajan said the bank will continue strengthening its data capabilities to support AI. He pointed to its cloud-based technology stack, years of machine-learning use and well-managed data, and said the bank will rebuild its architecture from the bottom up when needed.

Bank of America and Capital One executives said at the Evident AI Symposium in New York that routing requests to smaller AI models can help banks control costs while using more capable models for tasks that need them. Bank of America plans to double its AI budget next year, with a focus on handling more advanced conversations through Erica, its AI-enabled financial assistant. Chief technology and information officer Hari Gopalkrishnan said the bank is also improving Erica’s institutional memory and ability to complete more complex transactions. He made the remarks at the symposium last week.

Bank of America’s Orchestra routing layer uses a small classifier model (a model that sorts requests by type) to direct each query to the most cost-effective model. Gopalkrishnan said managing token spending and routing queries to lower-cost models are effective cost-management tactics. He said the bank wants flexibility to use existing small language models and fine-tuned models, and that simple decisions do not always require a large language model.

Capital One distilled open-weight models fine-tuned with proprietary data into smaller models for Chat Concierge, a multiagent application that helps car buyers navigate the purchasing process. Chief enterprise AI officer and chief scientist Prem Natarajan said the smaller models reduced latency by five times and increased dealership customer engagement by 50%. Snowflake has also added dynamic model routing to Cortex AI Gateway, which automatically selects a model for each query.

Bank of America’s AI strategy heading into 2027 includes guardrails to keep agents within assigned runtime limits and prevent them from completing multiple tasks autonomously. Gopalkrishnan said the bank is developing instrumentation, telemetry and feedback loops. Capital One executive Prem Natarajan said the bank will continue strengthening its data capabilities to support AI. He pointed to its cloud-based technology stack, years of machine-learning use and well-managed data, and said the bank will rebuild its architecture from the bottom up when needed.