BlackLine CIO Sumit Johar said customer service AI can undermine trust and expose companies to brand damage and regulatory risk when it gives inaccurate answers, makes biased recommendations or mishandles data. Generative AI can produce different answers to the same question, and a confident, persuasive response can still be wrong. Mistakes matter more when customers rely on AI for financial, insurance or health information. Johar said incorrect financial guidance can undermine the accuracy of a balance sheet and create legal and personal liability. Companies also need to identify the data sources behind answers to prevent confidential information from being exposed.

Citizen Health, which serves people with rare diseases and their caregivers, traces important AI-provided information to its original sources, including patient records, clinical information and medical research. It advises patients to discuss information with clinicians before making medical decisions. CTO Daniel Wang said incorrect answers can still sound caring, useful and authoritative. The company delayed launches when it needed more time to understand how answers were generated or verify their accuracy. It tests with small patient groups, incorporates feedback and improves the system before expanding access. Before launch, it tests questions with known answers, combines automated tests with human review, monitors unusual answers, limits the system’s scope and provides a way for patients to send feedback.

Jonathan Turschmann, CTO and CAIO of insurance technology company Hi Marley, said the company spent additional time on testing, observability (the ability to understand how a system behaves), legal review and training, even when the technology was ready for release. The company calls this “working at the speed of trust.” Turschmann said customer-facing AI operations require reviewing individual conversations and finding patterns across tens of thousands of interactions, as well as having a kill switch and a way to roll back changes. Customer service, IT, DevOps and CloudOps teams also need procedures for responding to AI problems. Udaya Bhaskar of agricultural company Corteva Agriscience identified three risks: adopting AI based on impressive demos without adequately checking real-world performance, relying too heavily on vendor claims, and launching without a way for customers to reach a person.

AI chatbots may reduce staff workload by handling more inquiries, but the savings shrink when employees must verify or correct answers or repeat work after bad results. The “hallucination tax” includes staff review time, rework, regulatory and legal risks, and the harder-to-measure loss of customer trust. Jess Lamph of Launch Consulting said operating and monitoring costs may be underrepresented in development budgets. Chris Slovak of Unframe said a poor customer experience can cost a brand more than token expenses. Kevin Yallowitz of Accenture said the hidden cost is the loss of user trust after a mistake; successful organizations acknowledge errors quickly, explain them transparently and describe how they will prevent a recurrence.

Johar said AI should not independently handle decisions where accuracy is essential in heavily regulated fields such as finance. Companies need to provide reliable data and be able to audit the system when problems arise. Lamph said oversight should match the actions an AI tool is allowed to take: summarizing information carries different risks from acting on customer funds or accounts. Performance also needs continuous testing after launch. Smartsheet’s Toyan Espute said companies should keep customer data separate across tenants, prevent its use for training third-party models and ensure AI does not exceed customers’ existing permissions. Every AI action should be auditable, and customers should know when AI is involved and be able to review and verify its results.

Citizen Health investigates customer trust issues through support and on-call procedures, reports serious cases to executives and maintains a direct channel for users to report problems. Turschmann said the CTO has ultimate responsibility for customer-facing AI, even when a customer success team or help desk receives the first complaint. Bhaskar also said companies adopting AI remain responsible for its actions. Lamph said companies should name an accountable person, establish escalation procedures and let customers challenge results and reach a human. Success measures should go beyond customer counts. Espute said companies should track how quickly customers’ problems are resolved, decision-making and business outcomes, fewer support requests, wider organizational adoption, and the quality of feedback and feature requests.

BlackLine CIO Sumit Johar said customer service AI can undermine trust and expose companies to brand damage and regulatory risk when it gives inaccurate answers, makes biased recommendations or mishandles data. Generative AI can produce different answers to the same question, and a confident, persuasive response can still be wrong. Mistakes matter more when customers rely on AI for financial, insurance or health information. Johar said incorrect financial guidance can undermine the accuracy of a balance sheet and create legal and personal liability. Companies also need to identify the data sources behind answers to prevent confidential information from being exposed.

Citizen Health, which serves people with rare diseases and their caregivers, traces important AI-provided information to its original sources, including patient records, clinical information and medical research. It advises patients to discuss information with clinicians before making medical decisions. CTO Daniel Wang said incorrect answers can still sound caring, useful and authoritative. The company delayed launches when it needed more time to understand how answers were generated or verify their accuracy. It tests with small patient groups, incorporates feedback and improves the system before expanding access. Before launch, it tests questions with known answers, combines automated tests with human review, monitors unusual answers, limits the system’s scope and provides a way for patients to send feedback.

Jonathan Turschmann, CTO and CAIO of insurance technology company Hi Marley, said the company spent additional time on testing, observability (the ability to understand how a system behaves), legal review and training, even when the technology was ready for release. The company calls this “working at the speed of trust.” Turschmann said customer-facing AI operations require reviewing individual conversations and finding patterns across tens of thousands of interactions, as well as having a kill switch and a way to roll back changes. Customer service, IT, DevOps and CloudOps teams also need procedures for responding to AI problems. Udaya Bhaskar of agricultural company Corteva Agriscience identified three risks: adopting AI based on impressive demos without adequately checking real-world performance, relying too heavily on vendor claims, and launching without a way for customers to reach a person.

AI chatbots may reduce staff workload by handling more inquiries, but the savings shrink when employees must verify or correct answers or repeat work after bad results. The “hallucination tax” includes staff review time, rework, regulatory and legal risks, and the harder-to-measure loss of customer trust. Jess Lamph of Launch Consulting said operating and monitoring costs may be underrepresented in development budgets. Chris Slovak of Unframe said a poor customer experience can cost a brand more than token expenses. Kevin Yallowitz of Accenture said the hidden cost is the loss of user trust after a mistake; successful organizations acknowledge errors quickly, explain them transparently and describe how they will prevent a recurrence.

Johar said AI should not independently handle decisions where accuracy is essential in heavily regulated fields such as finance. Companies need to provide reliable data and be able to audit the system when problems arise. Lamph said oversight should match the actions an AI tool is allowed to take: summarizing information carries different risks from acting on customer funds or accounts. Performance also needs continuous testing after launch. Smartsheet’s Toyan Espute said companies should keep customer data separate across tenants, prevent its use for training third-party models and ensure AI does not exceed customers’ existing permissions. Every AI action should be auditable, and customers should know when AI is involved and be able to review and verify its results.

Citizen Health investigates customer trust issues through support and on-call procedures, reports serious cases to executives and maintains a direct channel for users to report problems. Turschmann said the CTO has ultimate responsibility for customer-facing AI, even when a customer success team or help desk receives the first complaint. Bhaskar also said companies adopting AI remain responsible for its actions. Lamph said companies should name an accountable person, establish escalation procedures and let customers challenge results and reach a human. Success measures should go beyond customer counts. Espute said companies should track how quickly customers’ problems are resolved, decision-making and business outcomes, fewer support requests, wider organizational adoption, and the quality of feedback and feature requests.