Companies deploying customer-facing AI in chatbots, virtual assistants and other service tools face risks from inaccurate answers, biased recommendations and mishandled data, technology leaders told CIO. Errors can damage customer trust and brands, require staff time and money to correct, expose confidential information and create legal or regulatory risks. BlackLine CIO Sumit Johar warned that bad AI advice in finance can undermine balance-sheet accuracy and bring severe legal and personal liabilities. At Citizen Health, whose users include rare-disease patients and caregivers, AI-provided information is traced to patient records, clinical information or medical studies, and patients are advised to consult healthcare providers before medical decisions.

Citizen Health has delayed some AI launches while checking how systems produce answers and whether those answers are accurate. It tests tools with questions whose answers are known, using automated tests and human review, monitors unusual responses, limits what the AI can answer and lets patients submit feedback. The company may first test tools with smaller patient groups and use their feedback before a wider release. CTO Daniel Wang said a wrong answer can sound caring, informative and authoritative despite inaccurate underlying information.

Hi Marley CTO and CAIO Jonathan Tushman said the insurance technology company held back technology that was ready to ship for years to allow more testing, observability, legal review and training. He said the trust and governance bar delayed launch by a year and adoption, a choice the company calls working at the speed of trust. After launch, companies should monitor individual conversations and patterns across tens of thousands of interactions, retain kill switches and rollback options, and prepare customer service, IT, DevOps and CloudOps teams to respond to problems. Corteva Agriscience security analyst Udaya Bhaskar cautioned against relying on impressive demonstrations or vendor claims without testing with actual customers, and against launching without an easy route to a human.

AI efficiency gains can be offset by the “hallucination tax”: staff time spent checking and correcting AI responses. Launch Consulting technologist Jess Lampe said budgets may omit ongoing operating and monitoring costs, employee rework, and legal and regulatory risks. Unframe field CTO Chris Slovak said poor customer experiences can harm a company’s brand and trust, beyond token costs. Accenture’s Kevan Yalowitz said organizations should acknowledge mistakes, explain them transparently and describe how they will prevent recurrence.

BlackLine’s Johar said critical decisions need human oversight, especially in regulated industries such as finance, and companies need reliable data and the ability to audit failures. Lampe said oversight should match an AI tool’s authority: summarizing information carries a different risk from acting on a customer’s money or account. Companies should keep testing after launch because AI behavior can change over time. Smartsheet customer officer Toyan Espeut said customer data must not be mixed across tenants or used to train third-party models; AI must follow existing permissions, make every action auditable, disclose when AI is involved and let customers review or validate outputs.

Citizen Health maintains support and on-call procedures, escalates serious concerns to leadership and gives users direct channels to report problems. Tushman said a help desk may receive the first complaint, but accountability for customer-facing AI rests with him as CTO and cannot be outsourced to a system or vendor. Bhaskar likewise said the deploying company remains responsible, while Lampe called for a named person, an escalation process and ways for customers to challenge results and reach a human.

Technology leaders said adoption speed alone does not measure success. Espeut said companies should track whether customers solve problems faster, make better decisions and achieve better business results, as well as whether support tickets decline, engagement expands beyond one buying center, and customer feedback and feature requests show active participation.

Companies deploying customer-facing AI in chatbots, virtual assistants and other service tools face risks from inaccurate answers, biased recommendations and mishandled data, technology leaders told CIO. Errors can damage customer trust and brands, require staff time and money to correct, expose confidential information and create legal or regulatory risks. BlackLine CIO Sumit Johar warned that bad AI advice in finance can undermine balance-sheet accuracy and bring severe legal and personal liabilities. At Citizen Health, whose users include rare-disease patients and caregivers, AI-provided information is traced to patient records, clinical information or medical studies, and patients are advised to consult healthcare providers before medical decisions.

Citizen Health has delayed some AI launches while checking how systems produce answers and whether those answers are accurate. It tests tools with questions whose answers are known, using automated tests and human review, monitors unusual responses, limits what the AI can answer and lets patients submit feedback. The company may first test tools with smaller patient groups and use their feedback before a wider release. CTO Daniel Wang said a wrong answer can sound caring, informative and authoritative despite inaccurate underlying information.

Hi Marley CTO and CAIO Jonathan Tushman said the insurance technology company held back technology that was ready to ship for years to allow more testing, observability, legal review and training. He said the trust and governance bar delayed launch by a year and adoption, a choice the company calls working at the speed of trust. After launch, companies should monitor individual conversations and patterns across tens of thousands of interactions, retain kill switches and rollback options, and prepare customer service, IT, DevOps and CloudOps teams to respond to problems. Corteva Agriscience security analyst Udaya Bhaskar cautioned against relying on impressive demonstrations or vendor claims without testing with actual customers, and against launching without an easy route to a human.

AI efficiency gains can be offset by the “hallucination tax”: staff time spent checking and correcting AI responses. Launch Consulting technologist Jess Lampe said budgets may omit ongoing operating and monitoring costs, employee rework, and legal and regulatory risks. Unframe field CTO Chris Slovak said poor customer experiences can harm a company’s brand and trust, beyond token costs. Accenture’s Kevan Yalowitz said organizations should acknowledge mistakes, explain them transparently and describe how they will prevent recurrence.

BlackLine’s Johar said critical decisions need human oversight, especially in regulated industries such as finance, and companies need reliable data and the ability to audit failures. Lampe said oversight should match an AI tool’s authority: summarizing information carries a different risk from acting on a customer’s money or account. Companies should keep testing after launch because AI behavior can change over time. Smartsheet customer officer Toyan Espeut said customer data must not be mixed across tenants or used to train third-party models; AI must follow existing permissions, make every action auditable, disclose when AI is involved and let customers review or validate outputs.

Citizen Health maintains support and on-call procedures, escalates serious concerns to leadership and gives users direct channels to report problems. Tushman said a help desk may receive the first complaint, but accountability for customer-facing AI rests with him as CTO and cannot be outsourced to a system or vendor. Bhaskar likewise said the deploying company remains responsible, while Lampe called for a named person, an escalation process and ways for customers to challenge results and reach a human.

Technology leaders said adoption speed alone does not measure success. Espeut said companies should track whether customers solve problems faster, make better decisions and achieve better business results, as well as whether support tickets decline, engagement expands beyond one buying center, and customer feedback and feature requests show active participation.