Salesforce Shares Four Lessons for AI Agents

Salesforce says its Help Agent uses Data 360 to organize product knowledge and Agentforce to answer customer questions from that material. The process, called grounding, connects an agent to trusted content. When a customer asks a question, the agent retrieves relevant passages and uses retrieval-augmented generation (RAG) to form an answer, so answer quality depends on the content retrieved. Salesforce shares four lessons from its Customer Zero work, in which it uses its own products first.

Before indexing content, Salesforce assesses whether each article answers a real customer question, fills an unmet need, or conflicts with another article. It scores articles using four signals: Article Score, Self-Help Failure Rate, Problem Resolution Rate, and Sessions. It compares those measures with the most common customer case drivers to find gaps, prioritizing high-traffic, low-performing material. Salesforce says adding more content can slow and confuse an agent. Given a tight timeline, it created a separate agent for the Informatica help portal rather than adding acquired content to Help Agent; that agent now resolves 92% of site questions without a case being filed. Content teams handle writing, while AI and Data teams prepare content for agents through metadata, chunking (splitting articles into search-sized pieces), retrieval tuning, and evaluation. Agent Managers connect the teams, analyzing production conversations and working with authors to address gaps.

Salesforce says agent development should proceed by hypothesis because results depend on choices such as included content, embedding models, and search indexes. Teams state an expected user-experience effect, build a Data 360 configuration called a “recipe,” test it quantitatively and qualitatively, then reject or deploy it, often through an A/B experiment, with continued monitoring. One test found that adding product names as metadata alone left before-and-after metrics flat, so Salesforce did not ship it. Early qualitative tests look better when metadata is paired with prompt instructions; Salesforce is building and plans to evaluate that combined recipe before deciding.

For quantitative evaluation, Salesforce uses a golden data set of about 200 user utterances, each paired with a human-validated expected answer and links to supporting articles. AI can draft these examples, but a person checks each answer against its sources. For qualitative evaluation, subject-matter experts review AI-generated acceptance criteria for correctness, completeness, and relevance; previously, they spent hundreds of hours writing criteria by hand for every utterance.

Salesforce tests retrieval deterministically, checks semantic quality, and measures business outcomes. Humans and LLM judges also assess answer quality, service quality such as tone and disambiguation, and AI quality, including retrieval metrics and latency. In a chunking comparison, Salesforce tested fixed token-size chunks against Data 360’s section-aware approach on 195 query-document pairs. Most metrics improved, while a few fell slightly without statistical significance. Automated metrics flagged 8 of 195 queries, about 4%, as failures; reviewers judged most to be false failures because the retriever found valid information in a different document. Salesforce put the real failure rate closer to 1%, and says that human-reviewed result cleared the approach for production.

Salesforce recommends assessing Fit, Gap, and Clash across the top 100 knowledge-base articles before changing a model if an agent is not live. For agents already in production, it recommends identifying the most frequent error—missing documents, wrong tone, or correct documents paired with wrong answers—and matching it to the relevant Data 360 lever. The company advises starting small and iterating.

Salesforce says its Help Agent uses Data 360 to organize product knowledge and Agentforce to answer customer questions from that material. The process, called grounding, connects an agent to trusted content. When a customer asks a question, the agent retrieves relevant passages and uses retrieval-augmented generation (RAG) to form an answer, so answer quality depends on the content retrieved. Salesforce shares four lessons from its Customer Zero work, in which it uses its own products first.

Before indexing content, Salesforce assesses whether each article answers a real customer question, fills an unmet need, or conflicts with another article. It scores articles using four signals: Article Score, Self-Help Failure Rate, Problem Resolution Rate, and Sessions. It compares those measures with the most common customer case drivers to find gaps, prioritizing high-traffic, low-performing material. Salesforce says adding more content can slow and confuse an agent. Given a tight timeline, it created a separate agent for the Informatica help portal rather than adding acquired content to Help Agent; that agent now resolves 92% of site questions without a case being filed. Content teams handle writing, while AI and Data teams prepare content for agents through metadata, chunking (splitting articles into search-sized pieces), retrieval tuning, and evaluation. Agent Managers connect the teams, analyzing production conversations and working with authors to address gaps.

Salesforce says agent development should proceed by hypothesis because results depend on choices such as included content, embedding models, and search indexes. Teams state an expected user-experience effect, build a Data 360 configuration called a “recipe,” test it quantitatively and qualitatively, then reject or deploy it, often through an A/B experiment, with continued monitoring. One test found that adding product names as metadata alone left before-and-after metrics flat, so Salesforce did not ship it. Early qualitative tests look better when metadata is paired with prompt instructions; Salesforce is building and plans to evaluate that combined recipe before deciding.

For quantitative evaluation, Salesforce uses a golden data set of about 200 user utterances, each paired with a human-validated expected answer and links to supporting articles. AI can draft these examples, but a person checks each answer against its sources. For qualitative evaluation, subject-matter experts review AI-generated acceptance criteria for correctness, completeness, and relevance; previously, they spent hundreds of hours writing criteria by hand for every utterance.

Salesforce tests retrieval deterministically, checks semantic quality, and measures business outcomes. Humans and LLM judges also assess answer quality, service quality such as tone and disambiguation, and AI quality, including retrieval metrics and latency. In a chunking comparison, Salesforce tested fixed token-size chunks against Data 360’s section-aware approach on 195 query-document pairs. Most metrics improved, while a few fell slightly without statistical significance. Automated metrics flagged 8 of 195 queries, about 4%, as failures; reviewers judged most to be false failures because the retriever found valid information in a different document. Salesforce put the real failure rate closer to 1%, and says that human-reviewed result cleared the approach for production.

Salesforce recommends assessing Fit, Gap, and Clash across the top 100 knowledge-base articles before changing a model if an agent is not live. For agents already in production, it recommends identifying the most frequent error—missing documents, wrong tone, or correct documents paired with wrong answers—and matching it to the relevant Data 360 lever. The company advises starting small and iterating.