Internal AI Search: What If It Answers with an Outdated Policy?
What if an employee asks AI about travel expenses and gets a policy your company has already retired? Faster document search can create more incorrect claims and follow-up work. The quality of internal AI search depends on which approved document it uses and who can see it, not just how fluent the answer sounds. Before connecting every file, test sources, versions and permissions with 10 questions employees often ask.
Start with common employee questions
Collect just 10 questions that your administration, HR or sales team answers repeatedly. Find the latest approved document for each answer. Start with one or two documents that contain no sensitive information, in an AI environment your company permits.
Examples include how to submit travel expenses, where to send receipts, rules for using the company logo in external materials, and the scope of services staff can describe to customers. Keep salary, performance reviews and individual contracts separate from general guidance. You do not need to begin by uploading all company files to an external chatbot.
Try this prompt:
Answer the question using only the approved documents provided. Include the document title, relevant section and supporting passage. If the documents do not establish an answer, mark it “Check with the responsible team.” If the rules conflict, point out the conflict rather than combining them.
This is a starting point for deciding what to review. It does not configure document connections or access permissions. Open the sources AI cites and check them yourself. For each question, record the expected answer, reference document, effective date and person responsible for checking it. That list gives you a basis for comparing results when you add a new search tool.
Check sources and approved versions
Even a natural-sounding answer is hard to use at work if it relies on the wrong document. Run your 10 questions and check three things separately:
- Source: Does the cited section actually support the answer?
- Version: Is this the currently approved policy, with the right effective date and scope?
- Coverage: Has the answer missed an exception or mixed in another department’s rules?
For a hypothetical example, imagine files named “Travel Meal Guidelines” and “Travel Meal Guidelines_Final.” The names alone cannot establish which one is authoritative. The most recently modified file might only contain a typo fix. The latest modification date and the latest approved version are different things. Record the effective date, approval status, responsible department and superseded version in the reference document.
When you connect a SharePoint site or folder URL in Microsoft Copilot Studio, it searches the registered scope and its subfolders. It does not follow links inside documents. An appendix mentioned as “See appendix” must also be registered as a separate knowledge source to be used in answers.
Before changing models to fix a wrong answer, identify the cause. Is the latest document missing? Was an old version retrieved alongside it? Or was the correct document misinterpreted? Each calls for a different fix. Save the exact question and result so you can repeat the test and check whether your changes helped.
Test employee access
Easier search should preserve differences in what employees are allowed to see. Copilot Studio handles permissions differently for direct file uploads and SharePoint connections.
- Direct upload from a device: People with access to the agent can access the uploaded file’s content. Do not assume the original repository’s employee-level permissions carry over.
- Connected SharePoint files and folders: The user’s access is checked before an answer is provided. If the original file is already shared too widely, AI will not narrow that access for you.
Check how the material was added before testing.
Do not stop after confirming that an administrator can get an answer. Run the same questions with separate test accounts for a general employee, the responsible team and a user who should not have access. Ask IT to provide approved test accounts instead of sharing real employees’ passwords. Compare the following:
- Does permitted guidance appear for employees who need it?
- Is restricted content kept out of answers?
- Do source links enforce the intended permissions when opened?
- When asked about inaccessible material, does the agent provide a route to the right person instead of guessing?
A missing answer is not always a permissions issue. With the “Upload files > SharePoint” route, documents protected by sensitivity labels or passwords cannot be processed for search. They may show as “Ready” yet provide no answers. Ask the responsible team whether separate guidance can be prepared rather than removing protection.
Keep outdated documents out of answers
Removing an old policy and excluding it from search are different decisions. If a file must be retained for contracts, audits or work history, consider a separate location or exclusion from current-policy search while following retention rules. Permanent deletion should not be the first step in improving AI search.
Distinguish “Currently approved,” “Not yet effective” and “Reference/archive” documents. If copies exist in multiple folders, choose one authoritative location. When different people own document changes and search connections, assign someone to update the connection after approval.
Changing the original does not update every copy AI uses. Files uploaded directly to Copilot Studio must be uploaded again after revision. Connected SharePoint files and folders synchronize changes automatically; check that answers actually reflect the new policy afterward. If you rename a site or folder, check the source link and access permissions too.
After the cleanup, run the original 10 questions again. Record whether the answers, sources and permissions are all correct, and repeat the checks when documents change. Record unanswered questions separately. When no supporting policy exists, directing someone to the responsible team may be more useful than inventing a plausible rule.
Can you explain why an employee should trust this AI answer, using its approved source and access permissions?
Starting with one department’s recurring questions is enough. Establishing the authoritative sources for frequently used answers helps you improve document management while introducing AI search.
If you are considering how to connect company documents to AI and review the results, start by checking one set of recurring questions with AIB. AIB operates an independent AI comparison platform that is not tied to a particular AI company. We help business leaders and practitioners choose AI that fits their work and budget, and put it to use.
Contact AIB about adopting and using AI
Sources and further reading
- Connections, permissions and updates: Add unstructured data as a knowledge source · Add SharePoint as a knowledge source