Agencies Urged to Audit AI Data Readiness

Public agencies often check vendors, security, procurement and staff training before launching an AI tool, but fewer audit the data it will use. As agencies move from pilots to production, errors can affect residents rather than remain in a test environment. The article cites a state review of Michigan’s automated unemployment fraud system: it examined more than 20,000 determinations and found a 93 percent error rate. The system was not today’s AI, but the example illustrates how automation can scale hidden assumptions and bad data.

Government Technology contributor Sumanth Kumar Gadde identifies three assumptions agencies should test: that official data is accurate, that a newer or larger system is right when records conflict, and that the outcome a model predicts is the outcome an agency should care about. Data ownership can be unclear across departments, leaving no designated person responsible for whether information is suitable for a model.

The source audit asks where each data set came from, who owns it, when it was last validated, which source takes precedence when records conflict, and whether the data is current enough for the decision. The target audit checks whether the model’s measured outcome reflects the real goal, whether the data represents the people affected, and whether error rates differ across groups. It also checks for protected characteristics or proxies, such as ZIP code, first name, language, graduation date, work history, employment gaps and certain medical or benefits codes.

Gadde illustrates the risk with a county that uses past rental inspection records to decide which buildings inspectors visit first. Because past inspections focused on the same neighborhoods, violations elsewhere may never have entered the records. A model using those records can keep directing inspectors to familiar areas and leave other problems unseen, reinforcing the pattern in its data.

The safeguard audit asks whether an authorized reviewer checks high-stakes recommendations before they take effect, whether residents can contest decisions, and whether launch data and model performance are documented as a baseline. Agencies should also assign someone to monitor drift and give that person authority to pause the system. Gadde recommends adding a data readiness gate to every AI project plan, starting with the use case where a wrong answer could cause the most harm, naming a data owner to sign off, and recording findings. He says unanswered questions found before launch are cheaper than questions discovered later, which can cost money, public trust, benefits or livelihoods.

Public agencies often check vendors, security, procurement and staff training before launching an AI tool, but fewer audit the data it will use. As agencies move from pilots to production, errors can affect residents rather than remain in a test environment. The article cites a state review of Michigan’s automated unemployment fraud system: it examined more than 20,000 determinations and found a 93 percent error rate. The system was not today’s AI, but the example illustrates how automation can scale hidden assumptions and bad data.

Government Technology contributor Sumanth Kumar Gadde identifies three assumptions agencies should test: that official data is accurate, that a newer or larger system is right when records conflict, and that the outcome a model predicts is the outcome an agency should care about. Data ownership can be unclear across departments, leaving no designated person responsible for whether information is suitable for a model.

The source audit asks where each data set came from, who owns it, when it was last validated, which source takes precedence when records conflict, and whether the data is current enough for the decision. The target audit checks whether the model’s measured outcome reflects the real goal, whether the data represents the people affected, and whether error rates differ across groups. It also checks for protected characteristics or proxies, such as ZIP code, first name, language, graduation date, work history, employment gaps and certain medical or benefits codes.

Gadde illustrates the risk with a county that uses past rental inspection records to decide which buildings inspectors visit first. Because past inspections focused on the same neighborhoods, violations elsewhere may never have entered the records. A model using those records can keep directing inspectors to familiar areas and leave other problems unseen, reinforcing the pattern in its data.

The safeguard audit asks whether an authorized reviewer checks high-stakes recommendations before they take effect, whether residents can contest decisions, and whether launch data and model performance are documented as a baseline. Agencies should also assign someone to monitor drift and give that person authority to pause the system. Gadde recommends adding a data readiness gate to every AI project plan, starting with the use case where a wrong answer could cause the most harm, naming a data owner to sign off, and recording findings. He says unanswered questions found before launch are cheaper than questions discovered later, which can cost money, public trust, benefits or livelihoods.