How can an audit firm validate an AI tool before production use?
An audit firm should validate an AI tool against its intended use, methodology and risk before allowing it into live engagement work. A general product demonstration is not enough.
The firm should test representative cases, including straightforward files, difficult evidence, known exceptions and situations where the correct answer is “insufficient information”. It should evaluate accuracy, completeness, consistency, explainability, security and the effectiveness of human review. The team should also understand what happens when the model, workflow or methodology changes.
A practical validation process includes:
- Define the permitted use case and expected output.
- Agree acceptance criteria and material failure modes.
- Test against known or independently reviewed cases.
- Review errors and adjust the workflow or controls.
- Pilot with increased review on live engagements.
- Monitor overrides, incidents and changes after launch.
The depth should match the risk. Drafting a meeting summary and assessing a control conclusion are not the same use case. Firms should validate the latter accordingly.
Platformed capability: Platformed can be tested against the firm's own methodology and representative prior or live cases before wider rollout. Validation remains the firm's responsibility, with increased review appropriate for new or higher-risk workflows.
References: FRC AI in Audit · IAASB technology quality management