Why AI Should Not Make the Final Compliance or Audit Decision
- Dániel Váczi
- 2 days ago
- 11 min read
AI can find a cybersecurity policy, identify the section relevant to a control, connect it with a configuration screenshot and a log extract, and summarize what those pieces of evidence appear to show. That can save a significant amount of work, but it does not necessarily answer the most important question:
Does the organization actually meet the requirement?
This distinction matters as AI becomes more capable of analyzing cybersecurity compliance evidence. Finding relevant information and assessing compliance are not the same task. Evidence may be relevant without being sufficient. A documented process may exist without being followed. A technical control may appear in a screenshot without operating consistently in the environment being assessed. AI can support these decisions, including through preliminary assessments. But the final compliance or audit conclusion still requires professional judgment.
Can AI determine whether an organization is compliant?
AI can analyze available information and provide a preliminary view of whether the evidence appears to support a cybersecurity requirement. It should not be treated as the final authority on whether an organization is compliant.
The reason is not simply that AI occasionally produces incorrect answers. Compliance assessment involves questions that cannot always be resolved from the content of individual documents alone. A reviewer may need to determine whether evidence is current, whether it covers the correct scope, whether a documented process actually operates, whether the sample is representative, and whether several pieces of evidence collectively demonstrate the requirement.
The distinction becomes clearer if we separate three different tasks:
Finding evidence: What information appears relevant to this requirement?
Evaluating evidence: What does this information actually demonstrate?
Making a compliance decision: Considering the requirement, scope, methodology, and available evidence, is the requirement satisfied?
AI can provide significant support for the first two. The further the process moves toward the final decision, the more important professional validation becomes.
Finding evidence is not the same as proving compliance
Suppose a cybersecurity requirement states that user access rights must be reviewed periodically. AI searches the organization's documentation and finds an access management policy. The policy states that user permissions must be reviewed every quarter. The relationship is strong. The evidence is clearly relevant. But what has actually been proven?
At this point, we know that the organization has documented a requirement for quarterly access reviews. We do not yet know whether the reviews happened. To establish that, we might need completed review records, IAM exports, approvals, tickets showing remediation of inappropriate permissions, or other operational evidence. Even then, more questions remain. Did the reviews cover all relevant systems? Were they performed during the required period? Were privileged accounts included? Were identified issues actually corrected?
This is why evidence relevance and evidence sufficiency should not be treated as the same thing. An AI system can be correct that a document relates to a control and still be wrong if it concludes that the control is therefore operating effectively.
The false confidence problem
One of the more dangerous AI failure modes in compliance work is not an obviously incorrect answer. It is a plausible answer that sounds more certain than the available evidence justifies. This can create false confidence.
Cybersecurity documentation is particularly vulnerable to this problem because good documentation often describes exactly what a compliant process should look like. A well-written access management policy may describe approvals, periodic reviews, privileged access, account termination, and segregation of duties. If the model sees strong semantic alignment between that policy and the applicable requirements, it may produce a very positive assessment.
But the policy primarily demonstrates what the organization says should happen. Whether those controls actually operate is another question. The same problem can occur with technical evidence. A convincing screenshot or log extract can look like direct proof while leaving important contextual questions unanswered.
The risk is therefore not limited to hallucination in the traditional sense. AI may analyze authentic evidence and still assign too much evidentiary weight to it. That distinction is critical in compliance and audit work.
A policy does not prove implementation
Policies and procedures are important compliance evidence. In many cases, their existence, approval, content, ownership, and review can themselves be relevant to specific requirements. However documentation of a process should not automatically be treated as evidence that the process is consistently executed.
Consider a policy stating: “User access rights are reviewed quarterly by the responsible system owner.” The document may provide evidence that a periodic review process has been formally established. It does not by itself answer:
Was the review performed in Q1?
Was it performed in Q2?
Which users and systems were reviewed?
Who performed the review?
Were exceptions identified?
What happened to inappropriate permissions?
Was there evidence of approval?
For requirements concerned with actual operation, those questions matter. AI can help connect the policy with the operational evidence needed to answer them. It should not silently bridge the gap between documented intent and actual execution.
A screenshot does not necessarily prove an active configuration
Screenshots are common cybersecurity evidence because they can provide a quick visual record of a system configuration. They can also be misleading when viewed without context. Suppose a screenshot shows that multi-factor authentication is enabled. The screenshot may indeed be relevant evidence, but a reviewer may still need to understand which environment it comes from, when it was captured, which users or systems the setting applies to, whether exceptions exist, and whether the configuration shown was active during the relevant assessment period.
A screenshot from a test environment may look almost identical to one from production. A screenshot taken three years ago may accurately represent the configuration at that time while saying little about the current environment. AI may be able to read and interpret the visible configuration. It cannot assume that everything needed to establish the evidentiary context is contained in the image.
A log entry does not prove continuous operation
Logs present a similar problem. A log entry can provide strong evidence that an event occurred. A series of log records may demonstrate recurring activity or support the operation of a technical control, but the existence of one successful event does not necessarily demonstrate that a control operated continuously throughout the period being assessed.
Imagine a requirement related to security monitoring. Finding a log entry showing that an alert was generated demonstrates something useful. It does not automatically demonstrate that relevant systems were continuously monitored, that alerts were consistently generated, that they were reviewed within the expected time, or that incidents were escalated correctly. The model needs to avoid turning: “I found evidence of this happening” into: “This process operates as required.” The second statement requires a stronger evidentiary basis.
Evidence can be relevant but insufficient
This distinction gives us a more useful way to think about AI-assisted compliance assessment. Evidence does not need to be either “useful” or “useless.” A document can provide partial support, a screenshot can support one element of a requirement, a procedure can explain the expected process while operational records demonstrate its execution.
Several weak pieces of evidence may become meaningful when interpreted together. Conversely, one apparently strong document may leave an important part of the requirement unsupported.
AI can be valuable in identifying these relationships. Instead of forcing an immediate compliant/noncompliant classification, a system can ask more useful questions:
What part of the requirement does this evidence support?
What appears to remain unsupported?
Is there evidence of both documented intent and actual operation?
Are several independent sources supporting the same conclusion?
Does any available evidence contradict the others?
What additional evidence would help validate the assessment?
This shifts AI from an automatic decision-maker toward a tool for structured professional reasoning.
Compliance evidence also has a time dimension
Evidence is not valid indefinitely simply because its content remains relevant. An old procedure may have been replaced. A configuration screenshot may no longer represent the system. An employee list may describe a previous organizational structure. A vulnerability scan may accurately show the state of an environment at one point in time but provide little evidence about its current condition. This makes evidence validity and version history important parts of compliance assessment.
Imagine that AI finds a procedure that perfectly matches a control requirement. If the procedure was withdrawn two years ago, semantic relevance alone is not enough. Similarly, a screenshot may show exactly the configuration an auditor wants to verify, but its evidentiary value changes significantly if it was captured before a major system migration.
AI-assisted analysis should therefore consider not only what the evidence says, but also contextual information such as when it was created, which version is current, what system or entity it relates to, and which assessment period is being examined. A professional reviewer must ultimately decide how those factors affect the conclusion.
Better prompting helps, but it does not eliminate the problem
The quality of an AI response depends heavily on the question being asked. A broad prompt such as:
“Does this organization have adequate access management?”
encourages a broad conclusion based on whatever information the model happens to retrieve. A more useful question would be:
“Which available evidence supports the implementation of this access management requirement? Separate evidence of documented policy from evidence of actual operation, identify what each item demonstrates, and highlight areas where the available evidence is insufficient.”
The second approach does several things better. It defines the requirement more precisely. It asks for evidence rather than an unsupported conclusion. It distinguishes documentation from implementation. And it explicitly allows uncertainty.
For audit use, this is particularly important because the objective is not simply to generate an answer. The reasoning must remain traceable to evidence that a professional can inspect.
Prompt design therefore matters, but better prompting does not transform an LLM into an auditor. The model still operates on the information available to it. A sophisticated prompt cannot recover evidence that was never provided, verify an organizational practice that is not observable in the available data, or assume responsibility for a professional conclusion.
Different AI models may reach different conclusions
Another reason not to build the final compliance decision around AI is that different models can interpret the same evidence differently. Models vary in how they handle long contexts, uncertainty, conflicting information, technical material, structured data, and relationships between multiple evidence items.
Even the same model can produce different emphasis depending on the instructions and context it receives. This does not make AI unusable for compliance. It means the architecture should be designed around the task rather than assuming that one model's output represents an objective compliance truth.
Different models or AI techniques may be appropriate for different stages. One component may classify evidence. Another may retrieve relevant information. An LLM may interpret the retrieved material. A separate process may compare the result with a structured requirement. The output can then be presented to a professional for validation. This is more robust than designing the process around a single question:
“AI, is this compliant?”
Why human-in-the-loop matters in cybersecurity compliance
A human-in-the-loop approach keeps professional review inside the AI-assisted workflow. The human does not need to repeat everything the AI has already done. The point is to divide the work according to what each side does well.
AI is well suited to processing large volumes of information, finding semantic relationships, summarizing evidence, identifying potentially relevant material, and highlighting possible gaps. The professional brings organizational context, methodological understanding, skepticism, responsibility, and judgment about evidentiary weight. A useful workflow might therefore look like this:
Requirement → Evidence discovery → Evidence mapping → AI pre-assessment → Human validation → Compliance or audit decision
AI can make the earlier stages significantly faster. The final stages remain under professional control. This approach is not simply a precaution against imperfect technology. It reflects the nature of compliance itself.
Cybersecurity requirements often need to be interpreted in the context of the organization, its scope, systems, risks, processes, applicable methodology, and available evidence. Those relationships cannot always be reduced to text classification.
What should an AI pre-assessment actually do?
The term AI pre-assessment can easily be misunderstood. A useful pre-assessment should not simply produce a green checkmark and declare a control compliant. It should help the reviewer understand the available evidence and provide a preliminary, traceable assessment that can be validated. Depending on the context, that may include:
identifying the evidence used for the preliminary assessment,
explaining which part of the requirement each evidence item supports,
distinguishing policy evidence from operational evidence,
identifying apparent gaps or missing evidence,
highlighting uncertainty,
identifying contradictory information,
suggesting areas that require further validation.
This makes the AI output useful even when the eventual human conclusion differs from the preliminary assessment. The value lies in accelerating analysis, not in avoiding professional responsibility.
What remains a professional decision?
The exact decision process depends on the framework, regulatory environment, audit methodology, and purpose of the assessment. In general, professional judgment remains necessary when determining whether the available evidence is appropriate and sufficient for the conclusion being reached.
That can include questions about scope, validity, completeness, sampling, exceptions, implementation, consistency, and whether the evidence demonstrates operation over the required period.
For an internal compliance team, the professional may need to combine the AI analysis with knowledge of how the organization actually works. For an auditor, the standard is stricter. Evidence suggested by AI still needs to be independently evaluated according to the applicable audit methodology. Additional interviews, samples, tests, or other validation may be necessary. The system can help determine where to look. It should not decide what the auditor must conclude.
AI ≠ Auditor
The distinction can be summarized simply:
AI can analyze evidence. AI can suggest relationships. AI can support a preliminary assessment. The final audit conclusion belongs to the auditor.
For organizations, the same principle applies to compliance decisions. AI can help teams understand their documentation, organize evidence, identify potential gaps, and prepare assessments. It cannot know every relevant fact about how the organization operates.
This is why replacing the professional with AI is the wrong objective. A more useful objective is to remove unnecessary manual work around the professional. Searching hundreds of documents for relevant passages is work AI can help reduce. Connecting scattered evidence around a requirement is work AI can help reduce.
Summarizing what several pieces of evidence appear to demonstrate is work AI can help reduce. Determining whether those findings justify a compliance or audit conclusion is where professional responsibility remains essential.
How Brind approaches AI-assisted assessment
Brind separates AI support from professional decision-making. Brind AI can analyze evidence and identify potentially relevant compliance information. AI Evidence Mapping can suggest relationships between evidence and requirements, while AI Pre-assessment can provide a preliminary assessment based on the available evidence and responses.
These outputs are designed to support the professional workflow rather than replace it. The reviewer can validate the evidence, interpret the requirement in context, consider information that may not be available to the AI, and make the final compliance or audit decision.
This separation also makes AI assistance more useful. The goal is not to make an AI answer look authoritative. The goal is to help professionals reach well-supported conclusions with less time spent searching, sorting, and connecting information.
Key takeaways
AI can significantly improve cybersecurity compliance and audit workflows, but its role needs to be defined carefully.
Finding relevant evidence does not prove compliance. Policies do not automatically prove implementation. Screenshots require context. Individual log entries do not necessarily demonstrate continuous operation. Evidence can also become outdated or lose relevance as systems and processes change.
Better prompting, retrieval, and AI models can improve the quality of preliminary analysis, but they do not remove the need for professional judgment.
The more useful model is human-in-the-loop compliance.
AI handles more of the information-intensive work. Professionals validate the evidence, interpret uncertainty, apply the relevant methodology, and retain responsibility for the final decision.
The objective is not an AI auditor. It is a better-supported human one.
FAQ
Can AI determine whether a cybersecurity control is compliant?
AI can provide a preliminary assessment based on the information available to it, but the final determination may require additional context, validation, sampling, interviews, or professional interpretation. AI-generated assessments should therefore be treated as decision support rather than final compliance conclusions.
Can AI replace a cybersecurity auditor?
AI can support an auditor by finding evidence, highlighting relevant information, identifying relationships, and assisting with preliminary analysis. It cannot assume responsibility for evidence evaluation or the final audit conclusion.
Why isn't a cybersecurity policy enough to prove compliance?
A policy generally demonstrates what an organization has formally defined or requires. Requirements concerning actual implementation or recurring operation may also require evidence showing that the documented process has been carried out in practice.
What is false confidence in AI compliance assessment?
False confidence occurs when an AI-generated conclusion appears convincing or certain even though the available evidence does not adequately support it. In compliance work, this can happen when relevant evidence is incorrectly treated as sufficient evidence.
What is human-in-the-loop compliance?
Human-in-the-loop compliance uses AI to support activities such as evidence analysis, evidence mapping, and preliminary assessment while keeping validation, interpretation, and final compliance decisions with qualified professionals.
What is the difference between AI assessment and AI pre-assessment?
An AI pre-assessment is a preliminary interpretation of the available evidence intended to support further professional review. It should make its evidentiary basis and uncertainties visible rather than presenting its output as the final compliance or audit decision.
Comments