The question under the question
The question everyone is asking right now is whether AI can do security assessments.
It can. It already does the enrichment, the correlation, the summarization, the triage, and increasingly the narrative that ties findings together. That part is settled, and arguing about it is a waste of a security leader's time.
The question that actually matters is underneath it:
Which security decisions are we willing to delegate to AI, and what level of human verification does each one require?
That isn't a technology question. It's a governance question, and most organizations are answering it by accident.
This piece extends the argument in Governance Is a Decision System Not a Document. If governance is a decision system, then AI-assisted assessment is where that system either holds or quietly fails, because the tooling now makes consequential judgments faster than the governance around it can track.
"Human in the loop" is not a control
The phrase has become the default answer to every AI risk question in security, and it means almost nothing.
"Human in the loop" describes a human clicking approve.
It also describes a qualified security professional independently reviewing the evidence, challenging the reasoning, and signing their name to the conclusion.
Those are not the same control. They are not remotely the same control. One is a speed bump. The other is verification. Organizations that write "human in the loop" into policy without defining which one they mean have written a sentence, not a safeguard.
The useful move is to stop asking whether a human is involved and start asking what kind of human judgment this specific decision warrants.
The judgment nobody verifies: triage
There's a failure mode here that gets missed.
The original instinct is right. Verify the critical findings. But who decides what's critical?
If the AI performs the assessment, it has already made several consequential judgments before a human sees anything:
- which 17 findings matter
- which 2,400 observations are noise
- which attack path is likely
- whether a control is "sufficiently implemented"
- whether a risk is low
Every one of those is a judgment. And the most consequential one is the one nobody reviews: what got filtered out.
If the model decides relevance, then the relevance decision is itself a finding. Verifying only what the model surfaced means verifying the story the model told you, not whether it missed the story. That's survivorship bias with an audit trail.
The practical requirement is to sample the discarded bucket. Not everything, but systematically, and with the sampling method documented. You are testing the filter, not just the output.
Two risks that get conflated
Security teams tend to collapse two different questions into one.
1. How bad is the thing? The severity of the underlying security risk. This is what most assessment frameworks measure. 2. What happens if we're wrong about it? The consequence of the conclusion being incorrect.
These are orthogonal, and the second one usually gets ignored.
A model can be 99% accurate across 10,000 endpoint events and still be unusable for a specific decision, if the 1% it misses is the artifact that changes a board-level risk determination. Accuracy at scale is not the same as adequacy for a decision.
This is why "the AI is pretty good" is not a governance position. Adequacy is decision-relative. The same model output can be entirely sufficient for one determination and grossly insufficient for the next one, and the difference has nothing to do with the model.
The principle
AI should be allowed to scale analysis faster than humans. The level of human judgment should scale with the consequence of being wrong.
Or stated as an operating rule:
The consequence of error governs the verification threshold.
- Low consequence of error: high tolerance for AI autonomy, sampling and exception review.
- Material security finding: a human validates the evidence and the conclusion.
- High-impact business decision: a human validates the evidence, the reasoning, and the recommendation.
- Legal, regulatory, employment, or criminal consequence: AI assists only. A human validates the complete evidentiary chain, item by item.
Notice what this replaces. "Human in the loop" becomes a defined control, calibrated to the decision rather than asserted as a value.
And notice what it doesn't do. It doesn't ask the AI to be trustworthy in the abstract. It asks what happens if this specific conclusion is wrong, and sets the oversight accordingly.
The chain of custody
For any of this to be auditable, you need to be able to distinguish the layers of an AI-assisted conclusion.
Fig. 02: The Antares AI verification model — chain of custody
- Observed. Raw evidence from the environment.
- Derived. Information calculated or correlated from that evidence.
- AI inference. What the model inferred or concluded.
- Human validation. What an analyst independently confirmed.
- Decision. What the organization ultimately decided to do.
Reading this figure: each stage is a distinct accountability layer, not a workflow step. The purpose is to answer, six months later, the question every regulator and every board eventually asks. Why did we classify this risk as low? If the honest answer is "the AI said so," you do not have an assessment process. You have outsourced judgment.
The chain is also what makes the verification threshold enforceable. You can't verify at the right level if you can't tell which layer produced the claim you're reading.
Escalation authority
Thresholds have to move. A low-severity EDR alert becomes a major investigation. A routine vulnerability assessment surfaces regulated data. A compliance review uncovers a control failure that changes a certification posture.
The process needs an explicit mechanism for saying: stop, this is no longer a low-consequence AI-assisted decision.
That requires two things most programs haven't defined.
A trigger. What observation forces reassessment of the tier? Discovery of regulated data, indications of insider activity, legal hold, executive involvement, anything with employment or liberty implications.
An owner. Who has the authority to escalate mid-investigation, and are they empowered to halt the low-tier process while they do it?
Without an owner, the low-consequence process keeps running on inertia after the case has stopped being low-consequence. That isn't a hypothetical failure mode. It's the default one.
The provenance gap
Here's the uncomfortable part.
Most EDR, XDR, and SIEM platforms bolting on GenAI right now cannot cleanly distinguish raw telemetry from AI enrichment from analyst annotation. The provenance chain above is the right model. In many current stacks, the data to populate it doesn't exist.
Say that plainly, because it changes what the framework is for.
If your tooling can't separate AI inference from observed evidence, then the verification threshold can't be enforced by the tool. It has to be enforced by process, which means it has to be enforced by people, which means it will drift the first busy week.
The honest position is that the chain is the target state, and most organizations aren't there. Closing that gap is a vendor question and an engagement deliverable, not a policy statement.
What to ask vendors
If provenance and verification thresholds matter, and for regulated clients they will, these belong in vendor selection conversations rather than in an internal SOP nobody reads.
- Can you tag and display provenance: observed, derived, AI-generated, human-validated?
- Is the audit trail exportable, and in what format?
- Are model versions recorded, and are changes to the model logged?
- Can a human validation field be attached to a specific finding, and does it persist?
- What happens to prior conclusions when the model is updated?
- Can the platform enforce different verification requirements by case tier?
- Is AI enrichment separable from raw telemetry in the UI and in the export?
If the answer to most of these is no, you aren't buying an AI-assisted assessment capability. You're buying an AI-assisted narrative generator with no chain of custody, and you should price that accordingly.
How you know it's working
Thresholds that aren't measured are slogans. A minimal set:
- False-negative sampling rate on the discarded bucket
- Override rate, meaning how often human review changes the AI's conclusion, by tier
- Escalation lag, meaning time from when a case actually became high-consequence to when it was treated that way
- Post-decision review findings, meaning cases where the verification level proved inadequate in hindsight
The override rate is the most honest signal. If humans never change the AI's conclusion, either the model is extraordinary or the verification is theater. In most programs, it's the second one.
What this is really about
AI changes the economics of analysis. It does not change accountability for the decision.
The consequences of a security assessment land on people. An employee under investigation. A board making a risk determination. An organization facing a regulatory question. Those consequences don't get cheaper because the analysis was fast.
Which means the level of human judgment has to scale with the consequence of being wrong. Not the confidence of the model. Not the polish of the output. The consequence.
Polished output is the trap. A model that writes a clean, confident narrative is more dangerous at the material-finding tier than a sloppy one, because sloppy output gets scrutinized and confident output doesn't.
So the operating rule is simple, and it's the whole insight:
Scale the analysis. Scale the judgment. The consequence of error governs the verification threshold.
This Insight extends the pillar argument in Governance Is a Decision System Not a Document. It's part of the Antares AI Governance series, alongside AI agent authorization and our ISO 42001 work.
