Who's Watching Your AI After You Hit Approve?

Why the evaluation pipeline is becoming your most important privacy control

For most of its history, the privacy function has been organized around a single, comforting assumption: that a system, once reviewed, stays reviewed. We assess a processing activity at design time. We paper the relationships with data processing agreements. We complete the impact assessment, log the lawful basis, and move on. The control is the review, and the review is a moment in time.

Generative and agentic AI quietly dismantles that assumption. These systems are not static artifacts that behave on Tuesday the way they behaved on Monday. They are probabilistic. The same prompt can produce different outputs across runs because the model is sampling from a distribution of possible responses, not executing a fixed instruction. They drift, because the data feeding them changes underneath them. And they regress in ways their builders never intended. A fix to one feature silently breaks another that no one thought to re-check.

"The most consequential privacy decisions an AI system makes happen after you hit approve. In production, thousands of times a day, with no one watching."

Approval was supposed to be the safeguard. For AI, it is barely the starting line. The real question is who, if anyone, is watching once the system goes live.

The discipline engineers built, and what it has to do with you

Engineering teams that ship serious AI have converged on an answer to the measurement problem, and it has a name: evaluation, or "evals." The premise is blunt. A large majority of AI projects never make it past the prototype stage, not because the model was bad, but because there was no feedback loop. Teams wait for complaints, try to reproduce the issue, and patch blindly, accumulating complexity until they lose confidence in their own system.

The Three Tiers of AI Evaluation

TIER 1

Unit Tests: Cheap, fast, deterministic assertions run on every change to catch obvious errors and regressions immediately.

TIER 2

LLM-as-a-Judge: Human reviewers establish a gold standard; automated scoring checks outputs at scale for tone, quality, and appropriateness.

TIER 3

Live A/B Tests: Reserved for mature systems; measures real-world impact on users with statistical rigor.

Read that description again with a privacy lens: the evaluation pipeline is already a privacy control. It is simply being operated without you. Everything the privacy function cares about, data minimization, prevention of unauthorized disclosure, honoring commitments in your privacy notice, is testable. And if it is testable, it belongs in the evals.

"Isn't that just a privacy impact assessment?"

It is the natural question, and the answer is no, though the confusion is instructive. A privacy impact assessment is a governance document: a structured analysis performed at a point in time, before launch, to identify privacy risks and decide whether the processing should proceed. Its output is a judgment.

Privacy Impact Assessment

• Governance document
• Point-in-time analysis
• Output: a judgment
• Asks: Should we build this?
• Rigorous but static

Evaluation Pipeline

• Engineering measurement system
• Automated & continuous
• Output: live pass/fail scores
• Asks: Is it still working today?
• Continuous but privacy-blind

The Core Argument

The assessment is the law. The evaluation is the enforcement. Most organizations today have written the law and left it unenforced the moment the system went live.

Where your obligations become assertions

Consider the most basic tier. Among the generic checks that mature teams run on every output is a simple assertion: that internal identifiers, database keys, and personal data never appear in user-facing text. A regular expression that fails the build the moment a UUID or an unredacted record leaks into a response.

To an engineer, that is a structural integrity test. To you, it is a continuously enforced control against unauthorized disclosure, a data-leakage safeguard that runs thousands of times a day and never gets tired.

"Data minimization stops being an aspiration in a policy document and becomes a line of code that either passes or fails."

The same reframing applies across the stack. The middle evaluation tier exists precisely to assess the nuanced, contextual qualities that a regex cannot capture, tone, appropriateness, whether a response over-collects or over-shares. This is where the specification gap closes. The act of building an evaluation forces a team to convert a vague requirement like "be careful with personal data" into concrete, measurable criteria.

For those of you in regulated environments, the connection is even more direct. If your organization is a covered entity or a business associate touching protected health information, an agent that can reach that information without passing through a sandbox and a tested release gate is not an engineering shortcut. It is an unaddressed risk that will, eventually, have your name attached to it.

The two things privacy leaders should insist on

1

A seat at the table when evals are designed

Before a single test case is written, privacy requirements should be on the specification sheet. Not as a compliance afterthought, as a first-class input that shapes what the system is scored on from day one.

2

Access to the results, on an ongoing basis

An evaluation that runs but whose results the privacy team never sees is not a privacy control. If the system is making privacy-consequential decisions at scale, the people accountable for those decisions need a live view of whether the controls are holding.

These are not big asks. They do not require hiring an army of data scientists or replatforming your AI governance stack. They require showing up to the right meetings and asking the right questions, before the system ships, and continuously after it does.

The bottom line

The era of one-and-done privacy reviews for AI systems is over. Not because regulators have said so, though that is coming, but because the systems themselves have made periodic review structurally inadequate. A system that changes its behavior with every new prompt version, every data drift, every model update cannot be governed by a document filed at launch.

The engineers building these systems have already figured out the answer. Continuous measurement. Automated assertions. A feedback loop that catches regressions before they become incidents. The privacy function did not build these tools, but it has every right, and, increasingly, every obligation, to use them.

The question "who is watching your AI?" should have a crisp, confident answer, a named system, a named person, and a dashboard that tells you whether the controls are holding right now. If it does not, that is the gap to close.

Ready to take the next step?

See how PrivacyPoint monitors your AI in production.

Schedule a demo with our privacy engineers and learn how to build a continuous, testable privacy program around your AI systems.

Schedule a Meeting →