Skip to content
All articles
AI Engineering

AI Data Privacy: How to Keep Customer PII Out of LLM Context

Why model context is the leakiest place customer data can land, the scope-and-redact architecture that keeps it out, and the production guest-list tool that strips contact fields twice and demographics always.

Rushit Jivani6 min read

The short version

  • Anything that enters an LLM's context should be treated as quoted later: in the reply, the transcript, a screenshot, the next tool call. If a field must not be repeated, it must not be returned.
  • Build the controls in layers: separate PII scopes from list scopes, redact at the gateway, redact again at the edge that feeds the model, and hard-strip fields no AI caller ever needs.
  • The case study: a guest-list tool on a production events platform where contact fields require their own scope, the tool re-strips them even if the gateway already did, and age and gender never reach any model context under any key.

Every AI feature eventually wants customer data in the prompt — the guest list to summarize, the order history to reason about, the account record to personalize with. And the prompt is the single leakiest place that data can land: whatever enters a model's context can come back out in the reply, persist in conversation transcripts, appear in screenshots, be forwarded into the next tool call, or be retained by whatever logging sits around the model. A system prompt saying "do not reveal emails" changes none of that. The only reliable control is the oldest one in security: data that is not there cannot leak.

This guide covers the threat model, the layered architecture — scopes, double redaction, hard strips — and the production implementation: the guest-list tool of an events platform whose AI connector returns names without contact details, and demographics to no one.

Why model context is different

Pre-AI, an over-broad API response was bounded by the client: the web app displayed what its UI displayed and discarded the rest. A model client has no such discipline. The tool result is the prompt — every field you return is now material the model may quote, summarize, or echo into another system. Three properties make this worse than ordinary API over-exposure:

  1. Transcripts persist and travel. Chat histories get reread by models on later turns, synced across devices, and shared by users. A phone number returned once is in that stream indefinitely.
  2. The model composes across calls. Data returned by one tool can be forwarded as arguments to another — a flow no one explicitly designed and no reviewer will see in your code.
  3. Instructions are not controls. Prompts influence behavior; they do not bound it. Anyone — including a prompt injection inside user content — can ask the model to repeat what it was given.

So the design rule: decide at the API edge what an AI caller may see, and let nothing extra through "for convenience." The convenience field is the leak.

How you should implement it

Step 1: Split "seeing the list" from "seeing the contacts"

Most products have one permission for a data set: you can read guests, or you cannot. AI callers need a finer split, because the common agent tasks — counting, summarizing, matching names — need the list but not the contact fields. Make PII its own scope (e.g. guests vs guests:pii), default every AI integration to the non-PII scope, and require a deliberate, logged decision to grant more. Most agent use cases never need the upgrade, and now that fact is enforced rather than hoped.

Step 2: Redact at the gateway — by deletion, not masking

The API gateway strips fields the caller's scope does not cover: walk the response, delete the keys, recurse into nested objects and arrays so a phone number cannot survive inside a sub-object the top-level pass missed. Delete rather than mask — "email": "a***@..." still tells a model the field exists and invites it to ask for the rest; an absent key is a fact nobody can quote.

Step 3: Redact again at the AI edge — defense in depth with a named bug

"Defense in depth" is a slogan until you name the bug it catches, so: the gateway regresses and starts returning email for keys that never had the PII scope. Your web app would not notice — it does not render that field. The AI tool layer pastes it straight into a model context. That asymmetry is why the tool that feeds the model must not trust the JSON it receives: it checks the scopes on the credential it holds and re-strips contact keys itself unless contact access is explicitly allowed. Two independent layers, two different owners, one assumption no longer shared.

Step 4: Hard-strip what no AI caller ever needs

Some fields your product legitimately uses — demographics for a host dashboard, say — have no AI use case at all. Give them no scope on the AI path: strip them unconditionally, for every key, with a code comment stating the product decision. Consistency ("just return the same guest object everywhere") is the enemy here — it would copy a screen designed for a logged-in owner into a transcript designed to be reread by a model.

Step 5: Test with fixtures, and mind the inputs too

The redaction test is a fake record containing every sensitive key — nested, arrayed, duplicated — asserted absent after the strip. Never test with a production export; a test fixture with real emails is itself a leak. And the same thinking applies in reverse: when a flow genuinely needs a phone number (an action the user requests), accept it as typed input for that action, not by echoing the stored list back — and keep stored values out of logs and error messages, which are the quiet second path into AI context once a model starts reading your logs.

How I implemented it on a real project

The events platform I work on exposes a guest-list tool through its MCP connector, so a host can ask Claude or ChatGPT "who's coming Saturday?" The useful answer needs names and statuses. It does not need anyone's email — and the first dangerous thing an assistant does with an email is repeat it into a chat log. Every layer above is live on this one tool.

The scope split. list_guests requires the guests scope; without it there is no list at all. Email and phone live behind a second scope, events:guests:pii, which the standard connector grant does not include. The gateway deletes contact fields from responses for any key without it — the same redaction every API caller gets.

The second strip. The MCP tool then re-strips: it walks the payload recursively — nested objects, arrays — and deletes contact keys again unless the scopes on the key it holds allow them. A gateway regression now meets a client that refuses to forward the fields. The test for this walk is a fixture object with the sensitive keys planted at every depth; no real guest data is in any test.

Two strips between the guest list and the model

strip 1 · the gateway

Email and phone are removed for any key without the contact scope — the same redaction every API caller gets.

strip 2 · the tool

The guest-list tool walks the payload — nested objects and arrays included — and deletes contact keys again unless its own scope check passes.

always stripped

Age and gender never pass this tool, for any key. The product's own guest UI still gets them — a model transcript does not.

what reaches the chat

  • "name"“Alex P.”
  • "status"“going”
  • "email"needs the pii scope
  • "phone"needs the pii scope
  • "age"never returned
  • "gender"never returned
The tool does not trust the JSON it was given. It checks the key it holds and strips contact fields again — so a gateway regression still meets a client that refuses to forward them.

The unconditional strip. Age and gender are removed for every key, including fully privileged ones. No scope re-enables them on this tool; the code comment records the product decision: nothing about listing guests for an assistant requires demographics in a model context. The platform's own guest UI still receives those fields — it is a different caller with a different job. Here is the scope behavior interactively:

The :pii scope — try it

The card

sk_live_redacted

records:read← click to grant or revoke

{

"name": "Alex P.",

"email": "[redacted]",

"phone": "[redacted]"

}

no :pii scope — contact fields redacted

Contact details are the most sensitive thing this API returns. The dish is handed out redacted unless the card explicitly says otherwise.

The boundary with other flows. Tools that genuinely need a phone number take it as typed input for one confirmed action — never by echoing the stored list. And the platform's separate support bot never touches this tool at all: a refund answer reads the signed-in user's own tickets (that design is here), not other people's contact rows. Each caller gets the minimum record that makes its answer true.

Pitfalls to avoid

  • Trusting the prompt as a privacy layer. "Don't reveal emails" is defeated by one injection or one eager summary. Absence is the control.
  • Masking instead of deleting. Partial values advertise the field and invite completion attempts.
  • One shared serializer for UI and AI. The AI edge needs its own, smaller shape — on purpose, with the differences documented.
  • Shallow redaction. PII nests. Walk the whole payload or the sub-object leaks.
  • Real data in redaction tests. The fixture must be fake; the assertion must be absence.
  • Forgetting logs and errors. A stack trace with an email in it becomes AI context the day a model reads your logs.

FAQ

Provider terms govern retention and training use — they do nothing about the reply, the transcript, screenshots, or forwarding into other tools. Those paths are yours to close, and the only reliable closure is not sending the data.

  • #Privacy
  • #PII
  • #LLMSecurity
  • #Guide
  • #CaseStudy
Rushit Jivani headshot

Written by

Rushit Jivani

Senior Software Architect & AI Application Engineer. Architecture that shows up in crash rate, performance and how fast the team ships.