AI in regulated industries: shipping intelligence under HIPAA, PCI and friends
Compliance doesn't have to kill your AI roadmap — but it does change the architecture. Patterns we use to ship AI into healthcare and fintech without turning every release into a legal review.
There's a belief in regulated industries that AI is something for later — after the auditors relax, after the rules catch up. Meanwhile the workflows that would benefit most from AI are precisely the ones buried in clinical notes, transaction reviews and compliance paperwork.
Having shipped AI into healthcare and financial workflows, our experience is that regulation rarely prohibits AI. What it prohibits is sloppy AI: systems that can't say what data they touched, can't explain their outputs, and can't show a human was accountable for the consequential decisions. Those constraints are real — and they're all designable. The teams that treat compliance as an architecture input, rather than a review gate at the end, ship faster and sleep better.
Decide what the model is allowed to see — then enforce it in code
The first design question in any regulated AI system is data flow: what leaves your boundary, where does it go, and under what agreement? Model providers now offer the contractual machinery — business associate agreements, zero-retention terms, regional processing — but the contract only covers the data you actually needed to send.
The deeper pattern is minimization at the boundary. Most AI tasks don't need the identifiers: a model summarizing a clinical note doesn't need the patient's name, and a model categorizing transactions doesn't need the full card number — PCI scoping already taught that lesson with tokenization. Build a redaction and tokenization layer in front of every model call, strip or pseudonymize identifiers on the way out, and re-attach them on the way back. Enforce it in the pipeline, not in a policy document, so the compliant path is the only path that compiles.
Build the audit trail before the feature
When an auditor, regulator or opposing counsel asks why the system flagged this transaction or produced that summary, 'we can't reconstruct it' is the worst possible answer. Probabilistic systems make this harder: the same input might not produce the same output next month, so if you didn't record the moment, it's gone.
The fix is to treat every AI decision as a record. Log the input the model actually saw (post-redaction), the prompt and model version, the retrieved context, the output, the confidence, and who acted on it — immutably, with retention matching your regulatory clock. This feels heavy until the first audit, when it becomes the cheapest thing you ever built.
- Version everything that shapes an output: prompts, models, retrieval indexes, thresholds
- Log inputs and outputs per decision, in a store designed for retention and legal hold
- Record the human action taken on each AI output — accepted, edited, overridden
- Make the trail queryable by case, not just by timestamp — audits ask about specific decisions
Put the human where the regulation points
Most regulatory frameworks converge on the same principle: a human must be accountable for decisions that materially affect a person — a denial of credit, a change in care, a closed account. The architecture that satisfies this is the one we'd recommend anyway: AI drafts, ranks and flags; a qualified human decides.
The trap is oversight theater — a reviewer rubber-stamping five hundred AI suggestions a day is legally a human in the loop and practically a formality that will not survive a serious audit. Design reviews people can genuinely perform: show the evidence behind each recommendation, keep queues at a volume where attention is real, and track override rates. A reviewer who never disagrees with the model isn't reviewing, and that metric is one regulators increasingly know to ask for.
Match the explainability to the decision
Regulated industries carry explanation obligations — adverse action notices in lending, clinical justification in care — and 'the model said so' meets none of them. But explainability isn't one requirement; it scales with consequence, and over-engineering it everywhere is how compliance programs stall.
For low-stakes assistive outputs, grounding is usually enough: citations back to the source document a human can verify. For consequential decisions, prefer architectures that are explainable by construction — a model that extracts defined, reviewable factors which then feed transparent rules will satisfy an examiner far better than a post-hoc rationalization bolted onto a black box. The choice of architecture is itself a compliance decision; make it deliberately, per use case.
Start where the risk is lowest and the paperwork is heaviest
The best first AI projects in regulated environments share a shape: internal-facing, human-reviewed, and aimed at the documentation burden rather than the regulated decision itself. Drafting the compliance narrative, summarizing the case file, pre-filling the audit checklist — work where the regulation is the reason the workload exists, and where a wrong draft is caught by the person who was already doing the task.
These projects build the muscle that the higher-stakes ones need: the redaction layer, the audit logging, the review workflows, and the working relationship with your compliance team. By the time you approach decisions that regulators watch closely, the controls aren't a proposal — they're running infrastructure with months of evidence behind them. In regulated industries that evidence, more than any model, is what gets the next project approved.