What HIPAA Actually Requires for AI
HIPAA's requirements for AI systems are not explicitly spelled out in the statute - it was written before modern AI existed. But the existing Privacy, Security, and Breach Notification Rules apply fully to AI systems that process, store, or transmit protected health information. Understanding what each rule requires architecturally is the foundation for evaluating any AI vendor's HIPAA claims.
The HIPAA Privacy Rule defines what counts as PHI: any individually identifiable health information created, received, maintained, or transmitted by a covered entity or business associate. In an AI context, this is broader than most organisations realise. A patient query to an AI chatbot ("What is my appointment tomorrow?") is PHI. A clinical note fed to an AI for coding assistance is PHI. A scheduling record that includes a diagnosis code is PHI. Even a patient name combined with a treatment date is PHI. The Privacy Rule limits how PHI can be used and disclosed - and "disclosing" PHI to a cloud AI vendor for processing is a disclosure that must be covered by a BAA and satisfy the minimum necessary standard.
The HIPAA Security Rule sets out what technical safeguards must protect electronic PHI. For AI systems, the material requirements are: access controls (unique user identification, automatic logoff, encryption and decryption), audit controls (hardware, software, and procedural mechanisms that record and examine activity in systems that contain PHI), integrity controls (ensuring PHI is not improperly altered or destroyed), and transmission security (encryption of PHI in transit). For an AI system, this means the model must run in an environment where access is gated by user authentication, every access to PHI by the AI is logged, and any transmission of PHI to or from the AI is encrypted end-to-end. Shared cloud AI infrastructure typically cannot satisfy the audit control requirement with the granularity HIPAA requires.
The Breach Notification Rule creates notification obligations when PHI is improperly accessed, used, or disclosed. If a cloud AI vendor experiences a security breach affecting your patient data, you - the covered entity - have mandatory notification obligations to affected individuals, HHS, and potentially media. This is why the question of where PHI is processed is not merely a technical preference but a financial risk management decision. A breach involving PHI processed in a vendor's shared cloud infrastructure triggers the same notification requirements as a breach of your own systems - but with the added complexity that the vendor controls the forensic investigation timeline. The Business Associate Agreement requirement means any third party that processes PHI on your behalf must sign a BAA. What a BAA covers: it legally obligates the business associate to safeguard PHI, use it only for authorised purposes, and report breaches. What a BAA critically does not cover unless explicitly contracted: it does not prevent the AI model from training on your PHI. OpenAI's standard enterprise BAA does not include a no-training commitment unless you are on a specific enterprise arrangement with that commitment explicitly contracted. Many healthcare organisations discover this gap only after an audit or incident reveals that patient data was used for model improvement.
How Most AI Tools Fail HIPAA
The majority of AI tools available in 2025 - including leading consumer and SMB platforms - fail one or more of the four core HIPAA architectural requirements. Healthcare organisations that have adopted these tools based on marketing claims of "HIPAA compliance" (which often mean only that a BAA is available) are carrying compliance risk they may not have fully assessed.
Cloud Processing Exposure
Consumer and SMB tiers of GPT-4, Gemini, and Microsoft Copilot process data in shared cloud infrastructure. PHI in a prompt - a patient question, a clinical note, a scheduling record - is PHI in the vendor's cloud the moment it is submitted. Regardless of a BAA, the data has left your controlled environment and is subject to the security controls of the vendor's shared infrastructure. These controls may be strong, but they are the vendor's controls, not yours, and you cannot independently audit them to the depth HIPAA's audit requirements assume.
Model Training on PHI
Unless you are on a no-training enterprise contract with an explicit contractual commitment, your data may be used to improve model performance. This applies to many AI platforms by default. PHI used for model training without patient authorisation is a HIPAA violation - it constitutes a use of PHI for a purpose not covered by the Privacy Rule's treatment, payment, and operations exception. The model improvement purpose is not a covered HIPAA use. Healthcare organisations must verify, in writing and in contract, whether their AI vendor's model trains on customer data - and this must be confirmed for the specific tier they are using, not the enterprise tier they have not subscribed to.
Audit Trail Gaps
HIPAA's Security Rule requires audit controls that record and examine activity in systems containing PHI. The required granularity: who accessed which PHI, when, and in what context. Most AI tools provide aggregate usage logs - number of queries, tokens consumed, API call volumes. They do not provide patient-level audit logs that show which patient's data was accessed in which AI interaction, by which user. This is not a configuration issue - it is an architectural limitation of how shared cloud AI systems work. HIPAA compliance requires patient-level PHI access logging, and most AI platforms simply do not produce it.
The HIPAA-Compliant AI Architecture
A HIPAA-compliant AI architecture is not more complex than a non-compliant one - it is different. The four requirements below are non-negotiable for any AI system that processes PHI. Each one eliminates a distinct failure mode that creates regulatory exposure.
-
On-Premise or Private Cloud Deployment
The AI model runs inside your perimeter - in your data centre or in a private cloud environment (a dedicated VPC) that your organisation governs. PHI never leaves your infrastructure to reach an external AI service. This single architectural decision eliminates cloud processing exposure and puts your organisation in control of the security environment the AI operates within. It also makes your compliance posture independently auditable - you can demonstrate to HHS, to a HITRUST assessor, or to your own compliance team exactly where PHI is processed and under what controls.
-
No External Model Calls
The AI agent uses a locally deployed model - no API calls to OpenAI, Anthropic, Google, or any external AI service. The model weights are deployed on your infrastructure and run entirely within your environment. When a clinician asks the AI to draft a discharge summary, that query and the patient data it references never leave the hospital network. This requirement eliminates both the cloud processing exposure and the model training risk in a single architectural decision. The locally deployed model is fine-tuned on your clinical workflows, your documentation standards, and your terminology - producing better outputs for your specific context than a generic cloud model would provide.
-
Role-Based Access Controls
Who can invoke the AI agent, what patient data it can access, and what actions it can take are all gated by the same RBAC system your EHR uses. A clinical documentation agent accessed by a physician should be able to access that physician's patient panel's records. A billing agent should be able to access claims data but not clinical notes beyond diagnosis codes. A patient-facing appointment agent should access scheduling data but not full clinical history. These access boundaries are enforced at the agent level, not just the application level, and every access decision is logged for audit. This satisfies HIPAA's minimum necessary standard and its access control requirements simultaneously.
-
Immutable Audit Logs
Every agent query, every data access event, every action taken is logged with four mandatory attributes: timestamp, user identity (the human who invoked the agent), data accessed (which patient record, which PHI fields), and action taken (what the agent read, wrote, or transmitted). Logs are immutable - they cannot be altered or deleted by any system user, only by a logged administrative action. These logs satisfy HIPAA's audit control requirements and provide the forensic record needed for breach investigation and OCR inquiries. They are also the foundation for your own AI governance programme - the ability to audit what the AI is doing with PHI on an ongoing basis, not just after an incident.
HIPAA-Compliant AI Use Cases with Upcore
The following use cases are fully deployable within a HIPAA-compliant architecture. Each one involves PHI processing - clinical notes, patient records, insurance data, scheduling records - and each is designed with on-premise deployment, no-training guarantees, RBAC, and immutable audit logging as standard.
Clinical Documentation
Physician note drafting, ICD-10 coding assistance, and discharge summary generation - all processing PHI entirely on-premise. AI dramatically reduces the documentation burden that consumes 30–40% of physician time, while producing structured, compliant documentation. The agent works within your EHR environment, accessed through your existing authentication, with every clinical note access logged at the patient level.
Prior Authorization Automation
Insurance pre-authorization requests - extracting clinical data from patient records, mapping to payer criteria, and drafting submission packages - with a complete compliance audit trail. Prior auth is one of the most time-intensive administrative burdens in US healthcare; AI reduces the time per request from hours to minutes while maintaining the documentation standard payers require. Every record access and submission is logged for compliance review.
Patient Communication Agent
Appointment reminders, care plan follow-up messages, no-show re-booking, and post-discharge check-ins - using HIPAA-compliant messaging channels (encrypted SMS, secure patient portal). The agent personalises communications based on the patient's care plan and history, improving engagement while maintaining the access controls and audit logging that HIPAA requires for PHI-containing communications.
Revenue Cycle Management
Claim preparation, denial triage, appeals letter generation, and payer follow-up - integrated with your billing system and governed by the same RBAC and audit framework as clinical AI. Revenue cycle AI can reduce denial rates by 20–30% through more accurate initial claim preparation and faster, more comprehensive appeals. All billing data access is logged at the patient and claim level for HIPAA audit purposes.
Care Coordination
Patient routing, referral management, and specialist handoff coordination - across care team members and sites of care. Care coordination AI maintains a shared view of the patient's care pathway that updates in real time as new encounters occur. All care team members with appropriate access can query the agent for current patient status, pending referrals, and outstanding care plan items. Every access is tied to a user identity and logged.
Compliance & Quality Reporting
HEDIS measure calculation, CMS quality reporting programme submissions, and patient safety event documentation - AI agents that query clinical records, calculate measure performance, and produce formatted submissions. Quality reporting is one of the most resource-intensive compliance obligations in healthcare; AI reduces the analyst time required per reporting cycle while improving accuracy. All PHI accessed for quality reporting is logged under the healthcare operations exception to HIPAA's minimum necessary standard.
Questions to Ask Any AI Vendor Before Signing
Healthcare technology leaders have a fiduciary obligation to evaluate AI vendors against HIPAA requirements, not just marketing claims. The following eight questions are the minimum due diligence for any AI vendor that will process PHI. A vendor that cannot answer these questions clearly and in writing should not be given access to patient data.
- Does your platform support on-premise deployment or dedicated VPC deployment where our data never leaves our perimeter? (Ask for a reference customer in healthcare who runs your platform on-premise.)
- Is your BAA comprehensive, or does it exclude AI model training? (Request the specific contractual language that governs training on customer data. "We offer a BAA" is not sufficient - you need to know what the BAA covers.)
- Can you provide audit logs of every PHI access event, at the patient level, with user identity and timestamp? (Ask for a sample log format, not a general description of your logging capability.)
- Does your model training use customer data by default? What is the opt-out mechanism, and is it available on our tier? (Get this in writing in the contract, not just in a privacy policy that can be changed.)
- What are your breach notification timelines and processes? (HIPAA requires notification within 60 days of discovery - confirm your vendor's discovery-to-notification timeline aligns with your obligations.)
- Do you support role-based access controls at the agent level - not just at the application level? (HIPAA's minimum necessary standard requires that access to PHI be limited to what is needed for the specific task. Ask how this is implemented for AI agents specifically.)
- What is your SOC 2 Type II and HITRUST status? (Self-certification is not sufficient for enterprise healthcare deployments. Require independently assessed certifications.)
- How is data encrypted at rest and in transit? What encryption standards are used, and are you willing to provide technical documentation? (AES-256 at rest and TLS 1.2+ in transit are baseline requirements; ask whether these apply to your specific deployment configuration.)