AI Data Governance:
The Enterprise CTO/CIO Checklist

AI data governance is the set of policies, controls, and processes that determine how data is collected, stored, accessed, processed, and retained within an AI system — and how the AI's decisions are documented, explained, and audited. It covers who can ask the AI to process data, what data the AI is permitted to access, where that data is processed and stored, how long it is retained, what happens when the AI produces a decision that needs to be explained or challenged, and how incidents are detected and responded to. AI data governance is an extension of an organisation's broader data governance framework, with additional requirements specific to AI: training data controls, model explainability, algorithmic accountability, and continuous monitoring for model drift and bias.

Yes — GDPR applies directly to AI systems that process personal data of EU data subjects, regardless of where the processing organisation is located. Key GDPR provisions include: Article 5 (data minimisation and purpose limitation), Article 13/14 (transparency — individuals must be informed when their data is processed by AI), Article 22 (automated decision-making — individuals have the right not to be subject to solely automated decisions with significant effects), Article 32 (security — appropriate technical measures for AI data processing), and Article 28 (data processing agreements with AI vendors acting as data processors). Additionally, the EU AI Act adds requirements for high-risk AI systems beyond GDPR, with phased compliance timelines through 2026.

Regular data governance covers data quality, cataloguing, access controls, retention policies, and compliance for data at rest and in use. AI data governance extends this with five additional requirement areas: training data governance (what data trained the model, was it representative and legally obtained?), model explainability (can the AI's decisions be explained to humans and regulators?), inference logging (every query and output must be auditably logged), model drift monitoring (AI models can degrade as real-world data diverges from training data), and algorithmic accountability (who is responsible when the AI produces a wrong or harmful output?). Regular data governance frameworks typically do not address these requirements.

The EU AI Act is the world's first comprehensive AI regulatory framework, passed in 2024 with phased compliance requirements through 2026–2027. It classifies AI systems by risk level. Prohibited AI includes social scoring and real-time biometric surveillance. High-risk AI — including AI used in credit scoring, insurance, employment decisions, benefits eligibility, critical infrastructure, and law enforcement — requires conformity assessments, risk management documentation, human oversight mechanisms, accuracy and robustness testing, audit trail maintenance, and registration in an EU database. For enterprise AI in regulated sectors, the high-risk classification requirements are the most significant. Compliance applies to both EU-based organisations and non-EU organisations whose AI systems affect EU individuals.

An enterprise AI governance policy should cover five areas. First, AI inventory and classification: a register of all AI systems, classified by risk level aligned with the EU AI Act or NIST AI RMF. Second, data governance extension: how existing data policies apply to AI-processed data and what additional controls are needed. Third, accountability and oversight: designated AI governance roles, approval processes for new deployments, and escalation paths for incidents. Fourth, transparency and explainability: requirements for AI systems to explain their decisions and documentation standards for AI model behaviour. Fifth, monitoring and incident response: ongoing performance monitoring, bias auditing schedules, and a defined incident response process. The policy should be reviewed annually as the regulatory landscape and your AI deployment profile evolve.

In regulated industries, an AI agent's data access is governed by the intersection of regulatory requirements, contractual obligations, and internal data governance policies. For BFSI: customer financial data access is governed by RBI guidelines and data localisation requirements; sharing with third-party AI vendors requires explicit customer consent or a data processing agreement. For healthcare: patient health information (PHI) is governed by HIPAA — AI systems processing PHI must have a Business Associate Agreement, PHI may not be used for model training without explicit consent, and access must be on a minimum necessary basis. The principle across all regulated industries: the AI agent should have access to the minimum data necessary for its stated function, and every data access must be loggable and attributable.

Ensuring AI explainability for regulatory purposes requires design decisions at the architecture level — it cannot be retrofitted after deployment. Three approaches provide different levels of explainability. Structured reasoning logs: the AI agent records the inputs it considered, the rules or policies applied, and the factors that determined its output in an auditable format — the most practical approach for regulatory compliance. Model-level interpretability: using architectures designed for explainability (decision trees, rule-based systems) rather than black-box neural networks. Post-hoc explanation methods: tools like SHAP values that provide feature attribution analysis — acceptable in some regulatory contexts but increasingly challenged. For regulated decisions, Upcore designs agents with structured reasoning logs that produce regulatory-ready explanation artefacts as a standard output.

Data lineage in AI governance refers to the complete documented history of a piece of data: where it originated, how it was transformed or used in AI training or inference, who accessed it, when, and for what purpose. For AI systems, lineage documentation is required at two levels. Training data lineage: what datasets trained the model, where they came from, what preprocessing was applied, and whether they contained personal data obtained with appropriate consent. Inference data lineage: for every AI decision, which specific data points informed the output — the customer record retrieved, the policy document consulted, the prior interactions considered. Regulators increasingly require organisations to demonstrate they can trace any AI decision back to its source data, to satisfy both explainability requirements and data subject rights (e.g., an individual's GDPR right to know what data was used in a decision about them).