AI Governance Framework
for Mid-Market SaaS

3.1  AI Policy & Program Management

Domain Statement

This domain covers the foundational policy infrastructure that makes every other governance domain enforceable. AI Policy & Program Management defines the organizational scope of AI governance, establishes acceptable-use boundaries across all AI systems, codifies approval workflows for new AI tool adoption, and sets a review cadence that keeps policy current as the technology and regulatory landscape evolve. Critically, this domain explicitly addresses shadow AI — tools adopted outside IT procurement — because an AI governance program that only covers sanctioned tools has a large, invisible blind spot.

Governing Principle

This domain enforces Principle 2: Evidence over policy. A governance control that cannot produce evidence of enforcement is a governance gap, not a governance control. The question is never "do you have a policy?" — it is "what evidence exists that the policy is followed?" Every component of Domain 1 is designed to generate auditable evidence, not just documentation.

Maturity Model

RACI Table

Governance Indicators

• Published AI governance policy with a version history showing at least one completed review cycle
• AI tool inventory (sanctioned and shadow) with a documented last-refreshed date no older than 90 days
• Shadow AI discovery scan results logged on a quarterly cadence, with disposition documented for each finding (approved, prohibited, or under review)
• Policy training completion rates by department, with records showing who completed training and when
• AI tool approval workflow audit log showing all requests, reviewers, and outcomes
• Named policy owner with a title, reporting line, and explicit accountability documented in the policy itself

Industry Alignment

This domain maps to the NIST AI RMF GOVERN function, which establishes the organizational practices, culture, and processes necessary for responsible AI risk management. It also aligns with ISO/IEC 42001 Clause 5 (Leadership and AI Policy), which requires top management to establish an AI policy that is appropriate to the organization’s context, provides a framework for setting AI objectives, and is communicated and made available to relevant stakeholders.

PE Due Diligence Signal

3.2  AI Risk Management

Domain Statement

This domain establishes the proactive infrastructure for identifying, assessing, and mitigating risks that are specific to AI systems — including model bias and fairness failures, hallucinated outputs that propagate into decisions or customer communications, data privacy exposure from model training or inference, adversarial manipulation through prompt injection or jailbreaking, and operational failures from agentic systems with real-world authority. These risk categories require a dedicated framework and cross-functional ownership; they cannot be adequately addressed by folding them into existing IT or cyber risk programs. This domain requires board-level visibility because the blast radius of a significant AI incident — reputational, regulatory, financial, or operational — can be material to enterprise value.

Governing Principle

This domain enforces Principle 3: Governance scales with blast radius. The rigor of AI risk oversight is proportional to what each system can do, spend, or break — and the core operational concept of this domain is blast-radius scoping: documenting the scope, privileges, and kill-switch ownership of every deployed AI agent. Principle 5 (AI incidents are distinct from cyber incidents) is also directly applicable here, particularly at maturity levels 4 and 5 where dedicated incident response playbooks and tabletop exercises are required.

Maturity Model

RACI Table

Governance Indicators

• AI risk register with named risk owners, severity ratings, mitigation status, and a documented review cadence no less frequent than quarterly
• Blast-radius documentation for every deployed AI agent, specifying the systems it can read or write to, the communications it can send, the maximum spend or action authority it holds, and the name and title of its kill-switch owner
• Adversarial testing log showing prompt injection and red-team exercise cadence, scenarios tested, and findings with disposition
• AI-specific incident response plan that is distinct from the cyber IR playbook, with its own escalation paths, role assignments, and communication templates
• Tabletop exercise record showing an AI-specific scenario has been run in the last 12 months, with findings and follow-up actions documented
• Quantified AI risk metrics tracked over time (e.g., hallucination rate per model, bias evaluation results, incident count by category)

Industry Alignment

This domain maps to the NIST AI RMF MAP and MEASURE functions, which together cover the identification of AI risks in context and the analysis and monitoring of those risks over the AI system lifecycle. It also aligns with the EU AI Act’s risk classification system (unacceptable, high, limited, minimal), which requires organizations deploying or integrating AI in the EU to assess risk tier before deployment and apply proportional conformity and monitoring requirements for high-risk applications.

PE Due Diligence Signal

3.3  Regulatory & Compliance Readiness

Domain Statement

This domain covers the systematic mapping of applicable AI-specific regulations to current systems, gap analysis against those requirements, remediation tracking with documented timelines, and proactive monitoring of emerging regulatory changes before they become deployment blockers. The key word is proactive. Compliance in AI governance is not a point-in-time certification exercise — it is a continuous posture maintained across the organization’s full AI portfolio, updated as both the regulatory landscape and the company’s AI capabilities evolve. In regulated verticals such as financial services, healthcare, and insurance, the gap between "we are aware of these requirements" and "we have mapped them to our product capabilities with documented remediation status" is the gap between a deployment-ready AI program and one that is perpetually at risk of being stalled by the compliance team.

Governing Principle

This domain enforces Principle 7: Governance enables velocity. Proactive compliance removes blockers before they stall deployment. The organizations that move fastest through regulated environments are not the ones who ask legal for approval after a feature ships — they are the ones who identified the applicable regulatory requirements before the sprint started, mapped them to design decisions early, and maintained a remediation log that legal can sign off on without a six-week discovery process.

Maturity Model

RACI Table

*Legal serves as both Responsible and Accountable for regulatory change monitoring, reflecting the specialized expertise required. Board-level reporting to CEO and Board (Informed) provides external oversight.

Governance Indicators

• Regulatory applicability matrix identifying which regulations apply to which AI features and markets, with a documented last-reviewed date
• Gap analysis with remediation status, named owner, and target completion date for each open item
• Compliance review cadence documented and operating at a minimum semi-annual frequency, with meeting records and sign-off
• Regulatory change alert process with a defined response SLA, assignment workflow, and log of changes assessed in the last 12 months
• Board-level compliance summary included in regular AI governance reporting

Industry Alignment

This domain maps to the NIST AI RMF GOVERN function (legal and regulatory compliance obligations) and the MANAGE function (response to identified AI risks including regulatory gaps). It also aligns with EU AI Act conformity assessment requirements for high-risk AI systems, state-level AI legislation (Colorado SB 205, Illinois BIPA, and emerging state frameworks), and SOC 2 AI-specific controls emerging in Type II audits for SaaS companies with AI features that process customer data.

PE Due Diligence Signal

3.4  AI Ethics & Responsible Use

Domain Statement

This domain covers the documented principles governing fairness, transparency, and accountability in AI systems deployed or developed by the organization. The critical distinction is between aspirational and operational: an ethics framework that lives in a slide deck presented at an all-hands and never referenced again is decoration, not governance. This domain moves beyond declarations to the enforcement mechanisms, testing protocols, and external accountability structures that make principles real — bias testing by AI feature, acceptable use policies with training completion tracking, and customer-facing disclosure practices that match what the technology actually does. An ethics framework that has been tested carries more governance weight than one that has merely been published.

Governing Principle

This domain enforces Principle 1: Human accountability is non-delegable. An AI system cannot be the accountable party. Every AI capability has a named human owner — a person with a title, a reporting line, and explicit responsibility for what the system does and the decisions it influences. In the ethics domain, this principle applies with particular force: when an AI system produces a biased outcome, makes a consequential recommendation, or fails in a way that harms a user, the question "who is responsible?" must have a human answer, and that answer must have been documented before the incident, not assigned after.

Maturity Model

RACI Table

Governance Indicators

• Published AI ethics framework (internal or external) with version history and a documented last-review date
• Bias testing results logged by AI feature, with methodology, evaluation cadence, and disposition of findings
• Acceptable use policy issued and tracked, with training completion rates by department and a last-updated date
• Customer-facing AI disclosure practices documented, with a review process to ensure disclosures remain accurate as models are updated or retrained
• Named accountability owner for each AI system that makes or influences consequential decisions

Industry Alignment

This domain maps to NIST AI RMF GOVERN 1.2, which establishes organizational values, principles, and policies as the foundation for responsible AI risk management. It also aligns with the IEEE 7000 series on ethical considerations in autonomous systems design, and ISO 42001 Annex A (responsible AI objectives), which requires organizations to define and operationalize responsible AI objectives as part of their AI management system.

PE Due Diligence Signal

3.5  Explainability & Audit

Domain Statement

This domain covers the organization’s ability to explain AI-driven decisions to customers, regulators, and the board — supported by audit trails, decision documentation, and designated accountability for each AI system. Explainability is not a retrospective activity; it is built into AI systems by design, with methodology and rationale captured at deployment, not reconstructed after the fact. In regulated environments and in any context where AI influences consequential outcomes, the ability to produce a clear, accurate account of how a decision was reached is a business requirement, not a technical nicety.

Governing Principle

This domain enforces Principle 1: Human accountability is non-delegable. Every AI system that makes or influences a consequential decision must have a named human owner — and explainability is what makes that accountability real. An accountable owner who cannot explain how their system reached a decision is accountable in name only. Principle 2 (Evidence over policy) is also directly applicable here: either an audit trail exists that can support a complete account of an AI-driven decision, or it does not.

Maturity Model

RACI Table

Governance Indicators

• Audit trail coverage per AI feature, with a log confirming operational status and last-verified date
• Explainability documentation per customer-facing AI decision, including methodology, inputs considered, and rationale format
• Board reporting template for AI systems, with a defined cadence and documented delivery history
• Designated accountability per AI system — named owner with title, reporting line, and explicit responsibility documented before deployment

Industry Alignment

This domain maps to EU AI Act Article 13 (transparency obligations for high-risk AI systems), which requires that high-risk AI systems be designed to allow deployers to interpret and explain outputs. It also aligns with NIST AI RMF MEASURE 2.6–2.8 (interpretability and explainability), which calls for systematic documentation of how AI outputs are generated and how decisions can be explained to affected parties. In financial services, FCRA and ECOA impose explanation requirements for credit decisions influenced by AI, making audit trail completeness a direct compliance obligation.

PE Due Diligence Signal

3.6  AI Vendor & Supply Chain Governance

Domain Statement

This domain covers the management of third-party AI providers, sub-processors, and model dependencies — including data handling practices, contractual protections, exit provisions, and concentration risk. AI vendors — model API providers, fine-tuning platforms, inference infrastructure, and AI-enabled SaaS tools that process customer data — are critical operational dependencies, not interchangeable software subscriptions. This domain treats AI vendor governance as supply chain governance: with the same rigor, visibility, and risk management discipline applied to any other single point of failure in the company’s operational infrastructure.

Governing Principle

This domain enforces Principle 6: Vendor governance is supply chain governance. AI providers are critical operational dependencies, not interchangeable SaaS utilities. Concentration risk, data residency, model versioning, and contractual exit provisions for AI vendors carry different risk profiles than typical software procurement — and the consequences of discovering that gap during a diligence process are materially worse than discovering it during a governance review.

Maturity Model

RACI Table

Governance Indicators

• AI vendor register with security attestations, risk tier assignments, and review dates for each vendor
• Data sub-processor tracking per vendor, documenting what customer or company data is shared, under what terms, and where it is retained
• Contractual exit clause enforcement log, confirming that AI-specific exit provisions exist and have been reviewed in active vendor contracts
• Vendor concentration analysis mapping single-provider dependencies at the AI capability layer, with documented mitigation or acceptance decisions
• AI-specific insurance endorsement confirmed — not generic cyber coverage, but coverage that addresses AI-related failure modes including model performance, data exposure, and third-party AI system failures

Industry Alignment

This domain maps to NIST AI RMF GOVERN 5, which addresses the governance of AI risks in the context of third-party relationships and supply chain dependencies. It also aligns with EU AI Act supply chain obligations for organizations that integrate AI components from third-party providers, and with ISO 27001 supplier relationship controls, which require systematic management of the security risks associated with supplier access to organizational information and systems.

PE Due Diligence Signal