Evaluation lenses and decision gates
In short: five checks can stop a use, however good its numbers are. A serious risk blocks it; other gaps lead to fixes, limits, or more evidence. How this differs from the domains
The lenses are checks that can override the numbers. Strong productivity results do not offset unacceptable harm or unresolved critical risks.
Capability and readiness
Use observable indicators: role-based proficiency, reviewer competence, process maturity, data readiness, operational support. Promising pilots do not guarantee scale.
Lifecycle and dependency risk
Track vendor dependence, model changes, integration maintenance, data portability, exit costs, service continuity, and switching costs, especially for core systems.
Decision thresholds
Set minimum quality, maximum risk, evidence and financial requirements, and approval authority, scaled to the use case's consequences. See review bands below.
What a failed check means
A failed check stops a use from going ahead as proposed, however good its numbers are. It doesn’t always mean shutting it down. The kind of failure sets the response.
| Finding | Response | Usual decision |
|---|---|---|
| A non-negotiable rule is broken (law, policy, data protection, accessibility requirement) | Stop, or pause the affected use | Restrict or pause |
| A serious issue that can be fixed | Narrow the use, fix it, then re-check | Restrict or Improve |
| Not enough evidence to judge | Keep the scope small and collect more | Improve, or continue as a pilot |
| Below the agreed threshold | Improve or redesign; retire if it can’t be fixed | Improve or Retire |
Review bands
From the AI Strategic CompassEvery use case faces the same gates, but the bar rises with the stakes. The bar comes from the review band the Compass assigns at intake, so a use case keeps one risk rating from proposal to decision. Approved everyday use that skips the Compass is treated as Low.
| Band | Compass review | Minimum quality | Evidence required | Decides | Monitoring |
|---|---|---|---|---|---|
| LowScore 5–10 | Light · self-assessment | 0.60 | Before-and-after timing or a spot check | Unit lead | End of pilot |
| ModerateScore 11–15 | Standard · cross-functional review | 0.70 | Baseline plus comparator, or a phased rollout | AI governance lead | Mid- and end-of-pilot |
| HighScore 16–20 | Full · specialist review | 0.75 | Controlled or phased design, subgroup results, accessibility audit | AI governance council | Ongoing, at least annually |
| CriticalScore 21–25 | Extended · executive decision | 0.80 | All of High, plus alternatives analysis and impact assessment | Executive sponsors | Ongoing; time-limited approval |
Band names, score ranges, deciders, and monitoring follow the Compass. Minimum quality and evidence are this framework’s additions. A 5 on ethics or stakeholder impact raises a use case one band, whatever the total. All values are starting points; set and publish your own.
Default equity checks
From the use case inventoryFour low-burden checks make the equity lens usable without a separate assessment. Low-band uses need the first and third. Moderate and above use all four.
Differential access
Share of eligible people who have and use the tool, by role, program, or student group.
Differential outcome quality
Accuracy, completion, or error rates compared across subgroups served.
Accessibility
Audit result for the interface and the share of reported barriers fixed within a set time.
Burden
Who carries extra review, correction, or wait time when AI falls short.