04

Measurement domains

In short: measure three things: what it costs, how it’s really used, and what changed. How this differs from the lenses

A · CostWhat goes in
What does it really cost?
  • Licenses and usage fees
  • Staff time to set up, check, and fix
  • Training, support, and exit
B · UseWhat happens
Is it used, and used well?
AccessReal useAccepted output
Logins aren’t value. Count work that passes review.
C · OutcomesWhat comes out
What changed because of it?
  • Faster, better, or fairer service
  • Progress on mission goals
  • Harms and side effects, too
Together Value for money full cost accepted results judged against the outcomes that changed

Cost: more than the license

The card lists the usual starting points. Real costs depend on the service, the vendor, and how it’s deployed. A chat assistant, an integrated workflow tool, and an agent with permissions carry very different costs.

  • Pricing modelPer seat, per use (tokens or credits), enterprise agreement, or bundled into a suite you already pay for. Per-use pricing can rise quickly as adoption grows.
  • Setup and integrationConnecting to campus systems, data preparation, identity and access, and configuration.
  • Reviews and approvalsSecurity, privacy, accessibility, legal, and procurement review, plus renewal reviews.
  • Human review and reworkTime spent checking, correcting, and redoing outputs. Often the largest hidden cost.
  • Training and supportOnboarding, guidance, help desk load, and communities of practice.
  • Governance and evaluationTime to monitor, measure, and report on the use.
  • Compute and environmentEnergy, hosting, and resource use, where they can be measured.
  • Transition and exitData export, switching vendors, retraining, and contract wind-down.

Shared costs (a campus-wide platform, a central team) need a stated rule for splitting them across units so nothing is counted twice.

Use: more than logins

How you measure use depends on the service. Chat tools, AI built into existing software, and delegated agents leave different traces, so pick measures that fit.

  • AccessWho is eligible and who actually has access, including gaps across groups.
  • Active useUse in the intended workflow, not one-off experiments. Frequency and depth both matter.
  • Accepted outputShare of outputs that pass required human review without major correction.
  • Rework and overridesHow often people correct, discard, or work around the output.
  • Equity of useWhether use and benefit are spread fairly across roles, units, and populations served.
  • Use outside approved toolsPersonal or unapproved tools the funnel can’t see. Estimate it yearly through anonymous surveys, a software-spending review, and self-reports.

The full funnel, from access to sustainable value, is shown in section B below.

Outcomes: more than time saved

Which outcomes matter depends on the service and its purpose. An advising assistant, a document-review tool, and a coding assistant each need their own measures and baseline.

  • EfficiencyCycle time, throughput, backlog, and staff time per task.
  • Quality and reliabilityError rates, accuracy, consistency, and compliance.
  • Service and experienceWait times, responsiveness, satisfaction, and accessibility for the people served.
  • Mission outcomesStudent success, research progress, and other institutional goals the use is meant to advance.
  • Negative or unintended effectsErrors reaching people, bias, privacy incidents, deskilling, and extra workload elsewhere.

Set a baseline before launch. Report observed change separately from what can be attributed to AI.

 Checklist of what to include, by depth →
A

Cost and investment

Distinguish the budget view (incremental expenditure and documented reductions) from the economic view (plus staff time and other resources).

Include direct spending, implementation, ongoing effort (including human review and correction), transition and exit costs, and measurable resource or environmental use.

State allocation rules to avoid double-counting shared costs. Prefer cost per successful, quality-assured, mission-relevant outcome over cost per active user alone.

B

Use and integration

Use a use funnel rather than a single adoption rate. Each stage answers a different question.

Access
Do eligible people have appropriate access?
Use
Are people using AI in the intended workflow?
Quality
Are outputs accepted after required review?
Performance
Has end-to-end workflow performance improved?
Mission impact
Has the intended institutional outcome improved?
Sustainable value
Do benefits justify total cost, risk, and opportunity cost?
Fewer uses clear each stage. Value is proven only at the bottom.
Configuration
Assistive
Supports a human task.
Integrated
Operates within a defined workflow.
Delegated
Bounded actions under permissions and oversight.

More autonomy isn’t inherently better; judge against purpose. Measure the whole human-AI system: prompting, checking, correction, exceptions, escalation, and rework.

Unapproved use

Count what the funnel can't see

From the use case inventory

The funnel misses personal and unapproved tools. Estimate that use yearly with an anonymous survey, a software-spending review, and self-reports. Aim for visibility, not surveillance. Falling unapproved use means approved tools are meeting real needs.

C

Outcomes

Efficiency Quality & reliability Service & experience Mission outcomes Negative or unintended

Distinguish efficiency (completing existing work faster) from effectiveness (improving the intended service or mission outcome).

Report observed change separately from attributed effect. Attribution depends on evaluation design, comparators, reliability, and rival explanations, not on level.