09

Use-case scorecard and decision options

In short: answer five questions against the baseline, then pick one decision, with an owner and a date.

A decision is never finalMonitoring and reflection loop back to measuring.
1MeasureFive scorecard questions against the baseline
2CheckThe five checks, at the review band’s bar
3Decide
ContinueContinue with conditionsImproveExpandRestrict or pauseRetire
With an owner and a date
4MonitorAt the band’s cadence, or sooner on a trigger
5ReflectWhat we learned, and what to change next time
Model change, quality drop, incident, or cost rise sends it back to step 1, with lessons from step 5 carried forward. Improve runs a 90-day cycle, then decides again.

The scorecard: five questions

Answer these five, each compared with the baseline. The example column uses the meeting summaries case.

MeasureThe questionHow to count itExample
CostWhat does it cost us now that we didn’t pay before?Extra cost8 min of review per summary; no new license
UseIs it used in the real workflow?Share of the work done with it9 of 12 staff; most weekly summaries
QualityHow much of the output is good enough after review?Accepted outputs ÷ all outputs85% accepted after light edits
OutcomeDid the result we care about change?Main outcome vs. baseline35 → 14 minutes per summary
Value for moneyWhat does each good result cost?Full cost, including rework, ÷ accepted resultsTotal cost ÷ accepted summaries

Report beside it: the five checks, open incidents, evidence limits, the owner, and the next review date.

Then pick one decision

Read down and choose the first row that fits. Every decision gets an owner and a date. The decision guide has the exact rules.

DecisionChoose it whenWhat happens next
Restrict or pauseA serious risk or failed check.Limit or stop it until the issue is fixed.
ExpandStrong results, strong evidence, support in place.Offer it to more people or uses.
ContinueIt works and every check passes.Keep it as is. Monitor on schedule.
Continue with conditionsIt works, but a check is on watch.Keep it, with named limits or fixes and a date.
ImprovePromising, but one weakness holds it back.Fix one thing, re-test within 90 days, decide again.
RetireNot worth the cost, or something better exists.Stop and remove it. Record what you learned.

The improvement cycle

From the use case inventory

Improve is a time-boxed cycle, not a holding category.

  1. Step 1

    Name the weakness

    Record a reason code: quality, adoption, evidence, cost, or equity.

  2. Step 2

    Make one targeted fix

    Change one thing, such as the prompt design, training, workflow step, or data source.

  3. Step 3

    Re-test the original hypothesis

    Use the same baseline and measures as the first evaluation.

  4. Step 4

    Decide by the deadline

    Close the cycle in 90 days or one term. Most use cases get two cycles. After that, Continue, Restrict, or Retire.

After the decision

From the use case inventory

Continue and Expand are not final. Monitor each use case at the cadence set by its review band: end of pilot for Low, mid- and end-of-pilot for Moderate, at least annually for High, and ongoing with time-limited approval for Critical.

Each check covers the headline scorecard, any model or vendor changes, and open incidents. Keep it short.

Once a year, review the whole portfolio. Put every use’s cost beside its use and value for money. Low-use, high-cost tools are the first candidates to restrict or retire. Time it before budget and renewal decisions.

Re-evaluate right away if
  • Model or vendor version change
  • Quality drops below the band’s threshold
  • Use shifts sharply up or down
  • Incident or complaint involving harm or data
  • Cost per outcome rises more than a set amount
  • New subgroup or population is served