Limit exposure, observe the entire workflow and compare like with like.
What should happen during rollout?
- Shadow testing: staff assess proposed outputs without customer-facing actions.
- Limited live use: enable one approved route, branch or message cohort.
- Close review: inspect every early action and all exceptions.
- Controlled expansion: increase scope only after the previous stage meets its gate.
- Outcome follow-through: track enquiries through completed work, not only initial response.
Choose the initial slice for operational controllability. Do not automatically select out-of-hours calls if that removes the people needed for escalation.
Where practical, compare equivalent eligible enquiries using the current process and the pilot during the same period. Do not remove existing safety protections from either group. If using a before-and-after comparison, record changes in staffing, advertising, service mix and capacity, and describe the result as indicative rather than clean causal proof.
What should the weekly scorecard show?
| Measure | Definition |
|---|---|
| Eligible opportunities | Unique enquiries that fit the agreed scope |
| Capture completeness | Opportunities with sufficient verified details for the next action |
| Confirmation accuracy | Claimed appointments or updates supported by the destination system |
| Human intervention | Minutes spent reviewing, correcting and resolving exceptions |
| Completed-job contribution | Contribution from completed work, with attribution rules stated |
| Customer friction | Complaints, abandonment and requests for a person |
| Failure rate | Failed or unresolved events divided by attempted events |
| Operating cost | Subscriptions, usage, support and supervision |
| Released staff capacity | Net time released, separately from cash savings |
Do not declare success because the AI answered every call. Your decision is whether the complete operating process improved after correction effort and costs.
PUT THE RESEARCH TO WORK