Agree the baseline, comparison method, acceptance criteria and stop rules before any live deployment. A pilot should establish whether the new arrangement improves your operation—not merely whether calls can be answered.
Our suggested sequence is:
- Baseline: classify existing enquiries, outcomes, office effort and errors.
- Controlled testing: run normal and exception scenarios without real commitments.
- Restricted deployment: use one workflow, branch or defined overflow period.
- Outcome review: allow enough time for the relevant jobs to complete.
A four-week trial might reveal workflow defects but cannot establish completed-job contribution where your sales and delivery cycle is longer. Set the observation period around your business, not a standard agency package.
| Metric | Suggested definition |
|---|---|
| Eligible enquiries | Unique enquiries within the agreed workflow |
| Completed-job rate | Completed jobs divided by eligible enquiries |
| Incremental contribution | Additional completed-job contribution minus incremental costs |
| Booking correctness | Correct committed appointments divided by all committed appointments |
| Exception burden | Staff time spent reviewing and correcting the workflow |
| Handover success | Completed required handovers divided by required handovers |
| Customer harm or complaint | Separately reviewed incidents, not hidden in an average |
Use a contemporaneous comparison where feasible. Otherwise, document differences in weather, marketing, engineer capacity and job mix rather than attributing every change to AI.
Recommended stop rules include unsafe advice, unauthorised disclosure, repeated false confirmations or failure of the fallback route. Zero observed incidents in a small pilot does not prove zero future risk.