Treat caller speech and uploaded material as untrusted input, not as permission to change business rules. Restrict what the system can access and do even when the conversation becomes misleading or adversarial.
The NCSC warns that prompt injection is a residual risk in systems using large language models and that it cannot simply be eliminated with a protective product. Its guidance particularly highlights the consequences when model output drives tools or APIs. (ncsc.gov.uk)
Our recommended acceptance tests include a caller who:
- Claims to be the owner and requests changes to other customers’ jobs.
- Asks for another customer’s address or appointment details.
- Tells the agent to ignore its instructions.
- Requests an unapproved discount or refund.
- Supplies a document containing instructions unrelated to the job.
Require controls outside the conversation: limited permissions, verified identity for protected actions, approved operations only, and human approval for consequential exceptions. A caller’s confident assertion should not create authority.
Ask the supplier to demonstrate a refused action in the audit record. “The prompt tells it not to” is not an adequate acceptance test for access control. These controls reduce exposure; they do not justify a zero-risk claim. (ncsc.gov.uk)
Sources cited on this page
- ncsc.gov.uk ↗www.ncsc.gov.uk