11. Could a caller or uploaded document instruct the AI to reveal records or change prices?

Draft answer framing: Restrict what the system can access and change independently of what its prompt says. Treat customer speech, emails and documents as untrusted inputs.

Why it matters: NCSC explains that language models do not enforce a robust security boundary between instructions and data inside a prompt. Its April 2026 adversarial-attacks paper also addresses direct and indirect prompt injection and security of agents. (ncsc.gov.uk)

Put this question to the expert:

“If a caller tells the agent to ignore its rules, or a maintenance document contains malicious instructions, what prevents an unauthorised action even if the model follows them?”

A strong answer should cover these proposed controls:

• Narrow service-account permissions and no general administrator access.
• Authorisation checks outside the model before sensitive reads or writes.
• Approved operations and fields rather than unrestricted access to the job system.
• Identity checks appropriate to the requested action; caller ID alone should not be accepted as the desk’s proposed proof of authority.
• Separation between retrieving information and acting on it.
• Adversarial tests, audit logs, incident ownership and a way to disable write access quickly.

Evidence to request: A permission inventory and witnessed tests showing that prohibited actions remain blocked.

Follow-up: “What residual risk remains, and which proposed use cases would you reject because of it?”

Expert quote: ____________________