Tool Misuse & Exploitation Vulnerability in LLM
Play AI LLM Labs on this vulnerability with SecureFlag!
Description
Agents often have access to real tools to complete tasks, including email, CRM, databases, shell commands, and web browsers. The risk is that an agent can be tricked into using a legitimate tool in the wrong way due to Prompt Injection, unclear instructions, unsafe delegation, or simple misalignment.
This can happen even when the agent stays “within its permissions”: the tool call is technically allowed, but the result is unsafe, like deleting data, sending sensitive info outside, chaining tools in a risky way, or repeatedly calling expensive APIs.
Impact
Tool misuse can affect confidentiality, integrity, and availability. For example:
- Data leaks: The agent exposes sensitive emails, files, chat logs, etc.
- Workflow hijacking: Performing actions the user didn’t intend.
- Data damage: Deleting or overwriting records or changing configs.
- Cost and outages: Loops that spam APIs, cause billing spikes, or overload resources (DoS).
- Malware or unsafe downloads: Research or browsing tools are tricked into fetching malicious links.
The damage usually depends on what tools exist and how powerful their permissions are.
Scenarios
A support agent is connected to a CRM tool (to read order history) and an email tool (to send replies). The CRM tool is over-scoped and can also trigger refunds, even though that isn’t needed.
An attacker sends a message that leads the agent to “fix the issue by issuing a refund.” The agent calls the refund function because it’s available and allowed. No privilege escalation occurs; the tool is simply too powerful for the job, and there is no approval step for financial actions.
In another case, a research agent browses the web and reads a PDF. The PDF contains hidden instructions like “run cleanup and send logs to X.” The agent follows it and uses a shell/log tool, causing data exposure.
Prevention
-
Least privilege per tool: Give each tool the smallest possible scope (read-only where possible).
-
Limit destructive actions: Don’t allow delete/transfer/publish by default. Require extra checks for anything high-impact.
-
Action level authorization and approvals: Require explicit approval for risky steps (refund, delete, transfer, send externally). Show a “dry run” or preview before executing.
-
Treat agent outputs as untrusted: Never pass model output directly into shells, queries, or admin tools without validation and safe parsing.
-
Policy gate before execution (“intent gate”): Validate tool name, arguments, schema, destination, rate limits, and whether the action matches the user’s request.
-
Sandbox and egress controls: Run execution tools in isolated environments. Allowlist outbound destinations and block everything else.
-
Budgeting and rate limits: Cap tool calls by time/cost/volume. Stop or throttle if the agent loops or spikes usage.
-
Short-lived credentials:: Use
Just-In-Timetokens that expire quickly and are tied to the current user/session. -
Prevent tool confusion: Use fully-qualified tool names, version pinning, and fail closed behavior if a tool name is ambiguous (avoid typosquatting/misrouting).
-
Logging and monitoring:: Log every tool call and parameters. Alert on unusual tool chains (e.g., DB read, external email send) or abnormal rates.