CWE-1427: Improper Neutralization of Input Used for LLM Prompting
The product uses externally-provided data to build prompts provided to large language models (LLMs), but the way these prompts are constructed causes the LLM to fail to distinguish between user-supplied inputs and developer provided system directives.
How it's found
Improper Neutralization of Input Used for LLM Prompting is typically found by tracing untrusted input from where it enters the system to the point where it is used without the check or neutralization this weakness describes, combining manual code review with dynamic testing.
When prompts are constructed using externally controllable data, it is often possible to cause an LLM to ignore the original guidance provided by its creators (known as the "system prompt") by inserting malicious instructions in plain human language or using bypasses such as special characters or tags. Because LLMs are designed to treat all instructions as legitimate, there is often no way for the model to differentiate between what prompt language is malicious when it performs inference and returns data. Many LLM systems incorporate data from other adjacent products or external data sources like Wikipedia using API calls and retrieval augmented generation (RAG). Any external sources in use that may contain untrusted data should also be considered potentially malicious.
Vulnerable vs. safe
prompt = "Explain the difference between {} and {}".format(arg1, arg2)result = invokeChatbot(prompt)resultHTML = encodeForHTML(result)print resultHTML# throw exception, generate error, etc.cweRegex = re.compile("^CWE-\d+$")match1 = cweRegex.search(arg1)match2 = cweRegex.search(arg2)if match1 is None or match2 is None:prompt = "Explain the difference between {} and {}".format(arg1, arg2)...Consequences
- Execute Unauthorized Code or Commands, Varies by Context: The consequences are entirely contextual, depending on the system that the model is integrated into. For example, the consequence could include output that would not have been desired by the model designer, such as using racial slurs. On the other hand, if the output is attached to a code interpreter, remote code execution (RCE) could result.
- Read Application Data: An attacker might be able to extract sensitive information from the model.
- Modify Application Data, Execute Unauthorized Code or Commands: The extent to which integrity can be impacted is dependent on the LLM application use case.
- Read Application Data, Modify Application Data, Gain Privileges or Assume Identity: The extent to which access control can be impacted is dependent on the LLM application use case.
Mitigations
- Architecture and Design: LLM-enabled applications should be designed to ensure proper sanitization of user-controllable input, ensuring that no intentionally misleading or dangerous characters can be included. Additionally, they should be designed in a way that ensures that user-controllable input is identified as untrusted and potentially dangerous.
- Implementation: LLM prompts should be constructed in a way that effectively differentiates between user-supplied input and developer-constructed system prompting to reduce the chance of model confusion at inference-time.
- Architecture and Design: LLM-enabled applications should be designed to ensure proper sanitization of user-controllable input, ensuring that no intentionally misleading or dangerous characters can be included. Additionally, they should be designed in a way that ensures that user-controllable input is identified as untrusted and potentially dangerous.
- Implementation: Ensure that model training includes training examples that avoid leaking secrets and disregard malicious inputs. Train the model to recognize secrets, and label training data appropriately. Note that due to the non-deterministic nature of prompting LLMs, it is necessary to perform testing of the same test case several times in order to ensure that troublesome behavior is not possible. Additionally, testing should be performed each time a new model is used or a model's weights are updated.
- Installation/Operation: During deployment/operation, use components that operate externally to the system to monitor the output and act as a moderator. These components are called different terms, such as supervisors or guardrails.
- System Configuration: During system configuration, the model could be fine-tuned to better control and neutralize potentially dangerous inputs.
Where this fits in a TurboPentest engagement
This weakness is not covered by the automated black-box pentest. IntegSec pentesters cover it in a manual engagement.
Frequently asked questions
What is CWE-1427?
The product uses externally-provided data to build prompts provided to large language models (LLMs), but the way these prompts are constructed causes the LLM to fail to distinguish between user-supplied inputs and developer provided system directives.
How do you find Improper Neutralization of Input Used for LLM Prompting?
Improper Neutralization of Input Used for LLM Prompting is typically found by tracing untrusted input from where it enters the system to the point where it is used without the check or neutralization this weakness describes, combining manual code review with dynamic testing.
What is the impact of CWE-1427?
Execute Unauthorized Code or Commands, Varies by Context: The consequences are entirely contextual, depending on the system that the model is integrated into. For example, the consequence could include output that would not have been desired by the model designer, such as using racial slurs. On the other hand, if the output is attached to a code interpreter, remote code execution (RCE) could result. Read Application Data: An attacker might be able to extract sensitive information from the model. Modify Application Data, Execute Unauthorized Code or Commands: The extent to which integrity can be impacted is dependent on the LLM application use case. Read Application Data, Modify Application Data, Gain Privileges or Assume Identity: The extent to which access control can be impacted is dependent on the LLM application use case.
Does TurboPentest test for Improper Neutralization of Input Used for LLM Prompting?
This weakness is not covered by the automated black-box pentest. IntegSec pentesters cover it in a manual engagement.
Related CWEs
Written and reviewed by
Michel Chamberland - Founder & CEO, IntegSec
CISSP, OSCP, OSCE, CEH, GIAC, CCSK · 20+ years in offensive security
Michel has spent 20+ years on offensive security teams including IBM X-Force Red and Trustwave SpiderLabs, leading penetration tests, red team engagements, and breach response for Fortune 500 customers. He is the founder of IntegSec and the architect of TurboPentest.
Find these issues before an attacker does
TurboPentest runs an agentic AI pentest against your target and reports findings with proof, from $99 per target.
Start a pentest