CWE-1426: Improper Validation of Generative AI Output
The product invokes a generative AI/ML component whose behaviors and outputs cannot be directly controlled, but the product does not validate or insufficiently validates the outputs to ensure that they align with the intended security, content, or privacy policy.
How it's found
Improper Validation of Generative AI Output is typically found by tracing untrusted input from where it enters the system to the point where it is used without the check or neutralization this weakness describes, combining manual code review with dynamic testing.
Consequences
- Execute Unauthorized Code or Commands, Varies by Context: In an agent-oriented setting, output could be used to cause unpredictable agent invocation, i.e., to control or influence agents that might be invoked from the output. The impact varies depending on the access that is granted to the tools, such as creating a database or writing files.
Mitigations
- Architecture and Design: Since the output from a generative AI component (such as an LLM) cannot be trusted, ensure that it operates in an untrusted or non-privileged space.
- Operation: Use "semantic comparators," which are mechanisms that provide semantic comparison to identify objects that might appear different but are semantically similar.
- Operation: Use components that operate externally to the system to monitor the output and act as a moderator. These components are called different terms, such as supervisors or guardrails.
- Build and Compilation: During model training, use an appropriate variety of good and bad examples to guide preferred outputs.
Where this fits in a TurboPentest engagement
This weakness is not covered by the automated black-box pentest. IntegSec pentesters cover it in a manual engagement.
Frequently asked questions
What is CWE-1426?
The product invokes a generative AI/ML component whose behaviors and outputs cannot be directly controlled, but the product does not validate or insufficiently validates the outputs to ensure that they align with the intended security, content, or privacy policy.
How do you find Improper Validation of Generative AI Output?
Improper Validation of Generative AI Output is typically found by tracing untrusted input from where it enters the system to the point where it is used without the check or neutralization this weakness describes, combining manual code review with dynamic testing.
What is the impact of CWE-1426?
Execute Unauthorized Code or Commands, Varies by Context: In an agent-oriented setting, output could be used to cause unpredictable agent invocation, i.e., to control or influence agents that might be invoked from the output. The impact varies depending on the access that is granted to the tools, such as creating a database or writing files.
Does TurboPentest test for Improper Validation of Generative AI Output?
This weakness is not covered by the automated black-box pentest. IntegSec pentesters cover it in a manual engagement.
Related CWEs
Written and reviewed by
Michel Chamberland - Founder & CEO, IntegSec
CISSP, OSCP, OSCE, CEH, GIAC, CCSK · 20+ years in offensive security
Michel has spent 20+ years on offensive security teams including IBM X-Force Red and Trustwave SpiderLabs, leading penetration tests, red team engagements, and breach response for Fortune 500 customers. He is the founder of IntegSec and the architect of TurboPentest.
Find these issues before an attacker does
TurboPentest runs an agentic AI pentest against your target and reports findings with proof, from $99 per target.
Start a pentest