CWE-1434: Insecure Setting of Generative AI/ML Model Inference Parameters
The product has a component that relies on a generative AI/ML model configured with inference parameters that produce an unacceptably high rate of erroneous or unexpected outputs.
How it's found
Insecure Setting of Generative AI/ML Model Inference Parameters is typically found by tracing untrusted input from where it enters the system to the point where it is used without the check or neutralization this weakness describes, combining manual code review with dynamic testing.
Generative AI/ML models, such as those used for text generation, image synthesis, and other creative tasks, rely on inference parameters that control model behavior, such as temperature, Top P, and Top K. These parameters affect the model's internal decision-making processes, learning rate, and probability distributions. Incorrect settings can lead to unusual behavior such as text "hallucinations," unrealistic images, or failure to converge during training. The impact of such misconfigurations can compromise the integrity of the application. If the results are used in security-critical operations or decisions, then this could violate the intended security policy, i.e., introduce a vulnerability.
Vulnerable vs. safe
"model": "my-coding-model","context_window": 8192,"max_output_tokens": 4096,"temperature", 1.5,...{}..."temperature", 0.2,...{}Consequences
- Varies by Context, Unexpected State: The product can generate inaccurate, misleading, or nonsensical information.
- Alter Execution Logic, Unexpected State, Varies by Context: If outputs are used in critical decision-making processes, errors could be propagated to other systems or components.
Mitigations
- Implementation/System Configuration/Operation: Develop and adhere to robust parameter tuning processes that include extensive testing and validation.
- Implementation/System Configuration/Operation: Implement feedback mechanisms to continuously assess and adjust model performance.
- Documentation: Provide comprehensive documentation and guidelines for parameter settings to ensure consistent and accurate model behavior.
Where this fits in a TurboPentest engagement
This weakness is not covered by the automated black-box pentest. IntegSec pentesters cover it in a manual engagement.
Frequently asked questions
What is CWE-1434?
The product has a component that relies on a generative AI/ML model configured with inference parameters that produce an unacceptably high rate of erroneous or unexpected outputs.
How do you find Insecure Setting of Generative AI/ML Model Inference Parameters?
Insecure Setting of Generative AI/ML Model Inference Parameters is typically found by tracing untrusted input from where it enters the system to the point where it is used without the check or neutralization this weakness describes, combining manual code review with dynamic testing.
What is the impact of CWE-1434?
Varies by Context, Unexpected State: The product can generate inaccurate, misleading, or nonsensical information. Alter Execution Logic, Unexpected State, Varies by Context: If outputs are used in critical decision-making processes, errors could be propagated to other systems or components.
Does TurboPentest test for Insecure Setting of Generative AI/ML Model Inference Parameters?
This weakness is not covered by the automated black-box pentest. IntegSec pentesters cover it in a manual engagement.
Related CWEs
Written and reviewed by
Michel Chamberland - Founder & CEO, IntegSec
CISSP, OSCP, OSCE, CEH, GIAC, CCSK · 20+ years in offensive security
Michel has spent 20+ years on offensive security teams including IBM X-Force Red and Trustwave SpiderLabs, leading penetration tests, red team engagements, and breach response for Fortune 500 customers. He is the founder of IntegSec and the architect of TurboPentest.
Find these issues before an attacker does
TurboPentest runs an agentic AI pentest against your target and reports findings with proof, from $99 per target.
Start a pentest