Forrester’s Panning For Gold: How To Evaluate Generative AI Capabilities In Security Tools delivers a reality check on the explosive growth of generative AI across the security market. While nearly every security vendor now markets AI-powered features, Forrester finds that many of these capabilities amount to “AI trash”. In other words, highly visible, heavily promoted features that add little operational value for security teams (we’re looking at you, AI chatbots).
Leading Forrester analysts argue that security leaders must move beyond surface-level feature comparisons and instead evaluate generative AI through three critical lenses: utility, trust, and cost. Without this disciplined approach, organizations risk paying more for tools that increase friction, introduce unreliable outputs, and create unpredictable spending models.
Key Findings
1. Most generative AI features deliver limited real-world utility
Forrester’s research shows that commonly marketed capabilities such as chatbots and automated incident summaries are among the lowest-utility AI features in security tools. These features often disrupt workflows, require unnecessary context switching, and fail to save time, making them nonstarters in already overloaded SOC environments.
In contrast, AI capabilities embedded directly into workflows (such as triage agents, investigation agents, and language translation) consistently demonstrate higher operational value.
2. Trust remains the biggest barrier to adoption
Generative AI’s nondeterministic nature makes trust difficult to establish. Outputs may vary between identical prompts, and hallucinations are still a risk. Forrester emphasizes that trust must be evaluated across three dimensions:
- Competence: Is the output accurate?
- Consistency: Are responses repeatable?
- Transparency: Can the tool explain how it reached its conclusion?
Vendors that rely primarily on end-user thumbs-up/thumbs-down feedback to validate AI outputs are flagged as particularly risky. Forrester strongly advises prioritizing vendors with robust internal testing frameworks, expert validation, and continuous evaluation mechanisms.
3. Pricing models are fragmented and unpredictable
There is no standard approach to pricing generative AI in security tools. Models range from fully bundled pricing to consumption-based, per-prompt, subscription, or hybrid approaches. Autonomous AI agents further complicate cost predictability, especially when AI operates without explicit user interaction. Forrester warns that as AI utilization increases (even as per-unit costs decline) overall spend is likely to rise, making cost transparency and governance essential.
What Separates “AI Gold” From “AI Noise”
According to Forrester, high-value generative AI features share several characteristics:
- They solve a concrete pain point at scale
- They are embedded directly into existing workflows
- They include continuous, vendor-owned validation and testing
- They provide clear explainability for every decision or recommendation
Managed detection and response (MDR) providers are highlighted as being uniquely positioned to deliver higher-quality AI, due to their ability to continuously validate outputs using in-house security experts.
Beyond AI Trash: Finding Real Value in Generative AI Capabilities
Generative AI in security is inevitable, but value is not. Security leaders should resist marketing hype and instead demand evidence of measurable utility, demonstrable trustworthiness, and transparent pricing. The organizations that apply this framework will be able to separate meaningful innovation from noise and realize productivity gains from generative AI rather than adding new risk and cost. Download the Forrester Report to get benchmark criteria for evaluating capabilities in the age of AI trash.
