Preserve the Vulnerability, Break the Alert: Testing the Robustness of CodeQL Taint Tracking with Natural Rewrites

Researchers tested the robustness of CodeQL's JavaScript taint tracking by using large language models to apply vulnerability-preserving rewrites to functions along reported flows. They found that three rewrites per flow silenced 26.2% of real npm vulnerabilities, with every silencing rewrite tracing to gaps at the library-modeling boundary. The results indicate that a disappearing alert after an ordinary-looking change is weak evidence of a fix, and robustness to natural rewrites should be part of how taint analyzers are evaluated. Open article →