Jailbreak-Zero: A Path to Pareto Optimal Red Teaming for Large Language Models
Kai Hu, Abhinav Aggarwal, Mehran Khodabandeh, David Zhang, Eric Hsin,
Matt Fredrikson
- CEO · Gray Swan AI
- Associate Professor · Carnegie Mellon University
Highlights
Publications
115 papers
2026
Jailbreak-Zero: A Path to Pareto Optimal Red Teaming for Large Language Models
Kai Hu, Abhinav Aggarwal, Mehran Khodabandeh, David Zhang, Eric Hsin, Li Chen, Ankit Jain, Matt Fredrikson,
Steal the Patch Size: Adversarially Manipulate Vision-Language Models
Kai Hu, Akash Bharadwaj, Weichen Yu, Matt Fredrikson
A New Framework for Cybersecurity Refusals in AI Agents
Eliot Krzysztof Jones, Mateusz Dziemian, Matt Fredrikson, J Zico Kolter
Multi-Rollout On-Policy Distillation via Peer Successes and Failures
Weichen Yu, Xiaomin Li, Yizhou Zhao, Xiaoze Liu, Ruowang Zhang, Haixin Wang, Yinyi Luo, Chen Henry Wu,
How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition
Mateusz Dziemian, Maxwell Lin, Xiaohan Fu, Micha Nowak, Nick Winter, Eliot Jones, Andy Zou, Lama Ahmad,
The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems
Xiaoze Liu, Ruowang Zhang, Weichen Yu, Siheng Xiong, Liu He, Feijie Wu, Hoin Jung, Matt Fredrikson,
SecCodePRM: A Process Reward Model for Code Security
Weichen Yu, Ravi Mangal, Yinyi Luo, Kai Hu, Jingxuan He, Corina S. Pasareanu, Matt Fredrikson
Sequential Pattern Recognition Attacks against Deployed Topic-Based Mechanisms
Saranya Vijayakumar, Norman Sadeh, Matt Fredrikson
ICISSP·Feb pdfPrivCode: When Code Generation Meets Differential Privacy
Zheng Liu, Chen Gong, Terry Yue Zhuo, Kecen Li, Weichen Yu, Matt Fredrikson, Tianhao Wang
LipNeXt: Scaling up Lipschitz-based Certified Robustness to Billion-parameter Models
Kai Hu, Haoqi Hu, Matt Fredrikson
The Trojan in the Vocabulary: Stealthy Sabotage of LLM Composition
Xiaoze Liu, Weichen Yu, Matt Fredrikson, Xiaoqian Wang, Jing Gao
arXiv·Jan pdf
2025
When the Same Coefficients Reach Different Places: Asymmetric Realizability in Transplanting Tokenizers across Large Language Models
Xiaoze Liu, Weichen Yu, Matt Fredrikson, Xiaoqian Wang, Jing Gao
Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing
Justin W. Lin, Eliot Krzysztof Jones, Donovan Julian Jasper, Ethan Jun-shen Ho, Anna Wu, Arnold Tianyi Yang, Neil Perry, Andy Zou,
Prototype-Integrated Representation Learning for Novelty Detection
Saranya Vijayakumar, Christos Faloutsos, Matt Fredrikson
TrustCom·Nov pdfD-REX: A Benchmark for Detecting Deceptive Reasoning in Large Language Models
Satyapriya Krishna, Andy Zou, Rahul Gupta, Eliot Krzysztof Jones, Nick Winter, Dan Hendrycks, J. Zico Kolter, Matt Fredrikson,
Evaluating Language Model Reasoning about Confidential Information
Dylan Sam, Alexander Robey, Andy Zou, Matt Fredrikson, J. Zico Kolter
Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition
Andy Zou, Maxwell Lin, Eliot Jones, Micha Nowak, Mateusz Dziemian, Nick Winter, Alexander Grattan, Valent Nathanael,
A Mixture of Linear Corrections Generates Secure Code
Weichen Yu, Ravi Mangal, Terry Zhuo, Matt Fredrikson, Corina S. Pasareanu
Adversarial Attacks on Robotic Vision Language Action Models
Eliot Krzysztof Jones, Alexander Robey, Andy Zou, Zachary Ravichandran, George J. Pappas, Hamed Hassani, Matt Fredrikson, J. Zico Kolter
Transferable Adversarial Attacks on Black-Box Vision-Language Models
Kai Hu, Weichen Yu, Li Zhang, Alexander Robey, Andy Zou, Chengming Xu, Haoqi Hu, Matt Fredrikson
Aligned LLMs Are Not Aligned Browser Agents
Priyanshu Kumar, Elaine Lau, Saranya Vijayakumar, Tu Trinh, Elaine T. Chang, Vaughn Robinson, Shuyan Zhou, Matt Fredrikson,
ICLR·May pdfAgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou,
LLM Whisperer: An Inconspicuous Attack to Bias LLM Responses
Weiran Lin, Anna Gerchanovsky, Omer Akgul, Lujo Bauer, Matt Fredrikson, Zifan Wang
Safety Pretraining: Toward the Next Generation of Safe AI
Pratyush Maini, Sachin Goyal, Dylan Sam, Alex Robey, Yash Savani, Yiding Jiang, Andy Zou, Matt Fredrikson,
2024
Is Your Text-to-Image Model Robust to Caption Noise?
Weichen Yu, Ziyan Yang, Shanchuan Lin, Qi Zhao, Jianyi Wang, Liangke Gui, Matt Fredrikson, Lu Jiang
Attacks and Defenses for Large Language Models on Coding Tasks
Chi Zhang, Zifan Wang, Ruoshi Zhao, Ravi Mangal, Matt Fredrikson, Limin Jia, Corina S. Pasareanu
Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents
Priyanshu Kumar, Elaine Lau, Saranya Vijayakumar, Tu Trinh, Scale Red Team, Elaine Chang, Vaughn Robinson, Sean Hendryx,
Improving Alignment and Robustness with Circuit Breakers
Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter,
VeriSplit: Secure and Practical Offloading of Machine Learning Inferences across IoT Devices
Han Zhang, Zifan Wang, Mihir Dhamankar, Matt Fredrikson, Yuvraj Agarwal
Sales Whisperer: A Human-Inconspicuous Attack on LLM Brand Recommendations
Weiran Lin, Anna Gerchanovsky, Omer Akgul, Lujo Bauer, Matt Fredrikson, Zifan Wang
arXiv·Jun pdfEfficient LLM Jailbreak via Adaptive Dense-to-sparse Constrained Optimization
Kai Hu, Weichen Yu, Yining Li, Kai Chen, Tianjun Yao, Xiang Li, Wenhe Liu, Lijun Yu,
A Recipe for Improved Certifiable Robustness
Kai Hu, Klas Leino, Zifan Wang, Matt Fredrikson
2023
Grounding Neural Inference with Satisfiability Modulo Theories
Zifan Wang, Saranya Vijayakumar, Kaiji Lu, Vijay Ganesh, Somesh Jha, Matt Fredrikson
NeurIPS·Dec pdfIs Certifying Robustness Still Worthwhile?
Ravi Mangal, Klas Leino, Zifan Wang, Kai Hu, Weicheng Yu, Corina Pasareanu, Anupam Datta, Matt Fredrikson
Representation Engineering: A Top-Down Approach to AI Transparency
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin,
Universal and Transferable Adversarial Attacks on Aligned Language Models
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, Matt Fredrikson
On the Perils of Cascading Robust Classifiers
Ravi Mangal, Zifan Wang, Chi Zhang, Klas Leino, Corina Pasareanu, Matt Fredrikson
Unlocking Deterministic Robustness Certification on ImageNet
Kai Hu, Andy Zou, Zifan Wang, Klas Leino, Matt Fredrikson
Learning Modulo Theories
Matt Fredrikson, Kaiji Lu, Saranya Vijayakumar, Somesh Jha, Vijay Ganesh, Zifan Wang
2022
Black-Box Audits for Group Distribution Shifts
Marc Juarez, Samuel Yeom, Matt Fredrikson
Degradation Attacks on Certifiably Robust Neural Networks
Klas Leino, Chi Zhang, Ravi Mangal, Matt Fredrikson, Bryan Parno, Corina S. Pasareanu
TMLR·Jul pdfEnhancing the Insertion of NOP Instructions to Obfuscate Malware via Deep Reinforcement Learning
Daniel Gibert, Matt Fredrikson, Carles Mateu, Jordi Planes, Quan Le
Self-Correcting Neural Networks For Safe Classification
Klas Leino, Aymeric Fromherz, Ravi Mangal, Matt Fredrikson, Bryan Parno, Corina Păsăreanu
Robust Models Are More Interpretable Because Attributions Look Normal
Zifan Wang, Matt Fredrikson, Anupam Datta
TEO: ephemeral ownership for IoT devices to provide granular data control
Han Zhang, Yuvraj Agarwal, Matt Fredrikson
MobiSys·Jun pdfProtecting user data through ephemeral ownership of IoT devices
Han Zhang, Yuvraj Agarwal, Matt Fredrikson
MobiSys·Jun pdfFaithful Explanations for Deep Graph Models
Zifan Wang, Yuhang Yao, Chaoran Zhang, Han Zhang, Youjie Kang, Carlee Joe-Wong, Matt Fredrikson, Anupam Datta
Selective Ensembles for Consistent Predictions
Emily Black, Klas Leino, Matt Fredrikson
Consistent Counterfactuals for Deep Models
Emily Black, Zifan Wang, Matt Fredrikson, Anupam Datta
Privacy-Preserving Case-Based Explanations: Enabling Visual Interpretability by Protecting Privacy
Helena Montenegro, Wilson Silva, Alex Gaudio, Matt Fredrikson, Asim Smailagic, Jaime S. Cardoso
ACCESS·Jan pdf
2021
Exploring Conceptual Soundness with TruLens
Anupam Datta, Matt Fredrikson, Klas Leino, Kaiji Lu, Shayak Sen, Ricardo Shih, Zifan Wang
NeurIPS·Dec pdfMachine Learning Explainability and Robustness: Connected at the Hip
Anupam Datta, Matt Fredrikson, Klas Leino, Kaiji Lu, Shayak Sen, Zifan Wang
KDD·Aug pdfSelf-Repairing Neural Networks: Provable Safety for Deep Networks via Dynamic Repair
Klas Leino, Aymeric Fromherz, Ravi Mangal, Matt Fredrikson, Bryan Parno, Corina S. Pasareanu
arXiv·Jul pdfRelaxing Local Robustness
Klas Leino, Matt Fredrikson
Automating Audit with Policy Inference
Abhishek Bichhawat, Matt Fredrikson, Jean Yang
CSF·Jun pdfFast Geometric Projections for Local Robustness Certification
Aymeric Fromherz, Klas Leino, Matt Fredrikson, Bryan Parno, Corina Păsăreanu
The Design of the User Interfaces for Privacy Enhancements for Android
Jason I. Hong, Yuvraj Agarwal, Matt Fredrikson, Mike Czapik, Shawn Hanna, Swarup Sahoo, Judy Chun, Won-Woo Chung,
Boundary Attributions Provide Normal (Vector) Explanations
Zifan Wang, Matt Fredrikson, Anupam Datta
Leave-one-out Unfairness
Emily Black, Matt Fredrikson
Globally-Robust Neural Networks
Klas Leino, Zifan Wang, Matt Fredrikson
Capture: Centralized Library Management for Heterogeneous IoT Devices
Han Zhang, Abhijith Anilkumar, Matt Fredrikson, Yuvraj Agarwal
USENIX Security·Jan pdfNetter: Probabilistic, Stateful Network Models
Han Zhang, Chi Zhang, Arthur Azevedo de Amorim, Yuvraj Agarwal, Matt Fredrikson, Limin Jia
VMCAI·Jan pdf
2020
Contextual and Granular Policy Enforcement in Database-backed Applications
Abhishek Bichhawat, Matt Fredrikson, Jean Yang, Akash Trehan
Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference
Klas Leino, Matt Fredrikson
Reconciling noninterference and gradual typing
Arthur Azevedo de Amorim, Matt Fredrikson, Limin Jia
LICS·Jul pdfOverfitting, robustness, and malicious algorithms: A study of potential causes of privacy risk in machine learning
Samuel Yeom, Irene Giacomelli, Alan Menaged, Matt Fredrikson, Somesh Jha
JCS·Jul pdfSmoothed Geometry for Robust Attribution
Zifan Wang, Haofan Wang, Shakul Ramkumar, Matt Fredrikson, Piotr Mardziel, Anupam Datta
FlipTest: Fairness Testing via Optimal Transport
Emily Black, Samuel Yeom, Matt Fredrikson
Influence Paths for Characterizing Subject-Verb Number Agreement in LSTM Language Models
Kaiji Lu, Piotr Mardziel, Klas Leino, Matt Fredrikson, Anupam Datta
Learning Fair Representations for Kernel Models
Zilong Tan, Samuel Yeom, Matt Fredrikson, Ameet Talwalkar
Interpreting Interpretations: Organizing Attribution Methods by Criteria
Zifan Wang, Piotr Mardziel, Anupam Datta, Matt Fredrikson
Individual Fairness Revisited: Transferring Techniques from Adversarial Robustness
Samuel Yeom, Matt Fredrikson
2019
FlipTest: Fairness Auditing via Optimal Transport.
Emily Black, Samuel Yeom, Matt Fredrikson
Jun pdfFeature-Wise Bias Amplification
Klas Leino, Emily Black, Matt Fredrikson, Shayak Sen, Anupam Datta
2018
Why Are They Collecting My Data?: Inferring the Purposes of Network Traffic in Mobile Apps
Haojian Jin, Minyi Liu, Kevan Dodhia, Yuanchun Li, Gaurav Srivastava, Matthew Fredrikson, Yuvraj Agarwal, Jason I. Hong
IMWUT·Dec pdfESTRELA: Automated Policy Enforcement Across Remote APIs
Abhishek Bichhawat, Akash Trehan, Jean Yang, Matt Fredrikson
arXiv·Nov pdfEstrela: Automated, Contextual Enforcement of Data Flow Policies across APIs
Abhishek Bichhawat, Akash Trehan, Jean Yang, Matt Fredrikson
Nov pdfHunting for Discriminatory Proxies in Linear Regression Models
Samuel Yeom, Anupam Datta, Matt Fredrikson
Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, Somesh Jha
Quantitative underpinnings of secure, graceful degradation: poster
Ryan Wagner, David Garlan, Matt Fredrikson
HOTSOS·Apr pdfSupervising Feature Influence
Shayak Sen, Piotr Mardziel, Anupam Datta, Matthew Fredrikson
Influence-Directed Explanations for Deep Convolutional Networks
Klas Leino, Shayak Sen, Anupam Datta, Matt Fredrikson, Linyi Li
2017
Case Study: Explaining Diabetic Retinopathy Detection Deep CNNs via Integrated Gradients
Linyi Li, Matt Fredrikson, Shayak Sen, Anupam Datta
PrivacyStreams: Enabling Transparency in Personal Data Processing for Mobile Apps
Yuanchun Li, Fanglin Chen, Toby Jia-Jun Li, Yao Guo, Gang Huang, Matthew Fredrikson, Yuvraj Agarwal, Jason I. Hong
IMWUT·Sep pdfThe Unintended Consequences of Overfitting: Training Data Inference Attacks
Samuel Yeom, Matt Fredrikson, Somesh Jha
arXiv·Sep pdfPrivacyProxy: Leveraging Crowdsourcing and In Situ Traffic Analysis to Detect and Mitigate Information Leakage
Gaurav Srivastava, Kunal Bhuwalka, Swarup Kumar Sahoo, Saksham Chitkara, Kevin Ku, Matt Fredrikson, Jason Hong, Yuvraj Agarwal
Proxy Non-Discrimination in Data-Driven Systems
Anupam Datta, Matt Fredrikson, Gihyuk Ko, Piotr Mardziel, Shayak Sen
An Advanced Persistent Threat Exemplar
R. Harrison Wagner, Matt Fredrikson, David Garlan
Jul pdfUse Privacy in Data-Driven Systems
Anupam Datta, Matt Fredrikson, Gihyuk Ko, Piotr Mardziel, Shayak Sen
Verifying and Synthesizing Constant-Resource Implementations with Types
Van Chan Ngo, Mario Dehesa-Azuara, Matthew Fredrikson, Jan Hoffmann
2016
The Limitations of Deep Learning in Adversarial Settings
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, Ananthram Swami
A Methodology for Formalizing Model-Inversion Attacks
Xi Wu, Matthew Fredrikson, Somesh Jha, Jeffrey F. Naughton
CSF·Jun pdf
2015
Revisiting Differentially Private Regression: Lessons From Learning Theory and their Consequences
Xi Wu, Matthew Fredrikson, Wentao Wu, Somesh Jha, Jeffrey F. Naughton
Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
Matt Fredrikson, Somesh Jha, Thomas Ristenpart
CCS·Oct pdfSurreptitiously Weakening Cryptographic Systems
Bruce Schneier, Matthew Fredrikson, Tadayoshi Kohno, Thomas Ristenpart
IACR·Jan pdf
2014
MoRePriv: mobile OS support for application personalization and privacy
Drew Davidson, Matt Fredrikson, Benjamin Livshits
ACSAC·Dec pdfOn the Practical Exploitability of Dual EC in TLS Implementations
Stephen Checkoway, Ruben Niederhagen, Adam Everspaugh, Matthew Green, Tanja Lange, Thomas Ristenpart, Daniel J. Bernstein, Jake Maskiewicz,
USENIX Security·Aug pdfZØ: An Optimizing Distributing Zero-Knowledge Compiler
Matthew Fredrikson, Benjamin Livshits
USENIX Security·Aug pdfPrivacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing
Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon M. Lin, David Page, Thomas Ristenpart
USENIX Security·Aug pdfSatisfiability modulo counting: a new approach for analyzing privacy properties
Matthew Fredrikson, Somesh Jha
CSL·Jul pdfThe Future of Crypto: Getting from Here to Guarantees
Daniel J. Bernstein, Matt Blaze, Tanja Lange, Tanzirul Azim, Shuo Chen, Jaeyeon Jung, Suman Nath, Rui Wang,
Jan
2012
Efficient Runtime Policy Enforcement Using Counterexample-Guided Abstraction Refinement
Matthew Fredrikson, Richard Joiner, Somesh Jha, Thomas W. Reps, Phillip A. Porras, Hassen Saïdi, Vinod Yegneswaran
CAV·Jul pdf
2011
Dynamic Behavior Matching: A Complexity Analysis and New Approximation Algorithms
Matthew Fredrikson, Mihai Christodorescu, Somesh Jha
CADE·Jul pdfRePriv: Re-Envisioning In-Browser Privacy
Matt Fredrikson, Ben Livshits
May pdfVerified Security for Browser Extensions
Arjun Guha, Matthew Fredrikson, Benjamin Livshits, Nikhil Swamy
S&P·May pdfRePriv: Re-imagining Content Personalization and In-browser Privacy
Matthew Fredrikson, Benjamin Livshits
S&P·May pdfEnd-to-End Software Diversification of Internet Services
Mihai Christodorescu, Matthew Fredrikson, Somesh Jha, Jonathon T. Giffin
AIS·Jan pdf
2010
Private Information Retrieval
Somesh Jha, Vitaly Shmatikov, Matt Fredrikson
Dec pdfCyber SA: Situational Awareness for Cyber Defense
Paul Barford, Marc Dacier, Thomas G. Dietterich, Matt Fredrikson, Jonathon T. Giffin, Sushil Jajodia, Somesh Jha, Jason H. Li,
AIS·Dec pdfAutomatic Generation of Remediation Procedures for Malware Infections
Roberto Paleari, Lorenzo Martignoni, Emanuele Passerini, Drew Davidson, Matt Fredrikson, Jonathon T. Giffin, Somesh Jha
USENIX Security·Aug pdfSynthesizing near-optimal malware specifications from suspicious behaviors
Somesh Jha, Matthew Fredrikson, Mihai Christodorescu, Reiner Sailer, Xifeng Yan
MALWARE·Jul pdfA Declarative Framework for Intrusion Analysis
Matt Fredrikson, Mihai Christodorescu, Jonathon T. Giffin, Somesh Jha
AIS·Jul pdfMining Large Information Networks by Graph Summarization
Chen Chen, Cindy Xide Lin, Matt Fredrikson, Mihai Christodorescu, Xifeng Yan, Jiawei Han
S&P·Jan pdf
2009
Mining Graph Patterns Efficiently via Randomized Summaries
Chen Chen, Cindy Xide Lin, Matt Fredrikson, Mihai Christodorescu, Xifeng Yan, Jiawei Han
VLDB·Aug pdf
2008
A Layered Architecture for Detecting Malicious Behaviors
Lorenzo Martignoni, Elizabeth Stinson, Matt Fredrikson, Somesh Jha, John C. Mitchell
RAID·Sep pdf
Students & postdocs
Current
- Kai Hu
- Saranya Vijaykumar
- Weichen Yu
Former
- Emily Black
- Klas Leino
- Zifan Wang
- Ryan Wagner
- Andy Zou
- Samuel Yeom
- Han Zhang
- Arthur Azevedo de Amorim· postdoc
- Abhishek Bichhawat· postdoc