Goodbye false sense of AI model security - hello, Dynamic Model Scanning
Machine learning models are a foundational element of the AI development and deployment process. But how do data scientists - and the security teams that support them - understand the risks of deploying third-party open source models, their vulnerabilities and if attackers have embedded malicious payloads that only run when the models are deployed?
The answer typically is that third party scanners are used to detect vulnerabilities, and to evaluate the risks of supply chain attacks, such as backdoors and what is referred to as serialization attacks - executing injected malicious code when a model is moved from a data scientist’s laptop or is put into production. But does the existing approach of searching for known malicious patterns and methods of injecting payloads into the model files actually identify adversarial and supply chain risks, or zero day exploits?
Analysis by our dedicated research team at Aim Labs demonstrates the answer is only a very partial yes - and these scanning approaches can lead to a false sense of security. The current generation of scanners are limited to performing pattern matching for known vulnerabilities or known attack methods for a subset of packages and functions, and are open to a range of attacker bypass techniques.
Aim Labs has developed a novel, dynamic approach to detect risks at load and inference time based on a model’s actual behavior. Rather than exhaustively predict known good behavior for a complex and extensive set of assembly-level functions and function interactions, this approach establishes a baseline and then flags potential risks by directly observing if the model is executing actions inconsistent with expected behavior.
Aim Labs head of research Itay Ravia will also be presenting on this research at BlackHat on August 6th at 4:20 PM in a briefing entitled: Smashing Model Scanners - Advanced Bypass Techniques and a Novel Detection Approach
The innovative research produced by Aim Labs on model scanning reflects the team’s mission to advance the security community’s understanding of AI threats. Aim Labs was responsible for the groundbreaking disclosure of the EchoLeak zero-click context injection leak vulnerability in M365 Copilot.
Why is model scanning important?
Models, both closed and open source, are a foundational component of how AI custom apps are built. As developers set out to address use cases with AI, they experiment and test the utility and applicability of models - including publicly available third-party models from Hugging Face and GitHub repositories. These services use third party tools to perform scans for models that are listed. But, with over 1.8 million models listed on HuggingFace alone, it’s a challenge to keep pace.
Security, governance and responsible AI teams usually require that models are scanned and discovered vulnerabilities addressed before being moved into production. Model scanning is an umbrella term for products designed to detect compromises made to a model in order to run arbitrary code, exfiltrate sensitive data or perform any other unauthorized action as part of the model’s loading, training or inference.
However, model scanning results based on static analysis alone can result in the model being moved into production without a clear assessment of what risks actually exist, since the malicious payload could have already been embedded in the model when downloaded from Hugging Face.
How existing scans work and their limitations
AI models are almost always implemented in Python, largely due to the language's ease of use, as well as extensive libraries for machine learning and data science like PyTorch, TensorFlow, Pandas and Keras for deep learning and neural network development.
When model files are downloaded or moved, they generally need to be serialized. In many cases, serialization involves injecting Python code to rebuild the model into memory, including the weights and architecture. Data scientists use the process of serialization to convert PyTorch objects and Tensorflow models into a format to transmit the data and then recreate the object when needed using the reverse process of deserialization. There are different formats for the serialization of data, such as JSON, HDF5,Python’s Pickle and SafeTensors for different purposes.
The general existing process for performing scans involves statically evaluating the payload code of the model to determine which Python function calls are going to be made when the model loads through the deserialization process.
The scanner providers maintain a denylist for known bad or malicious Python function calls based on known methods attackers can use to inject malicious payloads into the model files, and flag the functions if there is a match.
The challenge here is there are thousands of libraries and thousands of functions within each. There are vulnerabilities in the functions themselves, and attackers will inevitably engineer bypass techniques to known scanning methodologies.
This way leads to danger - the AV analogy
While not to appear overwhelmingly alarmist, the industry analogy that is apt for the direction of AI model scanning is the transition in the anti-virus category with the rise of malware. Anti-virus engines historically perform signature matching for known bad code packages and executables. Malware authors exploited a blind spot in AV detection, allowing them to embed code or execute actions after the AV scan did not identify the signatures of any known malicious code.
Still, there are clear parallels.
The current generation of model scanners rely on what we refer to as static scanning for known, documented risks and vulnerabilities - not how the model behaves once it’s been loaded into memory, and is interacting with the entirety of the AI stack. Static analysis alone lacks the algorithmic capability to accurately emulate the actual loading process, and evaluate how the model functions. Consequently, relying solely on static analysis may create a false sense of security when using models from unknown third-party sources.
Equally, however, just as anti-virus engines have evolved to remain relevant for defense-in-depth strategies, we see static scanning continue to play a role for AI security - in combination with dynamic scanning. Dynamic scanners have the capability to generate insights into novel attack methods, while existing scanners should be understood more as "known malware hashes" detection engines.
What is dynamic scanning?
In order to explain Aim's novel approach, it’s important to firstly understand how models are deployed.
As we mentioned, there are different formats for the serialization of data, such as JSON, HDF5, and Python’s Pickle, for different purposes. The vast majority of models utilize Pickle for assembly during serialization.
Scanners that operate statically evaluate Python objects at serialization for known vulnerabilities by reading the content of the file one byte at a time, not the potentially hundreds of actions by the assembly language. Secondly, it’s non-trivial for static scanners to emulate how code and files will function in production.
Dynamic scanning, by contrast, can identify those “hard to find” risks that may even use documented bypass techniques to evade detection. By observing actual execution during model loading, we eliminate the guesswork required by static scanning.
The question is how to achieve this outcome when model files can be multi-gig, and encapsulate complex sets operations? The Aim Labs team worked on defining the scope and observation methodology to optimize for insight and overhead. Firstly, model loading and inference are operations that make a well-defined and expected set of software library and system calls. This simplifies profiling benign models and yields a verifier for models - serving as a baseline for identifying malicious behavior.
Tackling supply chain risk means less risky deployment
Aim’s dynamic scanner - built using insights from Aim Labs research - in tandem with the static scanner, provides a much more comprehensive understanding of a model’s real risks and more actionable guidance for data scientists and developers for safe model deployment.
The rise of third-party, open source models and the popularity of public model repositories like Hugging Face and GitHub Marketplace point to the value they provide for data scientists and development teams to experiment and explore which models are best suited for their applications, datasets and use cases.
However, once these models are downloaded to a data scientist's laptop for finetuning or experimentation, and then potentially moved into production, attackers can exploit vulnerabilities, inject malicious code, create a backdoor, or manipulate the model through a serialization attack.
The research done by Aim Labs for prospectively inspecting model behavior is already integrated into the Aim AI Security Posture Management (AI-SPM) model scanning module to better identify these serialization attacks, as well as potentially uncover zero day exploits.
Combining the ability to both identify known vulnerabilities and detect zero day attacks, the Aim scanner module provides users with a comprehensive view of vulnerabilities and runtime risks. Model scanning is, in turn, a central element in Security Posture Management for the stack used to develop, test and deploy custom AI applications.
Performed on demand or as a step within existing workflows, Aim’s model scanner enables the level of assurance (if risks are appropriately remediated) necessary to move applications into production.
Comprehensive approach to model risk detection and remediation
Aim’s dynamic approach in concert with Aim’s static scanning for vulnerabilities provides a practical way to validate models and identify supply chain risks - creating the foundation for improvements in how models are deployed to production. Static model analyzers alone that parse code for signatures are an important yet insufficient tool in the domain of model scanning. The combination of static scanners with dynamic approaches provides a more complete protection against supply chain attacks based on model files.
