Keynotes

Reliable Evaluation and Benchmarking of Machine Learning Models
Maura Pintor, University of Cagliari

Abstract: Rigorous evaluation of machine learning (ML) models is essential before deployment. To understand ML’s sensitivity to adversarial attacks and counter their effects, ML model designers craft worst-case perturbations and test them against their products. However, many of the proposed defenses have been shown to provide a false sense of security due to failures of the attacks rather than actual robustness. To this end, it’s important to set up trustworthy evaluation tools. In this talk, we will investigate existing benchmarking tools and we will highlight their issues, avoiding known mistakes to ensure high-quality evaluations. Moreover, current ML benchmarks are a first step, but they only offer an in-vitro evaluation. Addressing practical aspects like how predictions react to data drift over time and model updates is also important in real-world applications. For this reason, we will provide insights on analyzing how both performance and robustness evolve over time. Finally, we will discuss new testing and benchmarking guidelines to develop novel techniques to ensure models behave robustly in real-world scenarios, where not only are they the target of attacks, but they are also subject to data drifts and situations unseen in training.

Bio: Maura Pintor is an Assistant Professor at the Pattern Recognition and Applications Laboratory (PRALab), in the Dept. of Electrical and Electronic Engineering of the University of Cagliari (Italy). She received her PhD in Electronic and Computer Engineering from the University of Cagliari in 2022. Her main research interests revolve around providing trustworthy security evaluations of ML. She was a visiting student at the University of Tuebingen (2020), Germany, and at SCCH, Austria (2021). She is a reviewer for ACM CCS and Usenix Security, Area Chair for NeurIPS, and she is co-chair of the ACM Workshop on Artificial Intelligence and Security (AISec), co-located with ACM CCS.

Lessons Learned in Mobile Malware Detection with Machine Learning
Daniel Arp, TU Berlin

Abstract: Mobile malware poses a significant threat to the security and privacy of mobile device users. Over the past decade, numerous detection approaches utilizing machine learning techniques have been developed to overcome the limitations of traditional signature-based methods in identifying malware on smart devices. Despite showing promising results in laboratory settings, these methods have yet to be widely adopted in practical applications, prompting the question of what is hindering their triumph over conventional techniques. In this talk, we will review a decade of research on learning-based detection of mobile malware, share key lessons learned, and explore unresolved research challenges that still need to be addressed.

Bio: Daniel Arp is a postdoctoral researcher in the Machine Learning and Security Group at Technische Universität Berlin. Prior to this, he was a visiting researcher at University College London and King’s College London. He received his Ph.D. with honors in Computer Science from Technische Universität Braunschweig for his thesis “Efficient and Explainable Detection of Mobile Malware with Machine Learning”. He also holds a master’s degree in Computer Engineering from Technische Universität Berlin. His research interests include the development of learning-based methods to enhance the security and privacy of systems. His work has been published in prestigious venues and also recognized with multiple awards, including the IEEE Symposium on Security and Privacy Test of Time Award and the USENIX Security Distinguished Paper Award.

[Demo] The Many Facets of Windows Malware and the Importance of Multi-Technology Analysis
Simone Aonzo, Eurecom

Abstract: Windows malware predominantly manifests itself as Portable Executable (PE) files, but this format can hide a variety of sophisticated technologies, each of which presents unique challenges. In this presentation, we will first identify and explore these embedded technologies, and then look at a recent attack campaign known as PY#RATION. This campaign exemplifies the use of multiple technologies to obfuscate malware analysis. By reverse engineering a sample from this campaign, we will demonstrate the complex process and highlight the need for machine learning feature extraction to account for multiple technologies beyond just the PE file format.

Bio: Simone Aonzo is an Assistant Professor at EURECOM (France) in the Digital Security Department. He teaches and conducts research in malware analysis (covering both Windows and Android platforms), reverse engineering, phishing and mobile security. He is also interested in the human factors of security processes and has recently started to publish on this topic. He is passionate about finding and solving real-world security challenges and educating the next generation of security professionals. Finally, he is a member of the programme committee for the USENIX Security and RAID conferences.

Call for Papers

Important Dates

• Paper submission deadline: March 14 March 26, 2024; 11:59 PM (AoE, UTC-12)
• Acceptance notification: April 30, 2024; 11:59 PM (AoE, UTC-12)
• Camera ready due: May 15, 2024; 11:59 PM (AoE, UTC-12)
• Workshop date: July 12th, 2024

Overview

Malware research is a discipline of information security that aims to provide protection against unwanted and dangerous software. Since the mid-1980s, researchers in this area have been leading a technological arms race against creators of malware. Many ideas have been proposed, to varying degrees of effectiveness, from more traditional systems security and program analysis to the use of AI and Machine Learning. Nevertheless, with increased technological complexity and despite more sophisticated defenses, malware’s impact has grown, rather than shrunk. It appears that the defenders are continually reacting to yesterday’s threats, only to be surprised by today's minor variations.

The rise of Generative AI and Large Language models opens the path for new attackers strategies at reduced costs, and complicates the work for defenders.

This lack of robustness is most apparent in signature matching, where malware is represented by a characteristic substring. The fundamental limitation of this approach is its reliance on falsifiable evidence. Mutating the characteristic substring, i.e., falsifying the evidence, is effective in evading detection, and cheaper than discovering the substring in the first place. Unsurprisingly, the same limitation applies to malware detectors based on machine learning, as long as they rely on falsifiable features for decision-making. Robust malware features are necessary.

Furthermore, robust methods for malware classification and analysis are needed across the board to overcome phenomena including, but not limited to, concept drift (malware evolution), polymorphism, new malware families, new anti-analysis techniques, and adversarial machine learning, while supporting robust explanations. This workshop solicits work that aims to advance robust malware analysis, with the goal of creating long-term solutions to the threats of today’s digital environment. Potential research directions are malware detection, benchmark datasets, environments for malware arms race simulation, and exploring limitations of existing work, among others.

Topics of Interest

Topics of interest include (but are not limited to):

GenAI, Large Language Models, and Malware
Topics related to the use of LLMs for both attack generation and detection, including:

• The use of Generative AI in creative ways to thwart attackers or defenders
• New risks rising from Generative AI
• Using LLMs to explain malware behaviors
• Using LLMS for better malware analysis
• Using LLMs for modifying code automatically

Malware Analysis
Topics related to understanding the malicious actions exhibited by malware:

• Identification of malware behaviors
• Identification of code modules which implement specific behaviors
• Unsupervised behavior identification
• Machine Learning and AI for behavior identification
• Reliable parsing of file formats and program code
• De-obfuscation and de-cloaking of malware
• Robust static and dynamic code analysis
• Feature extraction in presence of adversaries
• Robust signature generation and matching

Malware Detection
Topics related to techniques for malware detection:

• Developing robust malware detection, malware family recognition, identification of novel malware families
• Network-based malware analysis
• Host-based malware analysis
• Malware datasets: publication of new datasets for detection, e.g., family recognition, new family identification, behavior identification, generalization ability

Malware Attribution
Topics exploring methods and techniques to confidently attribute a piece of malware to its creators:

• Binary and source-code attribution
• Adversarial attribution

Malware Arms Race
Topics related to the malware arms race:

• Virtual malware arms race environments and competition reports – automated bots of malware and detectors simultaneously attacking and defending networked hosts, adaptively co-evolving in their quest towards supremacy
• Automated countermeasures to malware anti-analysis techniques, e.g., packing, anti-debugging, anti-emulation
• Bypassing anti-malware (anti-virus), e.g., via problem-space adversarial modifications

Robustness Evaluations of Malware Analysis
Topics exploring the limitations of existing research:

• Experiments demonstrating the limitations in robustness of existing methods (for detection, unpacking, behavior analysis, etc.), datasets, defenses
• Machine learning-based malware analysis and adversarial machine learning
• Overcoming limitations – demonstrating methods resilient to, e.g., concept drift (malware evolution), polymorphism, new malware families, new anti-analysis techniques, or adversarial machine learning defenses

Submission Guidelines

We invite the following types of papers:

• Original Research papers, which are expected to be 8 pages, not exceeding 12 pages in double-column IEEE format including the references and appendices. This category of papers should describe original work that is not previously published or concurrently submitted elsewhere. In this category, we strongly encourage the submission of open-source software artifacts and emphasize the importance of reproducibility in results. We acknowledge that while these elements may not be mandatory for the acceptance of a paper, they significantly contribute to enhancing the overall merit of original research, serving as valuable complements to scientific novelty.
• Position or open-problem papers, of up to 6 pages, using the same template (title for this category must include the text "Position:” at the beginning). Position research papers aim at fostering discussion and collaboration by presenting preliminary research activities, work in progress and/or industrial innovations. Position research papers may summarize research results published elsewhere or outline new emerging ideas.

Submissions must be anonymous (double-blind review), and authors should refer to their previous work in the third-person. Submissions must not substantially overlap with papers that have been published or that are simultaneously submitted to a journal or conference with proceedings.

Papers must be typeset in LaTeX in A4 format (not "US Letter") using the IEEE conference proceeding template supplied by IEEE EuroS&P: eurosp2023-template.zip. Please do not use other IEEE templates.

Submissions must be in Portable Document Format (.pdf). Authors should pay special attention to unusual fonts, images, and figures that might create problems for reviewers. Your document should render correctly in Adobe Reader XI and when printed in black and white.

Accepted papers will be published in IEEE Xplore. One author of each accepted paper is required to attend the workshop and present the paper for it to be included in the proceedings. Committee members are not required to read the appendices, so the paper should be intelligible without them. Submissions must be in English and properly anonymized.

Submission Site

HotCrp Submission Website