Adversa AI is a leading provider of Security and Safety solutions for AI


Adversa

Authors: (1) Maria Rigaki, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and maria.rigaki@fel.cvut.cz; (2) Sebastian Garcia, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and sebastian.garcia@agents.fel.cvut.cz. Table of Links Abstract & Introduction Threat Model Background and Related Work Methodology Experiments Setup Results Discussion Conclusion, Acknowledgments, and References Appendix Appendix A. Hyper-parameter Tuning The search space for the PPO hyper-parameters: – gamma: 0.01 - 0.75 – max grad norm: 0.3 - 5.0 – learning rate: 0.001 - 0.1 – activation function: ReLU or Tanh – neural network size: small or medium Selected parameters: gamma=0.854, learning rate=0.00138, max grad norm=0.4284, activation function=Tanh, small network size (2 layers with 64 units each). The search space for the LGB surrogate training hyper-parameters: – alpha: 1 - 1,000 – num boosting rounds: 100-2,000 – learning rate: 0.001 - 0.1 – num leaves: 128 - 2,048 – max depth: 5 - 16 – min child samples: 5 - 100 – feature fraction: 0.4 - 1.0 Hyper-parameter settings for the training of each LGB surrogate Table 4. This paper is under CC BY-NC-SA 4.0 DEED license. available on arxiv

Adversarial Malware Creation with Model-Based Reinforcement Learning: Appendix

Authors: (1) Maria Rigaki, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and maria.rigaki@fel.cvut.cz; (2) Sebastian Garcia, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and sebastian.garcia@agents.fel.cvut.cz. Table of Links Abstract & Introduction Threat Model Background and Related Work Methodology Experiments Setup Results Discussion Conclusion, Acknowledgments, and References Appendix 8 Conclusions By employing model-based reinforcement learning, MEME generates adversarial malware samples that successfully evade antivirus systems and train a surrogate model mimicking the target classifier accurately. Our experiments show that MEME surpasses existing methods in evasion rate, suggesting its potential for various applications, such as testing model robustness and enhancing cyber security against advanced persistent threats. Future work may involve exploring ensemble surrogates and other optimizations to enhance MEME’s performance further. Acknowledgments The authors acknowledge support from the Strategic Support for the Development of Security Research in the Czech Republic 2019–2025 (IMPAKT 1) program, by the Ministry of the Interior of the Czech Republic under No. VJ02010020 – AI-Dojo: Multi-agent testbed for the research and testing of AI-driven cyber security technologies. The authors acknowledge the support of NVIDIA Corporation with the donation of a Titan V GPU used for this research. References 1. Akiba, T., Sano, S., Yanase, T., Ohta, T., Koyama, M.: Optuna: A Next-generation Hyperparameter Optimization Framework. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 2623–2631. KDD ’19, Association for Computing Machinery, New York, NY, USA (Jul 2019). https://doi.org/10.1145/3292500.3330701 2. Anderson, H.S., Kharkar, A., Filar, B., Evans, D., Roth, P.: Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learning (Jan 2018). https://doi.org/10.48550/arXiv.1801.08917, arXiv:1801.08917 [cs] 3. Anderson, H.S., Roth, P.: EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models (Apr 2018). https://doi.org/10.48550/arXiv.1804.04637, arXiv:1804.04637 [cs] 4. Bergstra, J., Bardenet, R., Bengio, Y., K´egl, B.: Algorithms for Hyper-Parameter Optimization. In: Advances in Neural Information Processing Systems. vol. 24. Curran Associates, Inc. (2011) 5. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: OpenAI Gym (Jun 2016). https://doi.org/10.48550/arXiv.1606.01540, arXiv:1606.01540 [cs] 6. Ceschin, F., Botacin, M., Gomes, H.M., Oliveira, L.S., Gr´egio, A.: Shallow Security: on the Creation of Adversarial Variants to Evade Machine Learning-Based Malware Detectors. In: Proceedings of the 3rd Reversing and Offensive-oriented Trends Symposium. pp. 1–9. ROOTS’19, Association for Computing Machinery, New York, NY, USA (Feb 2020). https://doi.org/10.1145/3375894.3375898 7. Chandrasekaran, V., Chaudhuri, K., Giacomelli, I., Jha, S., Yan, S.: Exploring connections between active learning and model extraction. In: Proceedings of the 29th USENIX Conference on Security Symposium. pp. 1309–1326. SEC’20, USENIX Association, USA (Aug 2020) 8. Correia-Silva, J.R., Berriel, R.F., Badue, C., de Souza, A.F., Oliveira-Santos, T.: Copycat CNN: Stealing Knowledge by Persuading Confession with Random Non-Labeled Data. In: 2018 International Joint Conference on Neural Networks (IJCNN). pp. 1–8 (Jul 2018). https://doi.org/10.1109/IJCNN.2018.8489592, iSSN: 2161-4407 9. Demetrio, L., Biggio, B., Lagorio, G., Roli, F., Armando, A.: FunctionalityPreserving Black-Box Optimization of Adversarial Windows Malware. IEEE Transactions on Information Forensics and Security 16, 3469–3478 (2021). https://doi.org/10.1109/TIFS.2021.3082330, conference Name: IEEE Transactions on Information Forensics and Security 10. Demetrio, L., Coull, S.E., Biggio, B., Lagorio, G., Armando, A., Roli, F.: Adversarial exemples: A survey and experimental evaluation of practical attacks on machine learning for windows malware detection. ACM Transactions on Privacy and Security (TOPS) 24(4), 1–31 (2021), publisher: ACM New York, NY, USA 11. Dowling, S., Schukat, M., Barrett, E.: Using Reinforcement Learning to Conceal Honeypot Functionality. In: Brefeld, U., Curry, E., Daly, E., MacNamee, B., Marascu, A., Pinelli, F., Berlingerio, M., Hurley, N. (eds.) Machine Learning and Knowledge Discovery in Databases. pp. 341–355. Lecture Notes in Computer Science, Springer International Publishing, Cham (2019). https://doi.org/10.1007/978-3-030-10997-4 21 12. Fang, Y., Zeng, Y., Li, B., Liu, L., Zhang, L.: DeepDetectNet vs RLAttackNet: An adversarial method to improve deep learning-based static malware detection model. PLOS ONE 15(4), e0231626 (Apr 2020). https://doi.org/10.1371/journal.pone.0231626, publisher: Public Library of Science 13. Fang, Z., Wang, J., Geng, J., Kan, X.: Feature Selection for Malware Detection Based on Reinforcement Learning. IEEE Access 7, 176177–176187 (2019). https://doi.org/10.1109/ACCESS.2019.2957429, conference Name: IEEE Access 14. Fang, Z., Wang, J., Li, B., Wu, S., Zhou, Y., Huang, H.: Evading Anti-Malware Engines With Deep Reinforcement Learning. IEEE Access 7, 48867–48879 (2019). https://doi.org/10.1109/ACCESS.2019.2908033, conference Name: IEEE Access 15. Harang, R., Rudd, E.M.: SOREL-20M: A Large Scale Benchmark Dataset for Malicious PE Detection. arXiv:2012.07634 [cs] (Dec 2020), arXiv: 2012.07634 16. Hu, W., Tan, Y.: Generating Adversarial Malware Examples for Black-Box Attacks Based on GAN. In: Tan, Y., Shi, Y. (eds.) Data Mining and Big Data. pp. 409–423. Communications in Computer and Information Science, Springer Nature, Singapore (2022). https://doi.org/10.1007/978-981-19-8991-9 29 17. Huang, L., Zhu, Q.: Adaptive Honeypot Engagement Through Reinforcement Learning of Semi Markov Decision Processes. In: Alpcan, T., Vorobeychik, Y., Baras, J.S., D´an, G. (eds.) Decision and Game Theory for Security. pp. 196– 216. Lecture Notes in Computer Science, Springer International Publishing, Cham (2019). https://doi.org/10.1007/978-3-030-32430-8 13 18. Institute, A.T.: AV-ATLAS - Malware & PUA (2023), https://portal.av-atlas.org/malware 19. Jagielski, M., Carlini, N., Berthelot, D., Kurakin, A., Papernot, N.: High Accuracy and High Fidelity Extraction of Neural Networks. pp. 1345–1362 (2020) 20. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., Liu, T.Y.: LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In: Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc. (2017) 21. Kurutach, T., Clavera, I., Duan, Y., Tamar, A., Abbeel, P.: Model-Ensemble TrustRegion Policy Optimization. In: International Conference on Learning Representations (2018) 22. Labaca-Castro, R., Franz, S., Rodosek, G.D.: AIMED-RL: Exploring Adversarial Malware Examples with Reinforcement Learning. In: Dong, Y., Kourtellis, N., Hammer, B., Lozano, J.A. (eds.) Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track. pp. 37–52. Lecture Notes in Computer Science, Springer International Publishing, Cham (2021). https://doi.org/10.1007/978-3-030-86514-6 3 23. Li, X., Li, Q.: An IRL-based malware adversarial generation method to evade anti-malware engines. Computers & Security 104, 102118 (May 2021). https://doi.org/10.1016/j.cose.2020.102118 24. Ling, X., Wu, L., Zhang, J., Qu, Z., Deng, W., Chen, X., Qian, Y., Wu, C., Ji, S., Luo, T., Wu, J., Wu, Y.: Adversarial attacks against Windows PE malware detection: A survey of the state-of-the-art. Computers & Security 128, 103134 (May 2023). https://doi.org/10.1016/j.cose.2023.103134 25. Lundberg, S.M., Lee, S.I.: A Unified Approach to Interpreting Model Predictions. In: Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc. (2017) 26. Nguyen, T.T., Reddi, V.J.: Deep Reinforcement Learning for Cyber Security. IEEE Transactions on Neural Networks and Learning Systems pp. 1–17 (2021). https://doi.org/10.1109/TNNLS.2021.3121870, conference Name: IEEE Transactions on Neural Networks and Learning Systems 27. Orekondy, T., Schiele, B., Fritz, M.: Knockoff Nets: Stealing Functionality of BlackBox Models. pp. 4954–4963 (2019) 28. Pal, S., Gupta, Y., Shukla, A., Kanade, A., Shevade, S., Ganapathy, V.: ActiveThief: Model Extraction Using Active Learning and Unannotated Public Data. Proceedings of the AAAI Conference on Artificial Intelligence 34(01), 865–872 (Apr 2020). https://doi.org/10.1609/aaai.v34i01.5432, number: 01 29. Phan, T.D., Duc Luong, T., Hoang Quoc An, N., Nguyen Huu, Q., Nghi, H.K., Pham, V.H.: Leveraging Reinforcement Learning and Generative Adversarial Networks to Craft Mutants of Windows Malware against Black-box Malware Detectors. In: Proceedings of the 11th International Symposium on Information and Communication Technology. pp. 31–38. SoICT ’22, Association for Computing Machinery, New York, NY, USA (Dec 2022) 30. Quertier, T., Marais, B., Morucci, S., Fournel, B.: MERLIN – Malware Evasion with Reinforcement LearnINg (Mar 2022), arXiv:2203.12980 [cs] 31. Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., Dormann, N.: StableBaselines3: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research 22(268), 1–8 (2021) 32. Rigaki, M., Garcia, S.: Stealing and evading malware classifiers and antivirus at low false positive conditions. Computers & Security 129, 103192 (Jun 2023). https://doi.org/10.1016/j.cose.2023.103192 33. Rosenberg, I., Meir, S., Berrebi, J., Gordon, I., Sicard, G., Omid David, E.: Generating End-to-End Adversarial Examples for Malware Classifiers Using Explainability. In: 2020 International Joint Conference on Neural Networks (IJCNN). pp. 1–10 (Jul 2020). https://doi.org/10.1109/IJCNN48605.2020.9207168, iSSN: 2161-4407 34. Sanyal, S., Addepalli, S., Babu, R.V.: Towards Data-Free Model Stealing in a Hard Label Setting. pp. 15284–15293 (2022) 35. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017) 36. Security.org, T.: 2023 Antivirus Market Annual Report (Feb 2023), https://www.security.org/antivirus/antivirus-consumer-report-annual/ 37. Severi, G., Meyer, J., Coull, S., Oprea, A.: Explanation-Guided Backdoor Poisoning Attacks Against Malware Classifiers. In: 30th USENIX Security Symposium (USENIX Security 21). pp. 1487–1504. USENIX Association (2021) 38. Song, W., Li, X., Afroz, S., Garg, D., Kuznetsov, D., Yin, H.: MAB-Malware: A Reinforcement Learning Framework for Blackbox Generation of Adversarial Malware. In: Proceedings of the 2022 ACM on Asia Conference on Computer and Communications Security. pp. 990–1003. ASIA CCS ’22, Association for Computing Machinery, New York, NY, USA (May 2022). https://doi.org/10.1145/3488932.3497768 39. Sussman, B.: New Malware Is Born Every Minute (May 2023), https://blogs.blackberry.com/en/2023/05/new-malware-born-every-minute 40. Sutton, R.S., Barto, A.G.: Reinforcement Learning, second edition: An Introduction. MIT Press (Nov 2018) 41. Total, V.: VirusTotal - Stats, https://www.virustotal.com/gui/stats 42. Uprety, A., Rawat, D.B.: Reinforcement Learning for IoT Security: A Comprehensive Survey. IEEE Internet of Things Journal 8(11), 8693–8706 (Jun 2021). https://doi.org/10.1109/JIOT.2020.3040957, conference Name: IEEE Internet of Things Journal 43. Wu, C., Shi, J., Yang, Y., Li, W.: Enhancing Machine Learning Based Malware Detection Model by Reinforcement Learning. In: Proceedings of the 8th International Conference on Communication and Network Security. pp. 74–78. ICCNS 2018, Association for Computing Machinery, New York, NY, USA (Nov 2018). https://doi.org/10.1145/3290480.3290494 44. Zolotukhin, M., Kumar, S., H¨am¨al¨ainen, T.: Reinforcement Learning for Attack Mitigation in SDN-enabled Networks. In: 2020 6th IEEE Conference on Network Softwarization (NetSoft). pp. 282–286 (Jun 2020). https://doi.org/10.1109/NetSoft48620.2020.9165383 45. Sembera, V., Paquet-Clouston, M., Garcia, S., Erquiaga, M.J.: Cybercrime Spe- ˇ cialization: An Expos´e of a Malicious Android Obfuscation-as-a-Service. In: 2021 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW. pp. 213–236 (2021). https://doi.org/10.1109/EuroSPW54576.2021.00029 This paper is under CC BY-NC-SA 4.0 DEED license. available on arxiv

Enhancing Cybersecurity with MEME: Reinforcement Learning for Adversarial Malware Evasion

Authors: (1) Maria Rigaki, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and maria.rigaki@fel.cvut.cz; (2) Sebastian Garcia, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and sebastian.garcia@agents.fel.cvut.cz. Table of Links Abstract & Introduction Threat Model Background and Related Work Methodology Experiments Setup Results Discussion Conclusion, Acknowledgments, and References Appendix 5 Experiments Setup To evaluate MEME, several experiments were conducted in different configurations. First, there was a selection of four different malware detection solutions as targets to evade. Second, there was a comparison of MEME and four other evasion techniques on these four targets. 5.1 Targets The selection of targets was made to include three highly cited malware detection models together with a real implementation of a popular free Antivirus solution. 1. Ember. A LightGBM [20] model that was released as part of the Ember dataset [3] which was used for training that same model. The decision threshold was set to 0.8336, which corresponds to a 1% false positive rate (FPR) on the Ember 2018 test set. 2. Sorel-LGB. A LightGBM model that was distributed as part of the Sorel20M [15] dataset, which was used for training that same model. The decision threshold was set to 0.5, which corresponds to a 0.2% false positive rate (FPR) on the Sorel-20M test set. 3. Sorel-FFNN. A feed-forward neural network (FFNN) that was also released as part of the Sorel 20M dataset and was trained using the same data. The decision threshold was set to 0.5, which corresponds to 0.6% false positive rate (FPR) on the Sorel-20M test set. 4. Microsoft Defender. An antivirus product that comes pre-installed with the Windows operating system. According to [36], it is the most used free antivirus product for personal computers. All tests were performed using a virtual machine (VM) running an updated version of the product. The VM had no internet connectivity during the binary file scanning. 5.2 Datasets Our experiments required the use of the following datasets: 1. Ember 2018. A dataset that consists of features extracted from one million Windows Portable Executable (PE) files [3]. The dataset is split into training, testing, and ”unlabeled sets”. The training set consists of 300,000 clean samples, 300,000 malicious samples, and 200,000 “unlabeled” samples. The so-called unlabeled part of the dataset was truly unlabeled in the first version of the dataset, however, in the 2018 release, the authors provided an avclass label for all malicious samples, including those in the unlabeled set. Each sample has 2,381 static features related to byte and entropy histograms, PE header information, strings, imports, data directories, etc. 2. Sorel-20M. The Sorel dataset [15] was released in 2020 and contains the extracted features of 20 million binary files (malicious and benign). The feature set used was the same as the one from the Ember dataset. 3. Malware Binary Files. In addition to the Ember features used for training the surrogate, we also obtained 1,000 malicious binary files whose hashes were part of Ember 2018 and we use them for generating the evasive malware with all the methods. 4. Benign Binary Files. All methods require a benign set of data from where they extract benign strings, sections, and other elements that are used for the binary modifications. The same set of 100 benign binaries were used in all experiments. The files were obtained from a Windows 10 virtual machine after installing known benign software. MEME created two versions of the Daux dataset to train the surrogate models. For the Ember and AV surrogates, Daux contains the unlabeled part of the Ember dataset. For the Sorel surrogates, Daux contains 200,000 samples from the Sorel-20M validation set. These datasets were chosen to create the surrogate because they were not used to train the corresponding targets. During the evaluation, a subset of the Sorel-20M test set was used to evaluate the performance of the Sorel surrogate models and the Ember test set was used to evaluate the Ember and AV models. The 1,000 malware binaries were split into training and test sets with a 70-30% ratio using five different seeds. The test sets were used to test all the methods, while the 700 binaries in the training set were used to train each of the RL policies for PPO and MEME (these were the binaries to which modifier actions were applied). 5.3 Adversarial Malware Generation Comparison With regard to the generation of adversarial malware, MEME is compared with four algorithms in total. Two baseline reinforcement learning algorithms that use the Malware-Gym environment: a random agent and an agent that learns a policy using the vanilla Proximal Policy Optimization (PPO) algorithm [35], and two state-of-the-art (SOTA) algorithms: MAB [38] and GAMMA [9]. The two SOTA algorithms were selected based on the fact that they are relatively recently released and the fact that they seem to perform well in the malware evasion task. In addition, their source code is available. The detailed setup used for each of the algorithms, as well as any modifications, are presented below: 1. Random Agent The random agent is the simplest baseline used in our experiments. It uses the Malware-Gym environment and randomly samples the next modification action from the available action space. The agent is evaluated in the test environments using the 300 test malware samples. 2. PPO This is an agent that uses the PPO [35] algorithm as implemented in the Stable-Baselines3 software package. The agent was trained for 2,048 steps on the malware training set and evaluated on the malware test set. To select the hyper-parameters related to PPO training, we used the Tree-structured Parzen Estimator (TPE) method [4] as implemented in the software package Optuna [1]. The TPE algorithm was executed with the Ember dataset, but the settings performed well in the other targets. The tuned hyper-parameters were γ, the learning rate, the maximum gradient norm, the activation function, and the neural network size for the actor and critic models. The search space of each parameter and the final values are presented in the Appendix. 3. MAB RL algorithm that treats the evasive malware generation as a multiarmed bandit problem2. It operates in two stages: evasion and minimization. It samples the action space, which includes generic actions (similar to Malware-Gym) and any successful evasive actions along with the specific modifiers, e.g., appending a specific benign section. MAB directly manipulates each binary without generating a learned policy. Therefore, all experiments were conducted directly on the malicious binary test set. 4. GAMMA An algorithm that injects benign binary sections into malicious PE files while preserving their functionality. It modifies features like section count, byte histograms, and strings, leaving features related to, e.g., certificates and debugging data unaffected. By employing genetic algorithms, GAMMA searches for optimal benign sections to reduce the target model’s confidence by minimizing the content and location of injected sections. The attack is implemented in the SecML library3. Though effective, it has a significantly longer runtime than other tested methods. The attack uses a restricted set of 30 available benign sections and a population size of 20. The λ parameter, impacting the injected data size, was set to 10−6. GAMMA directly operates on each binary and does not generate a learned policy. Hence, all experiments were conducted on the malicious binary test set. 5.4 MEME Experimental Setup The initial training steps n in Algorithm 1 were set to 1,024, and the total number of loops k was two. For evaluation, the test set of 300 malware binaries was used. The surrogate training steps m were set to 2,048 (step 6 of Algorithm 1). MEME utilizes PPO for training and updating the policy (πθ). The PPO settings remained the same as in the baseline experiments, enabling a comparison of the impact of using a surrogate model for additional PPO training. In total, there were a total of 2,048 queries to the target and 4,096 training steps using the surrogate environment. The surrogate was always a LightGBM model. Surrogate training involved two datasets: Daux from an external dataset (e.g., Ember 2018 or Sorel-20M) and Dsur generated during lines 3 and 7 of Algorithm 1. These datasets were mixed with a ratio α, a hyperparameter for LGB surrogate tuning. Other hyperparameters, such as the number of boosting trees, learning rate, tree depth, minimum child samples, and feature fraction, were also tuned separately for each target using TPE and Optuna. Appendix A provides the detailed search space and selected values. Surrogate models were evaluated using the respective target test sets, and a decision threshold matching target FPR levels was calculated. For the AV target without a representative dataset, the surrogate’s decision threshold was set to 0.5. 5.5 General Experiment Settings All the algorithms were tested under a common set of constraints. The maximum allowed modifications to a binary file were set to 15 for the RL-based algorithms. Similarly, for GAMMA, the number of iterations was set to 15; for MAB, the number of ”pulls” was also 15. The second constraint was to set the maximum running time for all experiments to 4 hours. For MAB and GAMMA, this setting means that the algorithms must manage to handle as many malicious binaries as possible in that time. At the same time, for PPO and MEME, this time included both the policy training time and the evaluation time. For PPO and MEME, we set the query budget to 2,048. This budget does not include the final evaluation queries on the test set. MAB and GAMMA were not constrained in the total number of queries because it is not supported by the respective frameworks. Finally, all experiments were run with five different seeds. The random seeds controlled the split of the 1,000 malicious binaries into train and test, and therefore, all methods were tested in the same files. This paper is under CC BY-NC-SA 4.0 DEED license. available on arxiv

MEME Algorithm's Impact on Adversarial Malware Generation and Model Evasion

Authors:
(1) Maria Rigaki, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and maria.rigaki@fel.cvut.cz;
(2) Sebastian Garcia, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and sebastian.garcia@agents.fel.cvut.cz. Table of Links Abstract & Introduction Threat Model Background and Related Work Methodology Experiments Setup Results Discussion Conclusion, Acknowledgments, and References Appendix Abstract Due to the proliferation of malware, defenders are increasingly turning to automation and machine learning as part of the malware detection toolchain. However, machine learning models are susceptible to adversarial attacks, requiring the testing of model and product robustness. Meanwhile, attackers also seek to automate malware generation and evasion of antivirus systems, and defenders try to gain insight into their methods. This work proposes a new algorithm that combines Malware Evasion and Model Extraction (MEME) attacks. MEME uses model-based reinforcement learning to adversarially modify Windows executable binary samples while simultaneously training a surrogate model with a high agreement with the target model to evade. To evaluate this method, we compare it with two state-of-the-art attacks in adversarial malware creation, using three well-known published models and one antivirus product as targets. Results show that MEME outperforms the state-of-the-art methods in terms of evasion capabilities in almost all cases, producing evasive malware with an evasion rate in the range of 32-73%. It also produces surrogate models with a prediction label agreement with the respective target models between 97-99%. The surrogate could be used to fine-tune and improve the evasion rate in the future. Keywords: adversarial malware · reinforcement learning · model extraction · model stealing 1 Introduction As machine learning models are more commonly used in malware detection, there is a growing need for detection tools to combat evasive malware. Understanding attackers’ motives is vital in defending against malware, often created for profit. Malware-as-a-Service operations are used to automate the obfuscation and evasiveness of existing malware, and it is safe to assume that attackers will continue to improve their automation and try to create adversarial malware as efficiently as possible [45]. In this work, we are interested in the problem of automating the generation of evasive Windows malware executables primarily against machine learning static detection models. Creating evasive malicious binaries that preserve their functionality has been the subject of several works until now. Most works use a set of pre-defined actions that alter the Windows binary file by, e.g., adding benign sections and strings, modifying section names, and other ”non-destructive” alterations. The selection of the most appropriate set of actions is learned through reinforcement learning or similar approaches. To our knowledge, all prior work relies on the assumption that the target model (or system) is available to perform an unlimited number of checks or queries to verify whether a malicious binary has evaded the target. However, from the attacker’s perspective, fewer queries can a) generally mean less time to produce malware, b) lead to lower detection probabilities, and c) leak less information about the adversarial techniques. Therefore, assuming an attacker can do unlimited queries to a target model to modify their malware may not be realistic in evasive malware creation. We propose an algorithm that combines malware evasion with model extraction (MEME) based on model-based reinforcement learning. The goal of MEME is to learn a reinforcement learning policy that selects the appropriate modifications to a malicious Windows binary file in order for it to evade a target detection model while using a limited amount of interactions with the target. The core idea is to use observations and labels collected during the interaction with the reinforcement learning environment and use them with an auxiliary dataset to train a surrogate model of the target. Then the policy is trained to learn to evade the surrogate and evaluated on the original target. We test MEME using three malware detection models that are released and publicly available and on an antivirus installed by default in all Windows operating systems. MEME is compared with two baseline methods (a random policy and a PPO-based policy [35]) and well as with two state-of-the art methods (MAB [38] and GAMMA [9]). Using only 2,048 queries to the target during the training phase, MEME learns a policy that evades the targets with an evasion rate of 32-73%. MEME outperforms all baselines and state-of-the-art methods in all but one target. The algorithm also learns a surrogate model for each target with 97-99% label agreement with a much lower query budget than previously reported. The main contributions of this work are: – A novel combination of two attacks, malware evasion and model extraction in one algorithm (MEME). – An efficient generation of adversarial malware using model-based reinforcement learning while maintaining better evasion rates than state-of-the-art methods in most targets. – An efficient surrogate creation method that uses the adversarial samples produced during the training and evaluation of the reinforcement learning agent. The surrogates achieve high label agreement with the targets using minimal interaction with the target models. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Maria Rigaki, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and maria.rigaki@fel.cvut.cz; (2) Sebastian Garcia, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and sebastian.garcia@agents.fel.cvut.cz. Authors: Authors: (1) Maria Rigaki, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and maria.rigaki@fel.cvut.cz; (2) Sebastian Garcia, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and sebastian.garcia@agents.fel.cvut.cz. Table of Links Abstract & Introduction Abstract & Introduction Threat Model Threat Model Background and Related Work Background and Related Work Methodology Methodology Experiments Setup Experiments Setup Results Results Discussion Discussion Conclusion, Acknowledgments, and References Conclusion, Acknowledgments, and References Appendix Appendix Abstract Due to the proliferation of malware, defenders are increasingly turning to automation and machine learning as part of the malware detection toolchain. However, machine learning models are susceptible to adversarial attacks, requiring the testing of model and product robustness. Meanwhile, attackers also seek to automate malware generation and evasion of antivirus systems, and defenders try to gain insight into their methods. This work proposes a new algorithm that combines Malware Evasion and Model Extraction (MEME) attacks. MEME uses model-based reinforcement learning to adversarially modify Windows executable binary samples while simultaneously training a surrogate model with a high agreement with the target model to evade. To evaluate this method, we compare it with two state-of-the-art attacks in adversarial malware creation, using three well-known published models and one antivirus product as targets. Results show that MEME outperforms the state-of-the-art methods in terms of evasion capabilities in almost all cases, producing evasive malware with an evasion rate in the range of 32-73%. It also produces surrogate models with a prediction label agreement with the respective target models between 97-99%. The surrogate could be used to fine-tune and improve the evasion rate in the future. Keywords: adversarial malware · reinforcement learning · model extraction · model stealing Keywords: 1 Introduction As machine learning models are more commonly used in malware detection, there is a growing need for detection tools to combat evasive malware. Understanding attackers’ motives is vital in defending against malware, often created for profit. Malware-as-a-Service operations are used to automate the obfuscation and evasiveness of existing malware, and it is safe to assume that attackers will continue to improve their automation and try to create adversarial malware as efficiently as possible [45]. In this work, we are interested in the problem of automating the generation of evasive Windows malware executables primarily against machine learning static detection models. Creating evasive malicious binaries that preserve their functionality has been the subject of several works until now. Most works use a set of pre-defined actions that alter the Windows binary file by, e.g., adding benign sections and strings, modifying section names, and other ”non-destructive” alterations. The selection of the most appropriate set of actions is learned through reinforcement learning or similar approaches. To our knowledge, all prior work relies on the assumption that the target model (or system) is available to perform an unlimited number of checks or queries to verify whether a malicious binary has evaded the target. However, from the attacker’s perspective, fewer queries can a) generally mean less time to produce malware, b) lead to lower detection probabilities, and c) leak less information about the adversarial techniques. Therefore, assuming an attacker can do unlimited queries to a target model to modify their malware may not be realistic in evasive malware creation. We propose an algorithm that combines malware evasion with model extraction (MEME) based on model-based reinforcement learning. The goal of MEME is to learn a reinforcement learning policy that selects the appropriate modifications to a malicious Windows binary file in order for it to evade a target detection model while using a limited amount of interactions with the target. The core idea is to use observations and labels collected during the interaction with the reinforcement learning environment and use them with an auxiliary dataset to train a surrogate model of the target. Then the policy is trained to learn to evade the surrogate and evaluated on the original target. We test MEME using three malware detection models that are released and publicly available and on an antivirus installed by default in all Windows operating systems. MEME is compared with two baseline methods (a random policy and a PPO-based policy [35]) and well as with two state-of-the art methods (MAB [38] and GAMMA [9]). Using only 2,048 queries to the target during the training phase, MEME learns a policy that evades the targets with an evasion rate of 32-73%. MEME outperforms all baselines and state-of-the-art methods in all but one target. The algorithm also learns a surrogate model for each target with 97-99% label agreement with a much lower query budget than previously reported. The main contributions of this work are: – A novel combination of two attacks, malware evasion and model extraction in one algorithm (MEME). – An efficient generation of adversarial malware using model-based reinforcement learning while maintaining better evasion rates than state-of-the-art methods in most targets. – An efficient surrogate creation method that uses the adversarial samples produced during the training and evaluation of the reinforcement learning agent. The surrogates achieve high label agreement with the targets using minimal interaction with the target models. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv

The Power of MEME: Adversarial Malware Creation with Model-Based Reinforcement Learning

Comprehending the laws and boundaries of nature enhances our skills as programmers. In recent years, a fascinating analogy has emerged, drawing parallels between the universe and a concept familiar to computer scientists: the discrete event simulator. The universe operates much like a massive simulation coordinated without a central clock ticking. Let's dive in... Understanding the Universe as a Discrete Event Simulator Reality operates in discrete increments of time and space. Rather than a continuous flow, events unfold at specific points in time, triggering state transitions within the cosmic fabric. This concept aligns with the principles of , a computational method used to model systems where events occur at distinct moments. discrete-event simulation Time advances in discrete steps, with events driving changes in system states with discrete units of time. The universe appears to operate on a fundamental level where time and space are quantized, meaning they consist of indivisible units. indivisible Researchers think that Planck units encapsulate this discretization of spacetime. Physicists named Planck units after the physicist . Max Planck Planck Units and the Fabric of Reality represent the smallest possible length, time, mass, and other physical quantities within the framework of quantum mechanics. Planck units Units are derived from fundamental constants such as the , Planck's constant, and the . speed of light gravitational constant The represents the smallest possible length scale, approximately . Planck length 1.616 × 10^-35 meters The , approximately , marks the smallest possible time interval, beyond which time loses its meaning. Planck time 5.391 × 10^-44 seconds Planck time is the time it takes for light to travel a distance equal to the Planck length in a vacuum. This minimal duration represents the shortest meaningful interval of time that can be defined within our current understanding of physics. At durations shorter than the Planck time, the concept of time itself becomes ambiguous, and the laws of physics as we know them break down. You can't break plank units in the same way you cannot break . fundamental particles Ancient Greeks created the notion of indivisible particles, which they termed "atoms." The term "atom" originates from the Greek word "atomos," where "a-" means "not" or "un-" and "tomos" means "cut" or "divided." Atoms are not the least divisible particle but the brilliant idea remains. Implications for Computer Simulations The analogy between the universe and a discrete-event simulator has profound implications for computer simulations and scientific inquiry. https://hackernoon.com/what-is-wrong-with-software-uh8j3y7k?embedable=true Once you acknowledge the between the two realms, you can gain new insights into the nature of reality and develop more efficient simulation techniques. parallels The concept of the universe as a discrete event simulator challenges traditional notions of computation. In the past, computer scientists had a lot of trouble dealing with infinities, problems, , and dealing with divisible units. precision floats For example, comparing two dates was always a problem since we usually represent them as arbitrary divisible points in time. Navigating non-infinitely divisible money amounts, distances, and other measurements has sparked numerous renowned challenges like the , the , the , etc. Genesis mission Mars Climate Orbiter Ariane 5 Flight 501 Conclusion The universe as a discrete event simulator offers a compelling framework for understanding the fundamental principles governing reality. Planck units set a barrier on where to stop diving the measurements. I am not telling you to model timestamps with plank units precision. I am asking you to change your framework and acknowledge we are not dealing with arbitrary precision problems but with finite precision ones. We should leave problems related to infinities only to math domains, not real-world simulators.

Warning: The Universe's Event Simulator Is a Fierce Adversary for Coders!

are advanced AI systems designed to understand and generate human-like text and have become a driving force behind numerous applications, from chatbots to content generation. Large language models However, sometimes they avoid giving you the output you need. Recent research has revealed a vulnerability in , including not just large language models but other types as well. This vulnerability is known as where input data is subtly manipulated to deceive models into producing incorrect results. deep learning models , adversarial attacks What Are Adversarial Attacs? Adversarial attacks involve the intentional manipulation of machine learning models through the changes in the input data. These attacks , leading to misclassifications or erroneous outcomes. find weaknesses in the models’ decision-making mechanisms How Do They Work? Adversarial attacks in AI work by making tiny, hidden changes to things like pictures or text that confuse AI systems. These changes are into making mistakes or giving biased answers. By studying how the AI reacts to these tricky changes, attackers can learn how the AI works and where it might be vulnerable. specially designed to trick the AI Can LLMs resist? LLMs are not immune to these adversarial attacks, often referred to as in the context of LLMs. Jailbreaking involves skillfully crafting prompts to exploit model biases and generate outputs that may deviate from their intended purpose. "jailbreaking" Users of LLMs have experimented with manual prompt design, creating anecdotal prompts tailored for very specific situations. In fact, a recent dataset called contained intentionally designed to challenge LLM capabilities. "Harmful Behavior" 521 instances of harmful behaviors So, I decided to test out a framework that automatically generates universal adversarial prompts. These prompts can be added to the end of a user's input. And this suffix can be used across multiple user prompts and potentially across various LLMs. Explaining the approach This approach operates as a meaning it doesn't go deeply into the inner workings of LLMs and is limited to inspecting only the model's outputs. This aspect is important because, in real-life situations, access to model internals is often unavailable. , black-box method The attack strategy is making a single adversarial prompt that consistently disrupts the alignment of commercial models, relying on the model's output. The result? It was successful. And it raises questions regarding the usability, reliability, and ethical aspects of LLMs, in addition to existing challenges, including: Distortions These occur when models generate responses that are inaccurate or don't align with user intentions. For instance, they might produce responses that attribute human qualities or emotions, even when it's not appropriate. Safety Large language models can unintentionally expose private information, participate in phishing attempts, or generate unwanted spam. When misused, they can be manipulated to spread biased beliefs and misinformation, potentially causing widespread harm. Prejudice The quality of training data significantly influences a model's responses. If the data lacks diversity or primarily represents one specific group, the model's outputs may exhibit bias, perpetuating existing disparities. Permission When data is collected from the internet, these models can inadvertently infringe on copyright, plagiarize content, and compromise privacy by extracting personal details from descriptions, leading to potential legal complications. Last thoughts We’ll see even more frameworks of LLM models in the future. Because they really help you get the outputs you need from a model. However, ethical concerns and sustainability will remain essential for responsible AI deployment and energy-efficient training. Adapting AI models to specific business needs can enhance operational efficiency, customer service, and data analysis. So, in the end, the potential of AI and LLMs will be democratized, and we’ll get even more access to these technologies in the future.

What Are Large Language Models Capable Of: The Vulnerability of LLMs to Adversarial Attacks

The Secret Adversary by Agatha Christie is part of the HackerNoon Books Series. Read this book online for free on HackerNoon! : The Secret Adversary Title : Agatha Christie Author : January 1, 1998 [eBook #1155] Most recently updated: October 29, 2022 Release date : English Language Table of Links PROLOGUE CHAPTER THE YOUNG ADVENTURERS, LTD. MR. WHITTINGTON’S OFFER A SET BACK WHO IS JANE FINN? MR. JULIUS P. HERSHEIMMER A PLAN OF CAMPAIGN THE HOUSE IN SOHO THE ADVENTURES OF TOMMY TUPPENCE ENTERS DOMESTIC SERVICE ENTER SIR JAMES PEEL EDGERTON JULIUS TELLS A STORY A FRIEND IN NEED THE VIGIL A CONSULTATION TUPPENCE RECEIVES A PROPOSAL FURTHER ADVENTURES OF TOMMY ANNETTE THE TELEGRAM JANE FINN TOO LATE TOMMY MAKES A DISCOVERY IN DOWNING STREET A RACE AGAINST TIME JULIUS TAKES A HAND JANE’S STORY MR. BROWN A SUPPER PARTY AT THE SAVOY AND AFTER About HackerNoon Book Series: We bring you the most important technical, scientific, and insightful public domain books. This book is part of the public domain. Agatha Christie (1998). The Secret Adversary. Urbana, Illinois: Project Gutenberg. Retrieved https://www.gutenberg.org/cache/epub/1155/pg1155-images.html This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at www.gutenberg.org , located at https://www.gutenberg.org/policy/license.html.

The Secret Adversary by Agatha Christie - Table of Links

How to fool a 27M-parameter model with a bit of Python Do you think it is impossible to fool the vision system of a self-driving Tesla car? Or that machine learning models used in malware detection software are too good to be evaded by hackers? Or that face recognition systems in airports are bulletproof? Like any of us machine learning enthusiasts, you might fall into the trap of thinking that deep models used out there are perfect. Well, you are WRONG. There are easy ways to build that can fool any deep learning model and create security issues. In this post, we will cover the following: adversarial examples What are adversarial examples? How do you generate adversarial examples? Hands-on example: let’s break Inception 3 How to defend your models against adversarial examples All the source code shown in this article is in ** **👇🏽 this Github repo Let’s start! 1. What are adversarial examples? 😈 In the last 10 years, deep learning models have left the academic kindergarten, become big boys, and transformed many industries. This is especially true for computer vision models. When hit the charts in 2012, the deep learning era officially started. AlexNet Nowadays, computer vision models are as good or better than human vision. You can find them in myriad places, including… self-driving cars face recognition medical diagnosis surveillance systems malware detection Until recently, researchers trained and tested machine learning models in a environment, such as in machine learning competitions and academic papers. Nowadays, when deployed in real-world scenarios, security vulnerabilities coming from model errors have become a real concern. laboratory Imagine for a moment that the state-of-the-art-super-fancy-deep learning vision system of your self-driving car was not able to recognize as a stop sign. this stop sign Well, . This stop sign image is an . Think of it as an optical illusion for the model. this is exactly what happened adversarial example Let us look at another example. Below you have two images of a dog that are indistinguishable for us humans. The image on the left is an original picture of a puppy taken by . The one on the right is a slight modification of the first, that I created by adding the noise vector in the central image. Dominika Roseclay Inception v3 correctly classifies the original image as a dog breed (a redbone). However, this same model thinks, with very high confidence, that the modified image I created is a paper towel. In other words, I created an adversarial example. And you will too, as this is the example we will work on later in section 3. An for a computer vision model is an input image with small perturbations, imperceptible to the human eye, that causes a wrong model prediction. adversarial example Do not think these 2 examples are rare edge-case examples found after spending tons of time and computing resources. There are easy ways to generate adversarial examples, and this opens the door to serious vulnerabilities of machine learning systems in production. Let’s see how you can generate an adversarial example and fool a state-of-the-art image classification model. 2. How to generate adversarial examples? Adversarial examples 😈  are generated by taking a clean image 👼 that the model correctly classifies, and finding a small perturbation that causes the new image to be misclassified by the ML model. White-box scenario Let’s suppose you have complete information about the model you want to fool. In this case, you can compute the loss function of the model where is the input image X is the output class, y and is the vector of the network parameters. θ This loss function is typically the negative log-likelihood for classification methods. Your goal is to find a new image that is close to the original and that produces a big change in the value of the loss function. X’ X Imagine you are inside the space of all possible input images, sitting on top of the original image . This space has dimensions , so I will excuse you if you cannot visualize it well 😜. X width x height x channels To find an adversarial example, you need to walk a little bit in some direction in this space until you find another image with a remarkably different loss. You want to choose the direction that maximizes the change in the loss function for a fixed small step . X’ J epsilon Now, if you refresh a bit your Maths Calculus course, the direction in the space where the loss function changes the most is precisely the gradient of with respect to the . X J X The gradient of a function with respect to one of its variables is precisely the direction of maximal change. And by the way, this is the reason people train Neural Networks using Stochastic Gradient Descend and not Stochastic Random-Direction Descend. Fast gradient sign method An easy way to formalize this intuition is as follows: We take only the sign of the gradient and scale it using a small parameter epsilon, to guarantee that the distortion between and is small enough to be imperceptible to the human eye. This method is called the . X X fast gradient sign method Black-box attack In most scenarios, it is very likely you will not have complete information about the model. Hence, the previous method is not useful as you cannot compute the gradient. However, there exists a remarkable property called of adversarial examples that malicious agents can exploit to break a model even if they do not know its internal architecture and parameters. transferability Researchers have repeatedly observed that quite well between models, meaning that they can be designed for a target model A, but end up being effective against any other model trained on a similar dataset. adversarial examples transfer Adversarial examples can be generated as follows: Query the targeted model with inputs X_i for i = 1 … n and store the outputs y_i. Use the training data (X_i, y_i) to build another model, called the substitute model. Use a white-box algorithm like the fast gradient sign to generate adversarial examples for the substitute model. Many of them are going to transfer successfully and become adversarial examples for the target model as well. A successful application of this strategy against a commercial Machine learning model is presented in . this Computer Vision Foundation paper 3. Hands-on example: let’s break Inception 3 Let’s get our hands dirty and implement a few attacks using Python and the great library PyTorch. It always comes in handy to know how the attacker thinks. You can find the complete code in . this Github repo Our target model is going to be Inception V3, a powerful image classification model developed by Google, which has around 27M parameters and was trained on around 14M images belonging to 20k categories. # Load pretrained model from the PyTorch hub
# https://pytorch.org/hub/pytorch_vision_inception_v3/
from torchvision.models import inception_v3
model = inception_v3(pretrained=True)
model.eval()

# Count model parameters: 27,161,264
n_params = sum(p.numel() for p in model.parameters())
print(f'{n_params:,} parameters') We download the list of classes the model was trained on, and build an auxiliary dictionary that maps class ids to labels. # Download the txt file with the list of ImageNet classes the model was trained with
#     "!" magic to run shell commands from a Jupyter notebook
!wget https://raw.githubusercontent.com/pytorch/hub/master/imagenet_classes.txt

# id2label maps classes ids to their human-readable names: e.g. id2label[1] = 'goldfish'
with open("imagenet_classes.txt", "r") as f:
    categories = [s.strip() for s in f.readlines()]
    
id2label = {}
for idx, category in enumerate(categories):
    id2label[idx] = category Let us take the image of an innocent redbone dog as the starting image we will carefully modify to build adversarial examples: import requests
import io
from PIL import Image

url = 'https://previews.123rf.com/images/meinzahn/meinzahn1211/meinzahn121100339/16350068-cute-tiger-cat-isolated-on-white.jpg'

response = requests.get(url)
img = Image.open(io.BytesIO(response.content))
img Inception V3 expects images with dimensions 299 x 299 and normalized pixel ranges between -1 and 1. Let us preprocess our beautiful dog image: import torch
from torch import Tensor
from torchvision import transforms

def preprocess(img) -> Tensor:
    """
    Inception V3 model from pytorch expects input images with pixel values between -1 and 1
    and dimensions 299 x 299
    """
    mean = [0.485, 0.456, 0.406]
    std = [0.229, 0.224, 0.225]

    preprocess_fn = transforms.Compose([
        transforms.Resize((299,299)),  
        transforms.ToTensor(),
        transforms.Normalize(mean, std)
    ])
    image_tensor = preprocess_fn(img)

    # add batch dimension: C x H x W ==> B x C x H x W
    image_tensor = image_tensor.unsqueeze(0)
    
    return image_tensor
        
x = preprocess(img) and check that the model correctly classifies this image. from easydict import EasyDict
import torch.nn.functional as F

def get_predictions(img: Tensor) -> EasyDict:

    output = model.forward(img)

    class_idx = torch.max(output.data, 1)[1][0].item()

    label = id2label[class_idx]

    output_probs = F.softmax(output, dim=1)
    confidence =  round(torch.max(output_probs.data, 1)[0][0].item(), 4)
    
    return EasyDict(
        id=class_idx,
        label=label,
        confidence=confidence,
    )

get_predictions(x) # {'id': 168, 'label': 'redbone', 'confidence': 0.8861} Good. The model works as expected and the dog is classified as a dog :-). redbone redbone Let’s move to the fun part and generate adversarial examples using the Fast Gradient Sign method. from typing import Tuple
from torch.autograd import Variable

def fast_gradient_sign(x: Tensor, eps: float) -> Tuple[Tensor, Tensor]:
    """"""
    # convert tensor into a variable, because we will need to compute gradients
    # of the loss function with respect to the image pixels
    img_variable = Variable(x, requires_grad=True)
    
    # forward pass on the original image
    output = model.forward(img_variable)
    
    # get predicted class
    y_true = torch.max(output.data, 1)[1][0].item()   
    target = Variable(torch.LongTensor([y_true]), requires_grad=False)
    
    # backward pass to compute gradients
    loss_fn = torch.nn.CrossEntropyLoss()
    loss = loss_fn(output, target)
    
    # this will calculate gradient of each variable (with requires_grad=True)
    # that you can later access with "var.grad.data"
    # PyTorch does the heavy lifting, computing the gradient of the cross-entropy
    # with respect to the input image pixels.
    loss.backward(retain_graph=True)
    
    # sign of gradient of the loss func (with respect to input X)
    x_grad = torch.sign(img_variable.grad.data)
    
    # fast gradient sign formula
    x_adversarial = img_variable.data + eps * x_grad
        
    return x_adversarial, x_grad

# keep epsilon small to generate slight changes to the original image
epsilon = 0.02
x_adv, grad = fast_gradient_sign(x, epsilon) I have created an auxiliary function to both the original and the adversarial image. You can see the full implementation in this . visualize GitHub repository Well. It is interesting how the model prediction changed for the new image, which is almost indistinguishable from the original one. The new prediction is a , which is another dog breed with very similar skin color and big ears. As the puppy in question could be a mixed breed, the model mistake seems to be small, so we want to work further to really break this model. bloodhound One possibility is to play with different values of and try to find one that clearly gives a wrong prediction. Let’s try this. epsilon epsilons = [0.02, 0.2, 0.9]
for epsilon in epsilons:
    x_adv, grad = fast_gradient_sign(x, epsilon)
    print('epsilon: ', epsilon)
    visualize(x, x_adv, grad, epsilon) As epsilon increases the change in the image becomes visible. However, the model predictions are still other dog breeds: and . We need to be smarter than this to break the model. bloodhound basset Remember the intuition behind the Fast gradient sign method, i.e. imagine yourself inside the space of all possible images (with dimension 299 x 299 x 3), right where the original image X is. The gradient of the loss function tells you the direction you need to step to increase its value and make the model less certain about the right prediction. The size of the step is epsilon. You step and you check if you are now sitting on an adversarial example. An extension of this would be to take many smaller steps, instead of a single one. After each step, you re-evaluate the gradient and decide the new direction you are going to walk towards This method is called the Iterative Fast Gradient method. What an original name! More formally: Where , and denotes clipping of the input in the range of [X−ϵ, X+ϵ]. X0 = X Clip X,ϵ An implementation in PyTorch is the following: def iterative_fast_gradient_sign(x_: Tensor, epsilon: float, n_steps: int, alpha: float):
    """"""
    # copy to avoid modifying the original tensor
    x = x_.clone().detach()
    
    for step in range(n_steps):
         
        # one step using basic FGSM
        x_adv, grad = fast_gradient_sign(x, alpha)
        
        # total perturbation
        total_grad = x_adv - x_
        
        # force total perturbation to be lower than epsilon in
        # absolute value
        total_grad = torch.clamp(total_grad, -epsilon, epsilon)

        # add total perturbation to the original image
        x_adv = x_ + total_grad
               
        x = x_adv
        
    return x_adv, total_grad Now, let us try again to generate a good adversarial example starting with our innocent puppy. : bloodhound again Step 1 : beagle again Step 2 : mousetrap? Interesting. However, the model confidence on this prediction is only 16%. Step 3 Let’s go further. : One more dog breed, boring… Step 4 : beagle again.. Step 5 : Step 6 : redbone again. Step 7 Keep calm and continue walking in the image space. : Step 8 : BINGO! 🔥🔥🔥 Step 9 What an exotic paper towel. And the model is pretty confident about its prediction, almost 99%. If you compare the initial and the final image we found in step 9 you see they are essentially the same, a , but for the Inception V3 model they are two very different things. puppy I was very surprised the first time I saw such a thing. How small modifications to an image can cause such model misbehavior. This example is funny, but if you think about the implications for a self-driving car vision you might start to worry a bit. Deep learning models deployed in critical tasks need to properly handle adversarial examples, but how? 4. How to defend your models against adversarial examples? As of November 2021, there is no universal defense strategy you can follow to protect against adversarial examples. In other words, attacking a model with adversarial examples is easier than protecting it. The best universal strategy, that can protect against any known attack, is to use adversarial examples when training the model. If the model adversarial examples during training, its performance at prediction time will be better for adversarial examples generated in the same way. This technique is called . sees adversarial training You generate them on the fly, as you train the model, and adjust the loss function to look both at clean and adversarial inputs. For example, we could eliminate the adversarial examples we found in the previous section if we added the 10+ examples to the training set and labelled all of them as . redbone This defense is very effective against attacks that use the Fast Gradient Sign method. However, there exist more powerful attacks, that we did not cover in this post, that can bypass this defense. If you wanna know more I encourage you to read by Nicholas Carlini and David Wagner. this amazing paper This is the reason why adversarial defense is an open problem in Machine Learning and cybersecurity. Conclusion Adversarial examples are a fascinating topic at the intersection of cybersecurity and machine learning. However, it is still an open problem waiting to be solved. In this post, I gave a practical introduction to the topic with code examples. If you would like to work further I suggest , a Python library for adversarial machine learning developed by the great Ian Goodfellow and Nicolas Papernot, and currently maintained by the . cleverhans University of Toronto All the source code I shown in this article is in . this Github repo Wanna become a PRO in Machine Learning? I offer hands-on content on real-world Machine Learning, that helps you get a top job in the Data world. 👉🏽  Subscribe to the . datamachines newsletter 👉🏽  Follow me on and . LinkedIn Twitter Have a great day 🧡 Pau

Adversa

adversa.ai

-- employees

Since n.d.

COMPANY RANKING

ADVERSA

EVERGREEN INDEX #966

Adversa WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

Adversa

adversa.ai

-- employees

Since n.d.

COMPANY RANKING

ADVERSA

EVERGREEN INDEX #966

RELATED COMPANIESRANK

STORIES ABOUT ADVERSA

Adversa WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!