Adversa AI is a leading provider of Security and Safety solutions for AI


Adversa

adversa

Authors: (1) Maria Rigaki, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and maria.rigaki@fel.cvut.cz; (2) Sebastian Garcia, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and sebastian.garcia@agents.fel.cvut.cz. Table of Links Abstract & Introduction Threat Model Background and Related Work Methodology Experiments Setup Results Discussion Conclusion, Acknowledgments, and References Appendix Appendix A. Hyper-parameter Tuning The search space for the PPO hyper-parameters: – gamma: 0.01 - 0.75 – max grad norm: 0.3 - 5.0 – learning rate: 0.001 - 0.1 – activation function: ReLU or Tanh – neural network size: small or medium Selected parameters: gamma=0.854, learning rate=0.00138, max grad norm=0.4284, activation function=Tanh, small network size (2 layers with 64 units each). The search space for the LGB surrogate training hyper-parameters: – alpha: 1 - 1,000 – num boosting rounds: 100-2,000 – learning rate: 0.001 - 0.1 – num leaves: 128 - 2,048 – max depth: 5 - 16 – min child samples: 5 - 100 – feature fraction: 0.4 - 1.0 Hyper-parameter settings for the training of each LGB surrogate Table 4. This paper is under CC BY-NC-SA 4.0 DEED license. available on arxiv

Adversarial Malware Creation with Model-Based Reinforcement Learning: Appendix

Authors: (1) Maria Rigaki, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and maria.rigaki@fel.cvut.cz; (2) Sebastian Garcia, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and sebastian.garcia@agents.fel.cvut.cz. Table of Links Abstract & Introduction Threat Model Background and Related Work Methodology Experiments Setup Results Discussion Conclusion, Acknowledgments, and References Appendix 8 Conclusions By employing model-based reinforcement learning, MEME generates adversarial malware samples that successfully evade antivirus systems and train a surrogate model mimicking the target classifier accurately. Our experiments show that MEME surpasses existing methods in evasion rate, suggesting its potential for various applications, such as testing model robustness and enhancing cyber security against advanced persistent threats. Future work may involve exploring ensemble surrogates and other optimizations to enhance MEME’s performance further. Acknowledgments The authors acknowledge support from the Strategic Support for the Development of Security Research in the Czech Republic 2019–2025 (IMPAKT 1) program, by the Ministry of the Interior of the Czech Republic under No. VJ02010020 – AI-Dojo: Multi-agent testbed for the research and testing of AI-driven cyber security technologies. The authors acknowledge the support of NVIDIA Corporation with the donation of a Titan V GPU used for this research. References 1. Akiba, T., Sano, S., Yanase, T., Ohta, T., Koyama, M.: Optuna: A Next-generation Hyperparameter Optimization Framework. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 2623–2631. KDD ’19, Association for Computing Machinery, New York, NY, USA (Jul 2019). https://doi.org/10.1145/3292500.3330701 2. Anderson, H.S., Kharkar, A., Filar, B., Evans, D., Roth, P.: Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learning (Jan 2018). https://doi.org/10.48550/arXiv.1801.08917, arXiv:1801.08917 [cs] 3. Anderson, H.S., Roth, P.: EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models (Apr 2018). https://doi.org/10.48550/arXiv.1804.04637, arXiv:1804.04637 [cs] 4. Bergstra, J., Bardenet, R., Bengio, Y., K´egl, B.: Algorithms for Hyper-Parameter Optimization. In: Advances in Neural Information Processing Systems. vol. 24. Curran Associates, Inc. (2011) 5. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: OpenAI Gym (Jun 2016). https://doi.org/10.48550/arXiv.1606.01540, arXiv:1606.01540 [cs] 6. Ceschin, F., Botacin, M., Gomes, H.M., Oliveira, L.S., Gr´egio, A.: Shallow Security: on the Creation of Adversarial Variants to Evade Machine Learning-Based Malware Detectors. In: Proceedings of the 3rd Reversing and Offensive-oriented Trends Symposium. pp. 1–9. ROOTS’19, Association for Computing Machinery, New York, NY, USA (Feb 2020). https://doi.org/10.1145/3375894.3375898 7. Chandrasekaran, V., Chaudhuri, K., Giacomelli, I., Jha, S., Yan, S.: Exploring connections between active learning and model extraction. In: Proceedings of the 29th USENIX Conference on Security Symposium. pp. 1309–1326. SEC’20, USENIX Association, USA (Aug 2020) 8. Correia-Silva, J.R., Berriel, R.F., Badue, C., de Souza, A.F., Oliveira-Santos, T.: Copycat CNN: Stealing Knowledge by Persuading Confession with Random Non-Labeled Data. In: 2018 International Joint Conference on Neural Networks (IJCNN). pp. 1–8 (Jul 2018). https://doi.org/10.1109/IJCNN.2018.8489592, iSSN: 2161-4407 9. Demetrio, L., Biggio, B., Lagorio, G., Roli, F., Armando, A.: FunctionalityPreserving Black-Box Optimization of Adversarial Windows Malware. IEEE Transactions on Information Forensics and Security 16, 3469–3478 (2021). https://doi.org/10.1109/TIFS.2021.3082330, conference Name: IEEE Transactions on Information Forensics and Security 10. Demetrio, L., Coull, S.E., Biggio, B., Lagorio, G., Armando, A., Roli, F.: Adversarial exemples: A survey and experimental evaluation of practical attacks on machine learning for windows malware detection. ACM Transactions on Privacy and Security (TOPS) 24(4), 1–31 (2021), publisher: ACM New York, NY, USA 11. Dowling, S., Schukat, M., Barrett, E.: Using Reinforcement Learning to Conceal Honeypot Functionality. In: Brefeld, U., Curry, E., Daly, E., MacNamee, B., Marascu, A., Pinelli, F., Berlingerio, M., Hurley, N. (eds.) Machine Learning and Knowledge Discovery in Databases. pp. 341–355. Lecture Notes in Computer Science, Springer International Publishing, Cham (2019). https://doi.org/10.1007/978-3-030-10997-4 21 12. Fang, Y., Zeng, Y., Li, B., Liu, L., Zhang, L.: DeepDetectNet vs RLAttackNet: An adversarial method to improve deep learning-based static malware detection model. PLOS ONE 15(4), e0231626 (Apr 2020). https://doi.org/10.1371/journal.pone.0231626, publisher: Public Library of Science 13. Fang, Z., Wang, J., Geng, J., Kan, X.: Feature Selection for Malware Detection Based on Reinforcement Learning. IEEE Access 7, 176177–176187 (2019). https://doi.org/10.1109/ACCESS.2019.2957429, conference Name: IEEE Access 14. Fang, Z., Wang, J., Li, B., Wu, S., Zhou, Y., Huang, H.: Evading Anti-Malware Engines With Deep Reinforcement Learning. IEEE Access 7, 48867–48879 (2019). https://doi.org/10.1109/ACCESS.2019.2908033, conference Name: IEEE Access 15. Harang, R., Rudd, E.M.: SOREL-20M: A Large Scale Benchmark Dataset for Malicious PE Detection. arXiv:2012.07634 [cs] (Dec 2020), arXiv: 2012.07634 16. Hu, W., Tan, Y.: Generating Adversarial Malware Examples for Black-Box Attacks Based on GAN. In: Tan, Y., Shi, Y. (eds.) Data Mining and Big Data. pp. 409–423. Communications in Computer and Information Science, Springer Nature, Singapore (2022). https://doi.org/10.1007/978-981-19-8991-9 29 17. Huang, L., Zhu, Q.: Adaptive Honeypot Engagement Through Reinforcement Learning of Semi Markov Decision Processes. In: Alpcan, T., Vorobeychik, Y., Baras, J.S., D´an, G. (eds.) Decision and Game Theory for Security. pp. 196– 216. Lecture Notes in Computer Science, Springer International Publishing, Cham (2019). https://doi.org/10.1007/978-3-030-32430-8 13 18. Institute, A.T.: AV-ATLAS - Malware & PUA (2023), https://portal.av-atlas.org/malware 19. Jagielski, M., Carlini, N., Berthelot, D., Kurakin, A., Papernot, N.: High Accuracy and High Fidelity Extraction of Neural Networks. pp. 1345–1362 (2020) 20. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., Liu, T.Y.: LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In: Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc. (2017) 21. Kurutach, T., Clavera, I., Duan, Y., Tamar, A., Abbeel, P.: Model-Ensemble TrustRegion Policy Optimization. In: International Conference on Learning Representations (2018) 22. Labaca-Castro, R., Franz, S., Rodosek, G.D.: AIMED-RL: Exploring Adversarial Malware Examples with Reinforcement Learning. In: Dong, Y., Kourtellis, N., Hammer, B., Lozano, J.A. (eds.) Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track. pp. 37–52. Lecture Notes in Computer Science, Springer International Publishing, Cham (2021). https://doi.org/10.1007/978-3-030-86514-6 3 23. Li, X., Li, Q.: An IRL-based malware adversarial generation method to evade anti-malware engines. Computers & Security 104, 102118 (May 2021). https://doi.org/10.1016/j.cose.2020.102118 24. Ling, X., Wu, L., Zhang, J., Qu, Z., Deng, W., Chen, X., Qian, Y., Wu, C., Ji, S., Luo, T., Wu, J., Wu, Y.: Adversarial attacks against Windows PE malware detection: A survey of the state-of-the-art. Computers & Security 128, 103134 (May 2023). https://doi.org/10.1016/j.cose.2023.103134 25. Lundberg, S.M., Lee, S.I.: A Unified Approach to Interpreting Model Predictions. In: Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc. (2017) 26. Nguyen, T.T., Reddi, V.J.: Deep Reinforcement Learning for Cyber Security. IEEE Transactions on Neural Networks and Learning Systems pp. 1–17 (2021). https://doi.org/10.1109/TNNLS.2021.3121870, conference Name: IEEE Transactions on Neural Networks and Learning Systems 27. Orekondy, T., Schiele, B., Fritz, M.: Knockoff Nets: Stealing Functionality of BlackBox Models. pp. 4954–4963 (2019) 28. Pal, S., Gupta, Y., Shukla, A., Kanade, A., Shevade, S., Ganapathy, V.: ActiveThief: Model Extraction Using Active Learning and Unannotated Public Data. Proceedings of the AAAI Conference on Artificial Intelligence 34(01), 865–872 (Apr 2020). https://doi.org/10.1609/aaai.v34i01.5432, number: 01 29. Phan, T.D., Duc Luong, T., Hoang Quoc An, N., Nguyen Huu, Q., Nghi, H.K., Pham, V.H.: Leveraging Reinforcement Learning and Generative Adversarial Networks to Craft Mutants of Windows Malware against Black-box Malware Detectors. In: Proceedings of the 11th International Symposium on Information and Communication Technology. pp. 31–38. SoICT ’22, Association for Computing Machinery, New York, NY, USA (Dec 2022) 30. Quertier, T., Marais, B., Morucci, S., Fournel, B.: MERLIN – Malware Evasion with Reinforcement LearnINg (Mar 2022), arXiv:2203.12980 [cs] 31. Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., Dormann, N.: StableBaselines3: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research 22(268), 1–8 (2021) 32. Rigaki, M., Garcia, S.: Stealing and evading malware classifiers and antivirus at low false positive conditions. Computers & Security 129, 103192 (Jun 2023). https://doi.org/10.1016/j.cose.2023.103192 33. Rosenberg, I., Meir, S., Berrebi, J., Gordon, I., Sicard, G., Omid David, E.: Generating End-to-End Adversarial Examples for Malware Classifiers Using Explainability. In: 2020 International Joint Conference on Neural Networks (IJCNN). pp. 1–10 (Jul 2020). https://doi.org/10.1109/IJCNN48605.2020.9207168, iSSN: 2161-4407 34. Sanyal, S., Addepalli, S., Babu, R.V.: Towards Data-Free Model Stealing in a Hard Label Setting. pp. 15284–15293 (2022) 35. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017) 36. Security.org, T.: 2023 Antivirus Market Annual Report (Feb 2023), https://www.security.org/antivirus/antivirus-consumer-report-annual/ 37. Severi, G., Meyer, J., Coull, S., Oprea, A.: Explanation-Guided Backdoor Poisoning Attacks Against Malware Classifiers. In: 30th USENIX Security Symposium (USENIX Security 21). pp. 1487–1504. USENIX Association (2021) 38. Song, W., Li, X., Afroz, S., Garg, D., Kuznetsov, D., Yin, H.: MAB-Malware: A Reinforcement Learning Framework for Blackbox Generation of Adversarial Malware. In: Proceedings of the 2022 ACM on Asia Conference on Computer and Communications Security. pp. 990–1003. ASIA CCS ’22, Association for Computing Machinery, New York, NY, USA (May 2022). https://doi.org/10.1145/3488932.3497768 39. Sussman, B.: New Malware Is Born Every Minute (May 2023), https://blogs.blackberry.com/en/2023/05/new-malware-born-every-minute 40. Sutton, R.S., Barto, A.G.: Reinforcement Learning, second edition: An Introduction. MIT Press (Nov 2018) 41. Total, V.: VirusTotal - Stats, https://www.virustotal.com/gui/stats 42. Uprety, A., Rawat, D.B.: Reinforcement Learning for IoT Security: A Comprehensive Survey. IEEE Internet of Things Journal 8(11), 8693–8706 (Jun 2021). https://doi.org/10.1109/JIOT.2020.3040957, conference Name: IEEE Internet of Things Journal 43. Wu, C., Shi, J., Yang, Y., Li, W.: Enhancing Machine Learning Based Malware Detection Model by Reinforcement Learning. In: Proceedings of the 8th International Conference on Communication and Network Security. pp. 74–78. ICCNS 2018, Association for Computing Machinery, New York, NY, USA (Nov 2018). https://doi.org/10.1145/3290480.3290494 44. Zolotukhin, M., Kumar, S., H¨am¨al¨ainen, T.: Reinforcement Learning for Attack Mitigation in SDN-enabled Networks. In: 2020 6th IEEE Conference on Network Softwarization (NetSoft). pp. 282–286 (Jun 2020). https://doi.org/10.1109/NetSoft48620.2020.9165383 45. Sembera, V., Paquet-Clouston, M., Garcia, S., Erquiaga, M.J.: Cybercrime Spe- ˇ cialization: An Expos´e of a Malicious Android Obfuscation-as-a-Service. In: 2021 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW. pp. 213–236 (2021). https://doi.org/10.1109/EuroSPW54576.2021.00029 This paper is under CC BY-NC-SA 4.0 DEED license. available on arxiv

Enhancing Cybersecurity with MEME: Reinforcement Learning for Adversarial Malware Evasion

Authors: (1) Maria Rigaki, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and maria.rigaki@fel.cvut.cz; (2) Sebastian Garcia, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and sebastian.garcia@agents.fel.cvut.cz. Table of Links Abstract & Introduction Threat Model Background and Related Work Methodology Experiments Setup Results Discussion Conclusion, Acknowledgments, and References Appendix 5 Experiments Setup To evaluate MEME, several experiments were conducted in different configurations. First, there was a selection of four different malware detection solutions as targets to evade. Second, there was a comparison of MEME and four other evasion techniques on these four targets. 5.1 Targets The selection of targets was made to include three highly cited malware detection models together with a real implementation of a popular free Antivirus solution. 1. Ember. A LightGBM [20] model that was released as part of the Ember dataset [3] which was used for training that same model. The decision threshold was set to 0.8336, which corresponds to a 1% false positive rate (FPR) on the Ember 2018 test set. 2. Sorel-LGB. A LightGBM model that was distributed as part of the Sorel20M [15] dataset, which was used for training that same model. The decision threshold was set to 0.5, which corresponds to a 0.2% false positive rate (FPR) on the Sorel-20M test set. 3. Sorel-FFNN. A feed-forward neural network (FFNN) that was also released as part of the Sorel 20M dataset and was trained using the same data. The decision threshold was set to 0.5, which corresponds to 0.6% false positive rate (FPR) on the Sorel-20M test set. 4. Microsoft Defender. An antivirus product that comes pre-installed with the Windows operating system. According to [36], it is the most used free antivirus product for personal computers. All tests were performed using a virtual machine (VM) running an updated version of the product. The VM had no internet connectivity during the binary file scanning. 5.2 Datasets Our experiments required the use of the following datasets: 1. Ember 2018. A dataset that consists of features extracted from one million Windows Portable Executable (PE) files [3]. The dataset is split into training, testing, and ”unlabeled sets”. The training set consists of 300,000 clean samples, 300,000 malicious samples, and 200,000 “unlabeled” samples. The so-called unlabeled part of the dataset was truly unlabeled in the first version of the dataset, however, in the 2018 release, the authors provided an avclass label for all malicious samples, including those in the unlabeled set. Each sample has 2,381 static features related to byte and entropy histograms, PE header information, strings, imports, data directories, etc. 2. Sorel-20M. The Sorel dataset [15] was released in 2020 and contains the extracted features of 20 million binary files (malicious and benign). The feature set used was the same as the one from the Ember dataset. 3. Malware Binary Files. In addition to the Ember features used for training the surrogate, we also obtained 1,000 malicious binary files whose hashes were part of Ember 2018 and we use them for generating the evasive malware with all the methods. 4. Benign Binary Files. All methods require a benign set of data from where they extract benign strings, sections, and other elements that are used for the binary modifications. The same set of 100 benign binaries were used in all experiments. The files were obtained from a Windows 10 virtual machine after installing known benign software. MEME created two versions of the Daux dataset to train the surrogate models. For the Ember and AV surrogates, Daux contains the unlabeled part of the Ember dataset. For the Sorel surrogates, Daux contains 200,000 samples from the Sorel-20M validation set. These datasets were chosen to create the surrogate because they were not used to train the corresponding targets. During the evaluation, a subset of the Sorel-20M test set was used to evaluate the performance of the Sorel surrogate models and the Ember test set was used to evaluate the Ember and AV models. The 1,000 malware binaries were split into training and test sets with a 70-30% ratio using five different seeds. The test sets were used to test all the methods, while the 700 binaries in the training set were used to train each of the RL policies for PPO and MEME (these were the binaries to which modifier actions were applied). 5.3 Adversarial Malware Generation Comparison With regard to the generation of adversarial malware, MEME is compared with four algorithms in total. Two baseline reinforcement learning algorithms that use the Malware-Gym environment: a random agent and an agent that learns a policy using the vanilla Proximal Policy Optimization (PPO) algorithm [35], and two state-of-the-art (SOTA) algorithms: MAB [38] and GAMMA [9]. The two SOTA algorithms were selected based on the fact that they are relatively recently released and the fact that they seem to perform well in the malware evasion task. In addition, their source code is available. The detailed setup used for each of the algorithms, as well as any modifications, are presented below: 1. Random Agent The random agent is the simplest baseline used in our experiments. It uses the Malware-Gym environment and randomly samples the next modification action from the available action space. The agent is evaluated in the test environments using the 300 test malware samples. 2. PPO This is an agent that uses the PPO [35] algorithm as implemented in the Stable-Baselines3 software package. The agent was trained for 2,048 steps on the malware training set and evaluated on the malware test set. To select the hyper-parameters related to PPO training, we used the Tree-structured Parzen Estimator (TPE) method [4] as implemented in the software package Optuna [1]. The TPE algorithm was executed with the Ember dataset, but the settings performed well in the other targets. The tuned hyper-parameters were γ, the learning rate, the maximum gradient norm, the activation function, and the neural network size for the actor and critic models. The search space of each parameter and the final values are presented in the Appendix. 3. MAB RL algorithm that treats the evasive malware generation as a multiarmed bandit problem2. It operates in two stages: evasion and minimization. It samples the action space, which includes generic actions (similar to Malware-Gym) and any successful evasive actions along with the specific modifiers, e.g., appending a specific benign section. MAB directly manipulates each binary without generating a learned policy. Therefore, all experiments were conducted directly on the malicious binary test set. 4. GAMMA An algorithm that injects benign binary sections into malicious PE files while preserving their functionality. It modifies features like section count, byte histograms, and strings, leaving features related to, e.g., certificates and debugging data unaffected. By employing genetic algorithms, GAMMA searches for optimal benign sections to reduce the target model’s confidence by minimizing the content and location of injected sections. The attack is implemented in the SecML library3. Though effective, it has a significantly longer runtime than other tested methods. The attack uses a restricted set of 30 available benign sections and a population size of 20. The λ parameter, impacting the injected data size, was set to 10−6. GAMMA directly operates on each binary and does not generate a learned policy. Hence, all experiments were conducted on the malicious binary test set. 5.4 MEME Experimental Setup The initial training steps n in Algorithm 1 were set to 1,024, and the total number of loops k was two. For evaluation, the test set of 300 malware binaries was used. The surrogate training steps m were set to 2,048 (step 6 of Algorithm 1). MEME utilizes PPO for training and updating the policy (πθ). The PPO settings remained the same as in the baseline experiments, enabling a comparison of the impact of using a surrogate model for additional PPO training. In total, there were a total of 2,048 queries to the target and 4,096 training steps using the surrogate environment. The surrogate was always a LightGBM model. Surrogate training involved two datasets: Daux from an external dataset (e.g., Ember 2018 or Sorel-20M) and Dsur generated during lines 3 and 7 of Algorithm 1. These datasets were mixed with a ratio α, a hyperparameter for LGB surrogate tuning. Other hyperparameters, such as the number of boosting trees, learning rate, tree depth, minimum child samples, and feature fraction, were also tuned separately for each target using TPE and Optuna. Appendix A provides the detailed search space and selected values. Surrogate models were evaluated using the respective target test sets, and a decision threshold matching target FPR levels was calculated. For the AV target without a representative dataset, the surrogate’s decision threshold was set to 0.5. 5.5 General Experiment Settings All the algorithms were tested under a common set of constraints. The maximum allowed modifications to a binary file were set to 15 for the RL-based algorithms. Similarly, for GAMMA, the number of iterations was set to 15; for MAB, the number of ”pulls” was also 15. The second constraint was to set the maximum running time for all experiments to 4 hours. For MAB and GAMMA, this setting means that the algorithms must manage to handle as many malicious binaries as possible in that time. At the same time, for PPO and MEME, this time included both the policy training time and the evaluation time. For PPO and MEME, we set the query budget to 2,048. This budget does not include the final evaluation queries on the test set. MAB and GAMMA were not constrained in the total number of queries because it is not supported by the respective frameworks. Finally, all experiments were run with five different seeds. The random seeds controlled the split of the 1,000 malicious binaries into train and test, and therefore, all methods were tested in the same files. This paper is under CC BY-NC-SA 4.0 DEED license. available on arxiv

MEME Algorithm's Impact on Adversarial Malware Generation and Model Evasion

Authors:
(1) Maria Rigaki, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and maria.rigaki@fel.cvut.cz;
(2) Sebastian Garcia, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and sebastian.garcia@agents.fel.cvut.cz. Table of Links Abstract & Introduction Threat Model Background and Related Work Methodology Experiments Setup Results Discussion Conclusion, Acknowledgments, and References Appendix Abstract Due to the proliferation of malware, defenders are increasingly turning to automation and machine learning as part of the malware detection toolchain. However, machine learning models are susceptible to adversarial attacks, requiring the testing of model and product robustness. Meanwhile, attackers also seek to automate malware generation and evasion of antivirus systems, and defenders try to gain insight into their methods. This work proposes a new algorithm that combines Malware Evasion and Model Extraction (MEME) attacks. MEME uses model-based reinforcement learning to adversarially modify Windows executable binary samples while simultaneously training a surrogate model with a high agreement with the target model to evade. To evaluate this method, we compare it with two state-of-the-art attacks in adversarial malware creation, using three well-known published models and one antivirus product as targets. Results show that MEME outperforms the state-of-the-art methods in terms of evasion capabilities in almost all cases, producing evasive malware with an evasion rate in the range of 32-73%. It also produces surrogate models with a prediction label agreement with the respective target models between 97-99%. The surrogate could be used to fine-tune and improve the evasion rate in the future. Keywords: adversarial malware · reinforcement learning · model extraction · model stealing 1 Introduction As machine learning models are more commonly used in malware detection, there is a growing need for detection tools to combat evasive malware. Understanding attackers’ motives is vital in defending against malware, often created for profit. Malware-as-a-Service operations are used to automate the obfuscation and evasiveness of existing malware, and it is safe to assume that attackers will continue to improve their automation and try to create adversarial malware as efficiently as possible [45]. In this work, we are interested in the problem of automating the generation of evasive Windows malware executables primarily against machine learning static detection models. Creating evasive malicious binaries that preserve their functionality has been the subject of several works until now. Most works use a set of pre-defined actions that alter the Windows binary file by, e.g., adding benign sections and strings, modifying section names, and other ”non-destructive” alterations. The selection of the most appropriate set of actions is learned through reinforcement learning or similar approaches. To our knowledge, all prior work relies on the assumption that the target model (or system) is available to perform an unlimited number of checks or queries to verify whether a malicious binary has evaded the target. However, from the attacker’s perspective, fewer queries can a) generally mean less time to produce malware, b) lead to lower detection probabilities, and c) leak less information about the adversarial techniques. Therefore, assuming an attacker can do unlimited queries to a target model to modify their malware may not be realistic in evasive malware creation. We propose an algorithm that combines malware evasion with model extraction (MEME) based on model-based reinforcement learning. The goal of MEME is to learn a reinforcement learning policy that selects the appropriate modifications to a malicious Windows binary file in order for it to evade a target detection model while using a limited amount of interactions with the target. The core idea is to use observations and labels collected during the interaction with the reinforcement learning environment and use them with an auxiliary dataset to train a surrogate model of the target. Then the policy is trained to learn to evade the surrogate and evaluated on the original target. We test MEME using three malware detection models that are released and publicly available and on an antivirus installed by default in all Windows operating systems. MEME is compared with two baseline methods (a random policy and a PPO-based policy [35]) and well as with two state-of-the art methods (MAB [38] and GAMMA [9]). Using only 2,048 queries to the target during the training phase, MEME learns a policy that evades the targets with an evasion rate of 32-73%. MEME outperforms all baselines and state-of-the-art methods in all but one target. The algorithm also learns a surrogate model for each target with 97-99% label agreement with a much lower query budget than previously reported. The main contributions of this work are: – A novel combination of two attacks, malware evasion and model extraction in one algorithm (MEME). – An efficient generation of adversarial malware using model-based reinforcement learning while maintaining better evasion rates than state-of-the-art methods in most targets. – An efficient surrogate creation method that uses the adversarial samples produced during the training and evaluation of the reinforcement learning agent. The surrogates achieve high label agreement with the targets using minimal interaction with the target models. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Maria Rigaki, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and maria.rigaki@fel.cvut.cz; (2) Sebastian Garcia, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and sebastian.garcia@agents.fel.cvut.cz. Authors: Authors: (1) Maria Rigaki, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and maria.rigaki@fel.cvut.cz; (2) Sebastian Garcia, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and sebastian.garcia@agents.fel.cvut.cz. Table of Links Abstract & Introduction Abstract & Introduction Threat Model Threat Model Background and Related Work Background and Related Work Methodology Methodology Experiments Setup Experiments Setup Results Results Discussion Discussion Conclusion, Acknowledgments, and References Conclusion, Acknowledgments, and References Appendix Appendix Abstract Due to the proliferation of malware, defenders are increasingly turning to automation and machine learning as part of the malware detection toolchain. However, machine learning models are susceptible to adversarial attacks, requiring the testing of model and product robustness. Meanwhile, attackers also seek to automate malware generation and evasion of antivirus systems, and defenders try to gain insight into their methods. This work proposes a new algorithm that combines Malware Evasion and Model Extraction (MEME) attacks. MEME uses model-based reinforcement learning to adversarially modify Windows executable binary samples while simultaneously training a surrogate model with a high agreement with the target model to evade. To evaluate this method, we compare it with two state-of-the-art attacks in adversarial malware creation, using three well-known published models and one antivirus product as targets. Results show that MEME outperforms the state-of-the-art methods in terms of evasion capabilities in almost all cases, producing evasive malware with an evasion rate in the range of 32-73%. It also produces surrogate models with a prediction label agreement with the respective target models between 97-99%. The surrogate could be used to fine-tune and improve the evasion rate in the future. Keywords: adversarial malware · reinforcement learning · model extraction · model stealing Keywords: 1 Introduction As machine learning models are more commonly used in malware detection, there is a growing need for detection tools to combat evasive malware. Understanding attackers’ motives is vital in defending against malware, often created for profit. Malware-as-a-Service operations are used to automate the obfuscation and evasiveness of existing malware, and it is safe to assume that attackers will continue to improve their automation and try to create adversarial malware as efficiently as possible [45]. In this work, we are interested in the problem of automating the generation of evasive Windows malware executables primarily against machine learning static detection models. Creating evasive malicious binaries that preserve their functionality has been the subject of several works until now. Most works use a set of pre-defined actions that alter the Windows binary file by, e.g., adding benign sections and strings, modifying section names, and other ”non-destructive” alterations. The selection of the most appropriate set of actions is learned through reinforcement learning or similar approaches. To our knowledge, all prior work relies on the assumption that the target model (or system) is available to perform an unlimited number of checks or queries to verify whether a malicious binary has evaded the target. However, from the attacker’s perspective, fewer queries can a) generally mean less time to produce malware, b) lead to lower detection probabilities, and c) leak less information about the adversarial techniques. Therefore, assuming an attacker can do unlimited queries to a target model to modify their malware may not be realistic in evasive malware creation. We propose an algorithm that combines malware evasion with model extraction (MEME) based on model-based reinforcement learning. The goal of MEME is to learn a reinforcement learning policy that selects the appropriate modifications to a malicious Windows binary file in order for it to evade a target detection model while using a limited amount of interactions with the target. The core idea is to use observations and labels collected during the interaction with the reinforcement learning environment and use them with an auxiliary dataset to train a surrogate model of the target. Then the policy is trained to learn to evade the surrogate and evaluated on the original target. We test MEME using three malware detection models that are released and publicly available and on an antivirus installed by default in all Windows operating systems. MEME is compared with two baseline methods (a random policy and a PPO-based policy [35]) and well as with two state-of-the art methods (MAB [38] and GAMMA [9]). Using only 2,048 queries to the target during the training phase, MEME learns a policy that evades the targets with an evasion rate of 32-73%. MEME outperforms all baselines and state-of-the-art methods in all but one target. The algorithm also learns a surrogate model for each target with 97-99% label agreement with a much lower query budget than previously reported. The main contributions of this work are: – A novel combination of two attacks, malware evasion and model extraction in one algorithm (MEME). – An efficient generation of adversarial malware using model-based reinforcement learning while maintaining better evasion rates than state-of-the-art methods in most targets. – An efficient surrogate creation method that uses the adversarial samples produced during the training and evaluation of the reinforcement learning agent. The surrogates achieve high label agreement with the targets using minimal interaction with the target models. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv

The Power of MEME: Adversarial Malware Creation with Model-Based Reinforcement Learning

Comprehending the laws and boundaries of nature enhances our skills as programmers. In recent years, a fascinating analogy has emerged, drawing parallels between the universe and a concept familiar to computer scientists: the discrete event simulator. The universe operates much like a massive simulation coordinated without a central clock ticking. Let's dive in... Understanding the Universe as a Discrete Event Simulator Reality operates in discrete increments of time and space. Rather than a continuous flow, events unfold at specific points in time, triggering state transitions within the cosmic fabric. This concept aligns with the principles of , a computational method used to model systems where events occur at distinct moments. discrete-event simulation Time advances in discrete steps, with events driving changes in system states with discrete units of time. The universe appears to operate on a fundamental level where time and space are quantized, meaning they consist of indivisible units. indivisible Researchers think that Planck units encapsulate this discretization of spacetime. Physicists named Planck units after the physicist . Max Planck Planck Units and the Fabric of Reality represent the smallest possible length, time, mass, and other physical quantities within the framework of quantum mechanics. Planck units Units are derived from fundamental constants such as the , Planck's constant, and the . speed of light gravitational constant The represents the smallest possible length scale, approximately . Planck length 1.616 × 10^-35 meters The , approximately , marks the smallest possible time interval, beyond which time loses its meaning. Planck time 5.391 × 10^-44 seconds Planck time is the time it takes for light to travel a distance equal to the Planck length in a vacuum. This minimal duration represents the shortest meaningful interval of time that can be defined within our current understanding of physics. At durations shorter than the Planck time, the concept of time itself becomes ambiguous, and the laws of physics as we know them break down. You can't break plank units in the same way you cannot break . fundamental particles Ancient Greeks created the notion of indivisible particles, which they termed "atoms." The term "atom" originates from the Greek word "atomos," where "a-" means "not" or "un-" and "tomos" means "cut" or "divided." Atoms are not the least divisible particle but the brilliant idea remains. Implications for Computer Simulations The analogy between the universe and a discrete-event simulator has profound implications for computer simulations and scientific inquiry. https://hackernoon.com/what-is-wrong-with-software-uh8j3y7k?embedable=true Once you acknowledge the between the two realms, you can gain new insights into the nature of reality and develop more efficient simulation techniques. parallels The concept of the universe as a discrete event simulator challenges traditional notions of computation. In the past, computer scientists had a lot of trouble dealing with infinities, problems, , and dealing with divisible units. precision floats For example, comparing two dates was always a problem since we usually represent them as arbitrary divisible points in time. Navigating non-infinitely divisible money amounts, distances, and other measurements has sparked numerous renowned challenges like the , the , the , etc. Genesis mission Mars Climate Orbiter Ariane 5 Flight 501 Conclusion The universe as a discrete event simulator offers a compelling framework for understanding the fundamental principles governing reality. Planck units set a barrier on where to stop diving the measurements. I am not telling you to model timestamps with plank units precision. I am asking you to change your framework and acknowledge we are not dealing with arbitrary precision problems but with finite precision ones. We should leave problems related to infinities only to math domains, not real-world simulators.

Warning: The Universe's Event Simulator Is a Fierce Adversary for Coders!

are advanced AI systems designed to understand and generate human-like text and have become a driving force behind numerous applications, from chatbots to content generation. Large language models However, sometimes they avoid giving you the output you need. Recent research has revealed a vulnerability in , including not just large language models but other types as well. This vulnerability is known as where input data is subtly manipulated to deceive models into producing incorrect results. deep learning models , adversarial attacks What Are Adversarial Attacs? Adversarial attacks involve the intentional manipulation of machine learning models through the changes in the input data. These attacks , leading to misclassifications or erroneous outcomes. find weaknesses in the models’ decision-making mechanisms How Do They Work? Adversarial attacks in AI work by making tiny, hidden changes to things like pictures or text that confuse AI systems. These changes are into making mistakes or giving biased answers. By studying how the AI reacts to these tricky changes, attackers can learn how the AI works and where it might be vulnerable. specially designed to trick the AI Can LLMs resist? LLMs are not immune to these adversarial attacks, often referred to as in the context of LLMs. Jailbreaking involves skillfully crafting prompts to exploit model biases and generate outputs that may deviate from their intended purpose. "jailbreaking" Users of LLMs have experimented with manual prompt design, creating anecdotal prompts tailored for very specific situations. In fact, a recent dataset called contained intentionally designed to challenge LLM capabilities. "Harmful Behavior" 521 instances of harmful behaviors So, I decided to test out a framework that automatically generates universal adversarial prompts. These prompts can be added to the end of a user's input. And this suffix can be used across multiple user prompts and potentially across various LLMs. Explaining the approach This approach operates as a meaning it doesn't go deeply into the inner workings of LLMs and is limited to inspecting only the model's outputs. This aspect is important because, in real-life situations, access to model internals is often unavailable. , black-box method The attack strategy is making a single adversarial prompt that consistently disrupts the alignment of commercial models, relying on the model's output. The result? It was successful. And it raises questions regarding the usability, reliability, and ethical aspects of LLMs, in addition to existing challenges, including: Distortions These occur when models generate responses that are inaccurate or don't align with user intentions. For instance, they might produce responses that attribute human qualities or emotions, even when it's not appropriate. Safety Large language models can unintentionally expose private information, participate in phishing attempts, or generate unwanted spam. When misused, they can be manipulated to spread biased beliefs and misinformation, potentially causing widespread harm. Prejudice The quality of training data significantly influences a model's responses. If the data lacks diversity or primarily represents one specific group, the model's outputs may exhibit bias, perpetuating existing disparities. Permission When data is collected from the internet, these models can inadvertently infringe on copyright, plagiarize content, and compromise privacy by extracting personal details from descriptions, leading to potential legal complications. Last thoughts We’ll see even more frameworks of LLM models in the future. Because they really help you get the outputs you need from a model. However, ethical concerns and sustainability will remain essential for responsible AI deployment and energy-efficient training. Adapting AI models to specific business needs can enhance operational efficiency, customer service, and data analysis. So, in the end, the potential of AI and LLMs will be democratized, and we’ll get even more access to these technologies in the future.

What Are Large Language Models Capable Of: The Vulnerability of LLMs to Adversarial Attacks

The Secret Adversary by Agatha Christie is part of the HackerNoon Books Series. Read this book online for free on HackerNoon! : The Secret Adversary Title : Agatha Christie Author : January 1, 1998 [eBook #1155] Most recently updated: October 29, 2022 Release date : English Language Table of Links PROLOGUE CHAPTER THE YOUNG ADVENTURERS, LTD. MR. WHITTINGTON’S OFFER A SET BACK WHO IS JANE FINN? MR. JULIUS P. HERSHEIMMER A PLAN OF CAMPAIGN THE HOUSE IN SOHO THE ADVENTURES OF TOMMY TUPPENCE ENTERS DOMESTIC SERVICE ENTER SIR JAMES PEEL EDGERTON JULIUS TELLS A STORY A FRIEND IN NEED THE VIGIL A CONSULTATION TUPPENCE RECEIVES A PROPOSAL FURTHER ADVENTURES OF TOMMY ANNETTE THE TELEGRAM JANE FINN TOO LATE TOMMY MAKES A DISCOVERY IN DOWNING STREET A RACE AGAINST TIME JULIUS TAKES A HAND JANE’S STORY MR. BROWN A SUPPER PARTY AT THE SAVOY AND AFTER About HackerNoon Book Series: We bring you the most important technical, scientific, and insightful public domain books. This book is part of the public domain. Agatha Christie (1998). The Secret Adversary. Urbana, Illinois: Project Gutenberg. Retrieved https://www.gutenberg.org/cache/epub/1155/pg1155-images.html This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at www.gutenberg.org , located at https://www.gutenberg.org/policy/license.html.

The Secret Adversary by Agatha Christie - Table of Links

How to fool a 27M-parameter model with a bit of Python Do you think it is impossible to fool the vision system of a self-driving Tesla car? Or that machine learning models used in malware detection software are too good to be evaded by hackers? Or that face recognition systems in airports are bulletproof? Like any of us machine learning enthusiasts, you might fall into the trap of thinking that deep models used out there are perfect. Well, you are WRONG. There are easy ways to build that can fool any deep learning model and create security issues. In this post, we will cover the following: adversarial examples What are adversarial examples? How do you generate adversarial examples? Hands-on example: let’s break Inception 3 How to defend your models against adversarial examples All the source code shown in this article is in ** **👇🏽 this Github repo Let’s start! 1. What are adversarial examples? 😈 In the last 10 years, deep learning models have left the academic kindergarten, become big boys, and transformed many industries. This is especially true for computer vision models. When hit the charts in 2012, the deep learning era officially started. AlexNet Nowadays, computer vision models are as good or better than human vision. You can find them in myriad places, including… self-driving cars face recognition medical diagnosis surveillance systems malware detection Until recently, researchers trained and tested machine learning models in a environment, such as in machine learning competitions and academic papers. Nowadays, when deployed in real-world scenarios, security vulnerabilities coming from model errors have become a real concern. laboratory Imagine for a moment that the state-of-the-art-super-fancy-deep learning vision system of your self-driving car was not able to recognize as a stop sign. this stop sign Well, . This stop sign image is an . Think of it as an optical illusion for the model. this is exactly what happened adversarial example Let us look at another example. Below you have two images of a dog that are indistinguishable for us humans. The image on the left is an original picture of a puppy taken by . The one on the right is a slight modification of the first, that I created by adding the noise vector in the central image. Dominika Roseclay Inception v3 correctly classifies the original image as a dog breed (a redbone). However, this same model thinks, with very high confidence, that the modified image I created is a paper towel. In other words, I created an adversarial example. And you will too, as this is the example we will work on later in section 3. An for a computer vision model is an input image with small perturbations, imperceptible to the human eye, that causes a wrong model prediction. adversarial example Do not think these 2 examples are rare edge-case examples found after spending tons of time and computing resources. There are easy ways to generate adversarial examples, and this opens the door to serious vulnerabilities of machine learning systems in production. Let’s see how you can generate an adversarial example and fool a state-of-the-art image classification model. 2. How to generate adversarial examples? Adversarial examples 😈  are generated by taking a clean image 👼 that the model correctly classifies, and finding a small perturbation that causes the new image to be misclassified by the ML model. White-box scenario Let’s suppose you have complete information about the model you want to fool. In this case, you can compute the loss function of the model where is the input image X is the output class, y and is the vector of the network parameters. θ This loss function is typically the negative log-likelihood for classification methods. Your goal is to find a new image that is close to the original and that produces a big change in the value of the loss function. X’ X Imagine you are inside the space of all possible input images, sitting on top of the original image . This space has dimensions , so I will excuse you if you cannot visualize it well 😜. X width x height x channels To find an adversarial example, you need to walk a little bit in some direction in this space until you find another image with a remarkably different loss. You want to choose the direction that maximizes the change in the loss function for a fixed small step . X’ J epsilon Now, if you refresh a bit your Maths Calculus course, the direction in the space where the loss function changes the most is precisely the gradient of with respect to the . X J X The gradient of a function with respect to one of its variables is precisely the direction of maximal change. And by the way, this is the reason people train Neural Networks using Stochastic Gradient Descend and not Stochastic Random-Direction Descend. Fast gradient sign method An easy way to formalize this intuition is as follows: We take only the sign of the gradient and scale it using a small parameter epsilon, to guarantee that the distortion between and is small enough to be imperceptible to the human eye. This method is called the . X X fast gradient sign method Black-box attack In most scenarios, it is very likely you will not have complete information about the model. Hence, the previous method is not useful as you cannot compute the gradient. However, there exists a remarkable property called of adversarial examples that malicious agents can exploit to break a model even if they do not know its internal architecture and parameters. transferability Researchers have repeatedly observed that quite well between models, meaning that they can be designed for a target model A, but end up being effective against any other model trained on a similar dataset. adversarial examples transfer Adversarial examples can be generated as follows: Query the targeted model with inputs X_i for i = 1 … n and store the outputs y_i. Use the training data (X_i, y_i) to build another model, called the substitute model. Use a white-box algorithm like the fast gradient sign to generate adversarial examples for the substitute model. Many of them are going to transfer successfully and become adversarial examples for the target model as well. A successful application of this strategy against a commercial Machine learning model is presented in . this Computer Vision Foundation paper 3. Hands-on example: let’s break Inception 3 Let’s get our hands dirty and implement a few attacks using Python and the great library PyTorch. It always comes in handy to know how the attacker thinks. You can find the complete code in . this Github repo Our target model is going to be Inception V3, a powerful image classification model developed by Google, which has around 27M parameters and was trained on around 14M images belonging to 20k categories. # Load pretrained model from the PyTorch hub
# https://pytorch.org/hub/pytorch_vision_inception_v3/
from torchvision.models import inception_v3
model = inception_v3(pretrained=True)
model.eval()

# Count model parameters: 27,161,264
n_params = sum(p.numel() for p in model.parameters())
print(f'{n_params:,} parameters') We download the list of classes the model was trained on, and build an auxiliary dictionary that maps class ids to labels. # Download the txt file with the list of ImageNet classes the model was trained with
#     "!" magic to run shell commands from a Jupyter notebook
!wget https://raw.githubusercontent.com/pytorch/hub/master/imagenet_classes.txt

# id2label maps classes ids to their human-readable names: e.g. id2label[1] = 'goldfish'
with open("imagenet_classes.txt", "r") as f:
    categories = [s.strip() for s in f.readlines()]
    
id2label = {}
for idx, category in enumerate(categories):
    id2label[idx] = category Let us take the image of an innocent redbone dog as the starting image we will carefully modify to build adversarial examples: import requests
import io
from PIL import Image

url = 'https://previews.123rf.com/images/meinzahn/meinzahn1211/meinzahn121100339/16350068-cute-tiger-cat-isolated-on-white.jpg'

response = requests.get(url)
img = Image.open(io.BytesIO(response.content))
img Inception V3 expects images with dimensions 299 x 299 and normalized pixel ranges between -1 and 1. Let us preprocess our beautiful dog image: import torch
from torch import Tensor
from torchvision import transforms

def preprocess(img) -> Tensor:
    """
    Inception V3 model from pytorch expects input images with pixel values between -1 and 1
    and dimensions 299 x 299
    """
    mean = [0.485, 0.456, 0.406]
    std = [0.229, 0.224, 0.225]

    preprocess_fn = transforms.Compose([
        transforms.Resize((299,299)),  
        transforms.ToTensor(),
        transforms.Normalize(mean, std)
    ])
    image_tensor = preprocess_fn(img)

    # add batch dimension: C x H x W ==> B x C x H x W
    image_tensor = image_tensor.unsqueeze(0)
    
    return image_tensor
        
x = preprocess(img) and check that the model correctly classifies this image. from easydict import EasyDict
import torch.nn.functional as F

def get_predictions(img: Tensor) -> EasyDict:

    output = model.forward(img)

    class_idx = torch.max(output.data, 1)[1][0].item()

    label = id2label[class_idx]

    output_probs = F.softmax(output, dim=1)
    confidence =  round(torch.max(output_probs.data, 1)[0][0].item(), 4)
    
    return EasyDict(
        id=class_idx,
        label=label,
        confidence=confidence,
    )

get_predictions(x) # {'id': 168, 'label': 'redbone', 'confidence': 0.8861} Good. The model works as expected and the dog is classified as a dog :-). redbone redbone Let’s move to the fun part and generate adversarial examples using the Fast Gradient Sign method. from typing import Tuple
from torch.autograd import Variable

def fast_gradient_sign(x: Tensor, eps: float) -> Tuple[Tensor, Tensor]:
    """"""
    # convert tensor into a variable, because we will need to compute gradients
    # of the loss function with respect to the image pixels
    img_variable = Variable(x, requires_grad=True)
    
    # forward pass on the original image
    output = model.forward(img_variable)
    
    # get predicted class
    y_true = torch.max(output.data, 1)[1][0].item()   
    target = Variable(torch.LongTensor([y_true]), requires_grad=False)
    
    # backward pass to compute gradients
    loss_fn = torch.nn.CrossEntropyLoss()
    loss = loss_fn(output, target)
    
    # this will calculate gradient of each variable (with requires_grad=True)
    # that you can later access with "var.grad.data"
    # PyTorch does the heavy lifting, computing the gradient of the cross-entropy
    # with respect to the input image pixels.
    loss.backward(retain_graph=True)
    
    # sign of gradient of the loss func (with respect to input X)
    x_grad = torch.sign(img_variable.grad.data)
    
    # fast gradient sign formula
    x_adversarial = img_variable.data + eps * x_grad
        
    return x_adversarial, x_grad

# keep epsilon small to generate slight changes to the original image
epsilon = 0.02
x_adv, grad = fast_gradient_sign(x, epsilon) I have created an auxiliary function to both the original and the adversarial image. You can see the full implementation in this . visualize GitHub repository Well. It is interesting how the model prediction changed for the new image, which is almost indistinguishable from the original one. The new prediction is a , which is another dog breed with very similar skin color and big ears. As the puppy in question could be a mixed breed, the model mistake seems to be small, so we want to work further to really break this model. bloodhound One possibility is to play with different values of and try to find one that clearly gives a wrong prediction. Let’s try this. epsilon epsilons = [0.02, 0.2, 0.9]
for epsilon in epsilons:
    x_adv, grad = fast_gradient_sign(x, epsilon)
    print('epsilon: ', epsilon)
    visualize(x, x_adv, grad, epsilon) As epsilon increases the change in the image becomes visible. However, the model predictions are still other dog breeds: and . We need to be smarter than this to break the model. bloodhound basset Remember the intuition behind the Fast gradient sign method, i.e. imagine yourself inside the space of all possible images (with dimension 299 x 299 x 3), right where the original image X is. The gradient of the loss function tells you the direction you need to step to increase its value and make the model less certain about the right prediction. The size of the step is epsilon. You step and you check if you are now sitting on an adversarial example. An extension of this would be to take many smaller steps, instead of a single one. After each step, you re-evaluate the gradient and decide the new direction you are going to walk towards This method is called the Iterative Fast Gradient method. What an original name! More formally: Where , and denotes clipping of the input in the range of [X−ϵ, X+ϵ]. X0 = X Clip X,ϵ An implementation in PyTorch is the following: def iterative_fast_gradient_sign(x_: Tensor, epsilon: float, n_steps: int, alpha: float):
    """"""
    # copy to avoid modifying the original tensor
    x = x_.clone().detach()
    
    for step in range(n_steps):
         
        # one step using basic FGSM
        x_adv, grad = fast_gradient_sign(x, alpha)
        
        # total perturbation
        total_grad = x_adv - x_
        
        # force total perturbation to be lower than epsilon in
        # absolute value
        total_grad = torch.clamp(total_grad, -epsilon, epsilon)

        # add total perturbation to the original image
        x_adv = x_ + total_grad
               
        x = x_adv
        
    return x_adv, total_grad Now, let us try again to generate a good adversarial example starting with our innocent puppy. : bloodhound again Step 1 : beagle again Step 2 : mousetrap? Interesting. However, the model confidence on this prediction is only 16%. Step 3 Let’s go further. : One more dog breed, boring… Step 4 : beagle again.. Step 5 : Step 6 : redbone again. Step 7 Keep calm and continue walking in the image space. : Step 8 : BINGO! 🔥🔥🔥 Step 9 What an exotic paper towel. And the model is pretty confident about its prediction, almost 99%. If you compare the initial and the final image we found in step 9 you see they are essentially the same, a , but for the Inception V3 model they are two very different things. puppy I was very surprised the first time I saw such a thing. How small modifications to an image can cause such model misbehavior. This example is funny, but if you think about the implications for a self-driving car vision you might start to worry a bit. Deep learning models deployed in critical tasks need to properly handle adversarial examples, but how? 4. How to defend your models against adversarial examples? As of November 2021, there is no universal defense strategy you can follow to protect against adversarial examples. In other words, attacking a model with adversarial examples is easier than protecting it. The best universal strategy, that can protect against any known attack, is to use adversarial examples when training the model. If the model adversarial examples during training, its performance at prediction time will be better for adversarial examples generated in the same way. This technique is called . sees adversarial training You generate them on the fly, as you train the model, and adjust the loss function to look both at clean and adversarial inputs. For example, we could eliminate the adversarial examples we found in the previous section if we added the 10+ examples to the training set and labelled all of them as . redbone This defense is very effective against attacks that use the Fast Gradient Sign method. However, there exist more powerful attacks, that we did not cover in this post, that can bypass this defense. If you wanna know more I encourage you to read by Nicholas Carlini and David Wagner. this amazing paper This is the reason why adversarial defense is an open problem in Machine Learning and cybersecurity. Conclusion Adversarial examples are a fascinating topic at the intersection of cybersecurity and machine learning. However, it is still an open problem waiting to be solved. In this post, I gave a practical introduction to the topic with code examples. If you would like to work further I suggest , a Python library for adversarial machine learning developed by the great Ian Goodfellow and Nicolas Papernot, and currently maintained by the . cleverhans University of Toronto All the source code I shown in this article is in . this Github repo Wanna become a PRO in Machine Learning? I offer hands-on content on real-world Machine Learning, that helps you get a top job in the Data world. 👉🏽  Subscribe to the . datamachines newsletter 👉🏽  Follow me on and . LinkedIn Twitter Have a great day 🧡 Pau

Adversarial Examples In Machine Learning Explained

The majority of application security problems stem from software bugs that leave the existing security controls, broken. However, even if the code is perfect it doesn’t mean an attacker can’t exploit it. The vulnerability can be hidden inside the business logic, not the code that powers it. This type of vulnerability is called . It’s when an attacker abuses a legitimate flow of an application so that it results in negative consequences. business logic vulnerability An example could be a contact form on a website that is used for sending out emails to the service owners. This form can be abused to send out spam messages instead of genuine support requests. When dealing with business logic vulnerabilities we enter a branch of security called anti-abuse. Fighting abuse often requires thinking about these problems more creatively, because unlike patching the bugs in the code the business logic often can’t be changed to mitigate the risk. We are stuck in a position where we can’t really the problem, we can only it. It requires active participation because an attacker will change the tactics as we put defense mechanisms in place. fix manage That's what makes it an . To better understand it let's look at the following diagram. adversarial problem So the defender directly controls stages 1 and 2, and the attacker directly controls 3 and 4. However, both of the adversaries control the entire cycle indirectly. The objective of the game for the attacker is to make sure that the attack doesn’t get detected. If it does, they need to be able to change the plans fast enough to try a different approach. The life of the defender is the same, but for different transitions. The defender wants to detect the attack and stage the defense as fast as possible. Translating that to the diagram the attacker wants for transitions 1 and 2 to be as long as possible and transitions 3 and 4 as short as possible. Contrary the defender wants transitions 3 and 4 take a very short time, but stretch out transitions 1 and 2. So, you may be wondering, how can it be applied in practice? 🤔 As a defender, it's easy to fall into the trap of spending too much time on intuitive solutions that don’t make the life of the attacker that much harder. I’ll give you an example. Let’s say we need to protect the contact form mentioned earlier from sending spam. We see that the attackers are using a bash script and that is reflected in the User-Agent header. Naturally you might think that we should block all attempts that come from this particular User-Agent. Since we don’t want the spam to be sent out we block the attempt right there on the page. However, what just happened is we gave the adversaries a very tight feedback loop to figure out our defense. To circumvent it they just need to spoof the User-Agent to look like a browser. It’s a simple change on their side and they are back in business. Not good for us, now it got harder to detect them. The attackers get stronger as we fight them. Keeping in mind the helps us avoid first-order-thinking and evaluate the consequences of an intuitive solution. adversarial cycle Co-published here.

How Does the Adversarial Cycle Apply to Coding and Security? 

is concerned with the design of ML algorithms that can resist security challenges. Adversarial machine learning Adversarial Machine Learning states that there are that ML models can suffer. four types of attacks Extraction attacks In a model , an adversary steals a copy of a remotely deployed machine learning model, given oracle prediction access. extraction attack It is produced by making requests to the target model with inputs to extract as much information as possible and with the set of inputs and outputs train a model called substitute model. , the attacker needs a huge compute capacity to re-training the new model with accuracy and fidelity, and substitute model is equivalent to training a model from the ground up. Extract model is hard Defenses Limit the output information when the model classifies a given input. Differential Privacy. Use . ensembles Proxy between end-user and model like . PRADA Limit the number of requests. Inference attacks Inference attacks aim to reverse the information flow of a machine learning model. They allow an adversary to have knowledge of the model that was not explicitly intended to be shared. Inference attacks pose severe privacy and security threats to individuals and systems. They are successful because private data are statistically correlated with public data, and ML classifiers can capture such statistical correlations. Includes three types of attacks: Membership Inference Attack (MIA). Property Inference Attack (PIA). Recovery training data. Defenses Use advanced cryptography. Differential cryptography. Homomorphic cryptography. Secure Multi-party Computation. Techniques such as Dropout. Model compression. Poisoning attacks This technique involves an in the training dataset to compromise a target machine learning model during training. attacker inserting corrupt data Some data aim to trigger a specific behavior in a computer vision system when it faces a specific pattern of pixels at inference time. Other data poisoning techniques aim to reduce the accuracy of a machine learning model on one or more output classes. poisoning techniques This attack is difficult to detect when performed on training data since the attack can propagate between different models using the same data. The adversary seeks to destroy the availability of the model by modifying the decision boundary and, as a result, producing incorrect predictions. Finally, the attacker could create a .  The model behaves correctly (returning the desired predictions) in most cases, except for certain inputs specially created by the adversary that produce undesired results. The the results of the predictions and launch future attacks. backdoor in a model adversary can manipulate Defenses Protect the integrity of training data. Protect the algorithms, use robust methods to train models. Evasion attacks An (in the form of noise) into the input of a machine learning model to make it (example adversary). adversary inserts a small perturbation classify incorrectly They are similar to poisoning attacks, but their main difference is that evasion attacks try to exploit weaknesses of the model in the inference phase, not in the training. The attacker’s knowledge of the target system is important. The more they know about your model and how it’s built — the easier it is for them to mount an attack on it. An evasion attack happens when the network is fed an “adversarial example” — a carefully perturbed input that looks and feels exactly the same as its — but that completely . untampered copy to a human throws off the classifier Defenses Training with adversarial examples which robust the model. Transform the input to the model (Input sanitization). Gradient regularization. Tools Adversarial Robustness Toolbox (ART) is a Python library for Machine Learning Security. ART provides tools that enable developers and researchers to defend and evaluate Machine Learning models and applications against the adversarial threats of , , , and . Adversarial Robustness Toolbox (ART) Evasion Poisoning Extraction Inference ART supports all popular machine learning frameworks: TensorFlow Keras PyTorch scikit-learn All data types: Images Tables Audio Video And machine learning tasks: Classification Object detection Speech recognition pip installation pip install adversarial-robustness-toolbox Attack example from art.attacks.evasion import FastGradientMethod
attack_fgm = FastGradientMethod(estimator = classifier, eps = 0.2)
x_test_fgm = attack_fgm.generate(x=x_test)
predictions_test = classifier.predict(x_test_fgm) Defense example from art.defences.trainer import AdversarialTrainer
model.compile(loss=keras.losses.categorical_crossentropy, optimizer=tf.keras.optimizers.Adam(lr=0.01), metrics=["accuracy"])
defence = AdversarialTrainer(classifier=classifier, attacks=attack_fgm, ratio=0.6)
(x_train, y_train), (x_test, y_test), min_pixel_value, max_pixel_value = load_mnist()
defence.fit(x=x_train, y=y_train, nb_epochs=3) Counterfit is a command-line tool and generic automation layer for assessing the security of machine learning systems. Counterfit Developed for security audits on ML models. Implements black box evasion algorithms and based on ART and TextAttack. Command list ---------------------------------------------------
Microsoft
                          __            _____ __
  _________  __  ______  / /____  _____/ __(_) /_
 / ___/ __ \/ / / / __ \/ __/ _ \/ ___/ /_/ / __/
/ /__/ /_/ / /_/ / / / / /_/  __/ /  / __/ / /
\___/\____/\__,_/_/ /_/\__/\___/_/  /_/ /_/\__/

                                        #ATML

---------------------------------------------------

list targets

list frameworks

load <framework> 

list attacks

interact <target>

predict -i <ind>

use <attack>

run

scan Final words and by "If you use machine learning, there is the risk for exposure, even though the threat does not currently exist in your space." "The gap between machine learning and security is definitely there." Hyrum Anderson, Microsoft ## References Towards Security Threats of DL Systems Adversarial Matrix Mitre Poisoning attacks Evasion attacks Thanks Special thanks to , co-writer of this article. @jiep This article was first published . here

Adversarial Machine Learning: A Beginner’s Guide to Adversarial Attacks and Defenses

PROBLEM STATEMENT Marine plastic pollution has been at the forefront of climate issues for the past decade. Not only does plastic in the ocean capable of killing marine life by strangulation or starvation but it is also a major factor in warming the ocean by trapping CO2. In recent years, there have been numerous attempts at cleaning the plastic that has been circling our oceans such as by the non-profit group, The Ocean Cleanup. The problem with a lot of cleanup processes is that it requires human labor and is not cost-efficient. There has been a lot of research done to automate this process by using Computer Vision and Deep Learning to detect marine debris to utilize ROVs and AUVs for cleanup. Note: This is theoretical and we are working on publishing our paper. Once it gets peer-reviewed and gets published in a journal I will update this article. So, watch this space for the release of our academic paper! : , , and Reading Resources Our Paper on Using CV and DL to detect Epipelagic plastic The University Of Minnesota's paper on a general-purpose marine debris detecting algorithm The Ocean Cleanup’s paper on floating plastic detection on rivers. The major issue for this approach is regarding the availability of datasets to train the computer vision models. The JAMSTEC-JEDI dataset is a good collection of Marine Debris off the coast of Japan on the seafloor. But, apart from this dataset, there is a huge discrepancy in the availability of datasets. For this reason, I’ve utilized the help of Generative Adversarial Networks; The DCGAN in particular to curate synthetic datasets that could theoretically be a close replica to real datasets over time. (I wrote theoretically because there are a lot of variables such as quality of real-images available, progress with GANs, training, etc). GAN and DCGAN GANs or Generative Adversarial Networks were proposed by Ian Goodfellow et al in 2014. GANs consist of a simple 2 components called Generators and Discriminators. An oversimplification of the process is as follows: The Generators role is to generate new data and the Discriminators role is to distinguish between the generated data and the actual data. In the ideal scenario, the discriminator is unable to distinguish between the generated data and real data which results in ideal synthetic data points. DCGAN is a direct extension of the GAN architecture mentioned above except it uses Deep Convolutional Layers in the discriminator and generator respectively. It was first described by Radford et. al in the paper . The discriminator is made up of strided convolutional layers while the generator is made up of convolutional transpose layers. Unsupervised Representation Learning With Deep Convolutional Generative Adversarial Networks GAN architecture (source: ) Here PyTorch Implementation In this method, we are going to be applying the DCGAN architecture on the dataset. If you’re not familiar with the DeepTrash dataset, consider reading my paper . DeepTrash is a collection of plastic images in the epipelagic layer and abyssopelagic layer of the ocean curated for marine plastic detection using computer vision. DeepTrash A Robotic Approach towards Quantifying Epipelagic Bound Plastic Using Deep Visual Models Let's begin coding! Installing Requirements We start by installing all the basic requirements to building our GAN model such as Matplotlib and Numpy. We will also be utilizing all the tools (such as neural networks, transformations) from Pytorch. If you’re not familiar with PyTorch I recommend giving this article a read: Initializing our Hyperparameters This step is fairly straightforward. We’re going to set up the hyperparameters that we want to train the Neural Network with. These hyperparameters are borrowed straight from the paper and PyTorch’s tutorial on training GANs. Generator and Discriminator Architectures Now, we define the architectures for the generator and discriminator. Defining the Training Function After defining the Generator and Discriminator classes, we move on to defining the training function. The training function takes in the generator, discriminator, optimization functions, and the number of epochs as the arguments. We train the generator and discriminator by recursively calling the train function until the required number of epochs. We do this by iterating through the dataloader and updating the discriminator with new images from the generator and calculating and updating the loss function. Monitoring and Training the DCGAN After we’ve established the Generator, Discriminator, and Train functions the final step is to simply call the train function for the number of epochs we defined. I’ve also used which allows us to monitor our training. Wandb Results We plot the loss incurred by the generator and discrimination during training. IMG - Generator and Discriminator Loss during Training We can also pull up an animation of how the generator was generating images to see a difference between the real and the fake images. Which looks something like this: And the final resulting image after training (the same image as above). CONCLUSION In this article, we discussed the use of Deep Convolution Generative Adversarial Networks to generate synthetic images of marine plastic that can be used by researchers to extend their current dataset of marine plastic. This can help with giving researchers the ability to extend their dataset with a mix of real and synthetic images. As you can see with the results, the GAN still needs a lot of work. The ocean is a complex environment with varying illumination, turbidity, blur, etc. But this is a theoretical starting point that other researchers can build off of. If you’re interested in improving the results and building a better network architecture, please reach out to me at gautamtata.blog@gmail.com.

Synthesizing Images of Marine Plastic Using Deep Convolutional Generative Adversarial Networks

Deep learning is the main force behind the recent advances in the field of artificial intelligence (AI). Deep learning models are capable of performing on par with, if not exceeding, human levels, at a variety of different tasks and objectives. However, deep neural networks are vulnerable to subtle adversarial perturbations applied to their inputs – adversarial AI. These adversarial perturbations, which can be imperceptible to the human eye, can easily mislead a trained deep neural network into making wrong decisions. The field of adversarial machine learning focuses on addressing this problem by developing high-performing deep learning models that are also robust against this type of adversarial attack. At Modzy, we’re conducting cutting-edge research to improve upon past approaches that defend against adversarial attacks, ensuring our models maintain peak performance and robustness when faced with adversarial AI. What you need to know Although impressive breakthroughs have been made by leveraging deep neural networks in many fields such as image classification and object detection, Szegedy et al. [1] discovered that these models can easily be fooled by adversarial attacks. For example, an image manipulated by an adversary with only a few modified pixels can easily fool an image classifier into confidently predicting the wrong class for that image (Figure, [2]). This exposes an unpleasant fact which is that deep learning models do not process information in a manner that is similar to humans. This phenomenon undermines the practicality of many current deep learning models that are trained solely for accuracy and performance, but not for robustness against these types of attacks. The research community is quite active in pursuing possible solutions to this problem. On the adversarial side, many attacking methods which utilize the vulnerabilities of trained deep neural networks have been proposed [2,3]. On the defense side, these attacking schemes are used to propose new training and design methodologies for deep neural networks in order to produce deep learning models that are relatively more robust against adversarial AI attacks. As an example, adversarial training has been proposed as a possible solution to enhance robustness. Adversarial training involves training a deep neural network on a larger training dataset that includes both original and adversarially perturbed inputs [2]. However, due to a lack of understanding of the adversarial phenomena described above, none of the current solutions proposed in the research community are capable of addressing this problem in a generalizable sense across different domains. Adversarial AI attacks can be divided into two categories: white-box attacks black-box attacks Mathematically speaking, all deep neural networks are trained to optimize their behavior in relationship to a specific task, such as language translation or image classification. During training, this desired behavior is usually formulated as an optimization problem that minimizes a loss value according to a specific formula that measures deviations from the desired behavior. The adversarial examples are inputs that do the opposite. They maximize this loss value and consequently maximize deviations from the desired behavior. Finding these adversarial examples requires knowledge of the inner workings of the deep neural network. The strong assumption under the white-box attack framework is that the adversary has full knowledge of the inner workings of the deep neural network and can utilize this knowledge to design adversarial inputs. Under the black-box attack framework, the adversary has a limited knowledge of the architecture of the deep neural network and can only estimate the behavior of the model and devise adversarial examples based on its estimation. Another type of attack, a poisoning attack, purely focuses on manipulating the training dataset so that any deep learning model trained dataset using it yields a sub-par performance during inference. A New Understanding of Adversarial AI Attacks At Modzy, we developed a on deep neural networks by utilizing the Lyapunov Theory of Robustness and Stability of Nonlinear Systems [4, 5]. Our robust solutions are based on this theory, which dates back further than a century and has been extensively used in the field of control theory to design automated systems such as aircrafts and automobile systems; the expectation is that these systems should be stable, robust, and able to maintain the desired performance in unknown environments. new understanding of adversarial AI attacks Our robust deep learning models, tested against strong white-box attacks, are trained with a similar expectation, so that they can make correct predictions in unknown environments and under the possibility of adversarial attacks. We also train our deep learning models in a novel way by enhancing the well-known backpropagation algorithm commonly used across industry to train deep learning models. Our robust models are trained to rely on a holistic set of features learned from the input when making predictions. For example, our image classifiers look at the context of the entire image before classifying an object. This means that modifying a few pixels will not affect the final classification decisions made by our models. Consequently, our robust models closely imitate how humans learn and make decisions. Adversarial attacks on deep neural networks pose a great risk to the successful deployment of deep learning models in mission-critical environments. One challenging aspect of adversarial AI is the fact that these small adversarial perturbations, while capable of completely fooling the deep learning model, are imperceptible to the human eye. The increasing reliance on deep learning models in the field of artificial intelligence further points to the adverse impact that adversarial AI can have on our society. We take this risk seriously and are actively developing new ways to enhance the defensive capabilities of our models. Our robust deep learning models guarantee high performance and resilience against adversarial AI and are trained to be deployed into unknown environments. References: Szegedy, Christian, et al. “Intriguing properties of neural networks.” arXiv preprint arXiv:1312.6199 (2013). Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. “Explaining and harnessing adversarial examples.” arXiv preprint arXiv:1412.6572 (2014). Madry, Aleksander, et al. “Towards deep learning models resistant to adversarial attacks.” arXiv preprint arXiv:1706.06083 (2017). Arash Rahnama, Andre T. Nguyen, and Edward Raff, “Connecting Lyapunov Control Theory to Adversarial Attacks,” ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2019. Arash Rahnama, Andre T. Nguyen, and E. Raff, “Robust Design of Deep Neural Networks Against Adversarial Attacks Based on Lyapunov Theory,” IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

What are Adversarial AI Attacks and How Do We Combat Them?

In our previous article, , i.e. attacks based on minor modifications of the input. we showed that neural networks are vulnerable to carefully crafted adversarial attacks While those perturbations are often invisible to a human observer or perceived as noise that does not influence the recognition, they can fool even the most successful . Thus, it is of great importance to be able to defend against such attacks, especially in safety-critical environments. state-of-the-art neural network based systems Multiple techniques have been proposed over the last years to achieve this, including but not limited to: adversarial training, image preprocessing or transformation, and feature denoising. The most popular defense strategy is adversarial training. In this method, images are adversarially attacked with a chosen approach and perturbed examples are introduced to the model during the training process so that it learns to classify them correctly [1,2]. Another group of methods involves image preprocessing. Since perturbations are often perceived as noise to a human observer, several approaches based on transforming or preprocessing input images before they are fed to the neural network have been introduced [3,4]. Additionally, Xie et al. [5] proposed feature denoising as a method of defense against adversarial attacks based on the observation that perturbations in images cause noise in the feature maps. In today’s article, we will present two state-of-the-art methods of defense against adversarial attacks. Adversarial training Adversarial training was first introduced by Szegedy et al. [1] and it is currently the most popular technique of defense against adversarial attacks. This method involves training a network on a mixture of adversarial and clean examples. Goodfellow et al. [6] followed them and trained a network using adversarial loss Ladv(x): where L(x) is the loss function computed on the original image and L(xperturbed) is the loss function computed on an adversarially perturbed image (using e.g. the fast gradient sign method described in our last post) and ɑ is a hyperparameter assigning weights to the components of the adversarial loss function. Multiple improvements have been proposed to the regular adversarial training. For example, Harini et al. [6] added an adversarial logit pairing component to the loss function, which encourages the logits (outputs from the network before applying softmax normalization function) obtained from two images (original and adversarial) to be similar. The intuition behind this idea is that both clean and perturbed versions of the same example should be assigned to the same class so big differences between their logits are penalized. Feature denoising Xie et al. [5] recently proposed another method of defense against adversarial attacks, namely, feature denoising. Authors showed that while adversarial perturbations are often barely perceptible in the pixel space, they cause a significant noise in the feature maps. As the image is passed through the network, the perturbation increases and not-existing activations of the feature maps arise, which leads to wrong predictions. Below feature maps of original and perturbed images are presented: Feature map calculated for an original image (top) and adversarially perturbed version (bottom). On the adversarial image, activations are present in regions not relevant to the prediction. Adapted from Ref. [5]. Based on this observation, the authors developed a new method of defense against adversarial attacks, namely, they added multiple blocks for feature denoising to the neural network and trained it end-to-end on adversarial examples. The authors utilized ResNet [7] with 4 denoising blocks. Below adversarial images are presented, together with the feature maps before (left) and after (right) applying denoising: Adversarial images are presented, together with the feature maps before (left) and after (right) applying denoising. Adapted from Ref. [5]. As can be seen, noise is substantially reduced as a result of denoising operation. Adding feature denoising during adversarial training improved the accuracy of classification on adversarially attacked images. The method won first place in the Competition of Adversarial Attacks and Defences (CAAD) 2018 [8]. To sum up Neural networks’ vulnerability to adversarial attacks casts doubt on their usability in the real-world environment, especially in safety-critical systems, like, for example, autonomous cars and medicine. Thus finding an effective defense strategy is of great importance. However, there are a few challenges that we have to face when applying the defense method. For example, weak performance on adversarial examples produced with attack approaches that haven’t been used during adversarial training is a common issue. Additionally, drop-in classification accuracy on unperturbed images may happen. Literature [1] Szegedy, Christian, et al. “Intriguing properties of neural networks.” arXiv preprint arXiv:1312.6199 (2013) [2] Madry, Aleksander, et al. “Towards deep learning models resistant to adversarial attacks.” arXiv preprint arXiv:1706.06083 (2017) [3] Xie, Cihang, et al. “Mitigating adversarial effects through randomization.” arXiv preprint arXiv:1711.01991 (2017) [4] Das, Nilaksh, et al. “Keeping the bad guys out: Protecting and vaccinating deep learning with jpeg compression.” arXiv preprint arXiv:1705.02900 (2017) [5] Xie, Cihang, et al. “Feature denoising for improving adversarial robustness.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019 [6] Kannan, Harini, Alexey Kurakin, and Ian Goodfellow. “Adversarial logit pairing.” arXiv preprint arXiv:1803.06373 (2018) [7] He, Kaiming, et al. “Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016 [8] http://hof.geekpwn.org/caad/en/index.html Also published on: https://neurosys.com/article/defense-strategies-against-adversarial-attacks/

An Introduction to Adversarial Attacks and Defense Strategies

Can you fool Artificial Intelligence? No? Think again. Three years ago, Apple launched IphoneX with cutting-edge facial recognition technology. This advanced AI technique (Face ID) replaced the old fingerprint recognition technology (Touch ID). The latest technology claimed to be more secure and robust. However, shortly after the launch of Face ID, researchers from Vietnam breached it by designing a 3D face mask. Such crafted attacks against ML-based AI systems come under the umbrella of a fascinating research field: adversarial machine learning. This attack against Face ID led to new debates in the field of cybersecurity: How secure are AI systems against adversarial attacks, and whether such attacks are practical in real-world scenarios or not. In this article, we will explore the answers to these questions. We will further explore adversarial machine learning beyond cybersecurity — its application in AI software testing. But first, let’s understand what is adversarial machine learning. What is Adversarial Machine Learning? You are familiar with optical illusions. They play with our minds into seeing things that are not there. In adversarial machine learning, attackers design optical illusions for machine learning algorithms. For example, say an AI system classifies apples and oranges. Now, look at the picture below. You can see that both are images of the same apple. However, the image of the apple on the right is crafted a little differently. An attacker has added a tiny perturbation in this image, making the ML algorithm misclassify it to orange. The human eye cannot see any difference between the left and right apple. However, the AI system is fooled to see it as an orange. Apple vs Adversarial Apple The above illustration is inspired by the panda example given in this by the guru of adversarial machine learning: Ian Goodfellow. This example seems pretty straightforward to understand right? There can be even simpler ways to sabotage AI systems. paper There are also highly sophisticated attacks designed to evade machine learning and artificial intelligence systems. In research, such attacks have proven to sabotage a variety of safety-critical systems, including and . Let’s have a look at them: self-driving cars medical imaging systems Fooling AI: Evasion Attacks Photo by Joshua Hoehne on Unsplash In evasion attacks, attackers design adversarial examples that fool the AI model. For example, attackers can target or paint on road signs. self-driving cars by using stickers The sign appears the same to the naked eye; however, a self-driving car may interpret it differently. This may cause deadly accidents. The example of the Face ID breach given at the start of this article is also a type of evasion attack. There can be a variety of evasion attacks — from simple image manipulation to sophisticate white-box attacks. Poisoning AI: Data Poisoning Attacks Photo by Markus Spiske on Unsplash In poisoning attacks, the threat actor can add incorrect examples in the training data, making a faulty, mutant model. The examples apparently seem harmless, and such attacks are usually launched on self-learning or reinforcement learning systems. For example, back in 2016, an interactive, self-learning ! chatbot Tay was poisoned (trained) by the users into a racist Nazi Rethinking AI Security and Software Testing Image Source: InvoZone Adversarial machine learning is also being used by software engineers to test the robustness of AI systems. The researchers are combining to generate and evaluate the test data. software testing techniques with adversarial machine learning Humans fight with the idea that whether seeing is believing or believing is seeing. In the case of ML-based AI, learning is seeing and learning is believing. It is quite tricky how most of the complex AI systems learn. The key to secure and robust AI systems is thorough testing. Also published here .

Adversarial Machine Learning and Its Role in Fooling AI

Material scientists often face the challenge of figuring out how to effectively search the vast chemical design space to locate the materials with their desired properties. To address this challenge, many scientists have turned to artificial intelligence in the race to discover new and advanced materials. Leveraging GANs for Efficient Sampling of Chemical Space A general adversarial network is a variety of machine learning framework that leverages the idea of " " where a network is trained on adversarial examples. It is an idea that originates from game theory and introduced to the machine learning community in 2014 by Ian J. Goodfellow. A GAN is comprised of three parts: adversarial training A for generating new data generator model A for classifying the generated data discriminator model The that pits the two models against each other. adversarial network With this in mind, a targeted strategy for developing novel chemical compositions is to develop sampling algorithms that can exploit both explicit chemical knowledge and implicit rules of composition embodied in a large database of materials. The proposed outcome would be a generative machine learning model based on a generative adversarial network (GAN) for efficient generation of new hypothetical inorganic materials. The use of GANs can be leveraged to generate hypothetical materials not existing in the training dataset, reaching a novelty of 92.53% when generating 2 million samples. Conclusion The implications of general adversarial networks extend far beyond the applications in material science, but I have to admit that this is one of the use cases I have really would have thought of applying AI to. If you think this is cool, It's a site featuring GAN generated chemicals that Pretty neat. Thanks for reading! you should definitely take a look at the site that inspired me to write about this topic! do not exist.

Leveraging General Adversarial Networks for Material Sciences

GAN or Generative Adversarial Network is one of the most fascinating inventions in the field of AI. All the amazing news articles we come across every day, related to machines achieving splendid human-like tasks, are mostly the work of GANs! For instance, if you ever heard of AI bots which create human-like paintings, it is essentially GANs behind the awe-inspiring strokes. Or if you have heard of AI bots which create human faces from scratch, faces which do not even exist yet, that too is entirely the imaginative work of powerful GANs. “GANGogh”: Paintings of flowerpots by GANs; Courtesy: towardsdatascience GANs have a lot of applications and one is often led to wonder how simple machines can achieve such fascinating and in fact, extensively creative accomplishments so efficiently. If you are an observer of the real world, you might have noticed that an individual, whether it be an individual from the animal or plant kingdom, often grows stronger when it faces any sort of competition. A seed which beats its siblings by being more absorbent to the nutrients and water in the soil, grows into a strong and healthy tree, increasing the chances of stronger descendants as the generations proceed. A boxing contender estimates the quality of his/her fellow contenders to prepare accordingly. The stronger the contenders, the higher is the quality of preparation. Almost mimicking the real world, GANs follow suit and go by the architecture of . Adversarial training is nothing but “learning by comparison”. The information possessed by the adversary is studied such that other instances of the information of the same category can be furnished with least flaws such that the adversary cannot detect that the information is from an alien source. adversarial training There are two counterparts in a GAN: the generator and the discriminator. The work of the generator is to tweak the outputs such that they are indistinguishable from the originals. The work of the discriminator is to judge if the generated piece is up to mark and can be classified at par with the already present samples. Otherwise, the cycle is repeated unless the generator generates a good enough image which the discriminator passes as a credible sample. GAN Architecture; Courtesy: GeeksforGeeks For instance, if a GAN is trained on a set of poems, it should be able to come up with poems of its own. However, as splendid as it sounds, training GANs is not an easy task. There are several ongoing research initiatives to identify ways to minimize the flaws in the generated data. Applications of GANs are manifold and among these, the healthcare industry has also benefited majorly. For instance, terenz.ai uses GANs effectively to provide super-resolution to medical imaging. Radiography, ultrasound, elastography, magnetic resonance imaging and several other such salient tests generate varied images which often needs refining, especially if the imaging equipment is of relatively low quality. Courtesy: terenz.ai Imagine using a GAN to provide super-resolution to say, radiography reports/images. This can be done by training the GAN with several previously acquired high-resolution radiography images. This would mean that the information from high-quality radiography imaging equipment can be leveraged even in areas where a relatively poorer quality of the equipment is at disposal. To use the GAN architecture, usually, the resolution of high-quality images is brought down to low resolution and fed to the generator. The generator tries to increase the resolution such that the discriminator passes it as a credible image. The cycle continues unless the loss of the generator function, with respect to the discriminator function (or adversarial loss), is minimized. It requires a high degree of patience to train GANs to attain super-resolution images because it has to learn to provide both detailed finer structures as well as larger structures at the same time. This makes training the generator difficult since it becomes very easy for the discriminator to distinguish tiny differences in the finer structures. For this, an optimum convergence point must be attained such that both generator and discriminator do not allow the demise of the other. Indeed, it is a very fine process, but once achieved, it has the ability to change the face of not only healthcare imaging but also a variety of use cases in major sectors worldwide.

Generative Adversarial Networks (GANs): An Overview

GANs or Generative Adversarial Networks: Learn By Building One

The machines have been trying to learn to recognize and identify the photos they have seen for years. In 2013, it succeeded in reaching the human level. Machine learning systems have provided simple output from a complex input. It can detect almost all details of a photos and display users exactly want they want. On the other hand, sometimes these algorithms can be trapped. While machine learning algorithms can easily recognize an apple, if you put the same apple in a net bag, it doesn't understand what it is. Because this is an unexpected, unusual visual. In 2014, Ian Goodfellow and his colleagues at the University of Montreal in Canada developed a new machine learning system using game theory to turn this weakness into an advantage. The coolest idea in machine learning in the last 20 years. -Facebook's artificial intelligence research manager Do you know why it is so important? Machine learning systems have provided simple output from a complex input. For instance, you are uploading photos in social media which are inputs. The ML algortihms produces a simple output using its neural networks. After analyzing, it produces output from these photos by detecting objects in photo. So, when you search something on internet, machine can easily find the keyword by filtering tons of data with produced tags. The newly developed "Generative Adversarial Networks" can do the opposite. It can produce complex output from simple input. If you give the computer random numbers, it creates extremely complex and realistic photographs of human faces. So the machine not only learns, but also produces. They are learning by producing! There is a productive network and it starts to draw the picture immediately from random noise. But there is also another network which examines images created by the productive network. We can compare these two neural networks to the opponents in a game. There's a constant struggle between them. The producer's goal in this game is to trick the discriminator and to convince him that the image he produces is real. The aim of the discriminator is to extract as many fake images as possible by looking at real images. You can play with Generative Adversarial Networks (GANs) in your browser by clicking link below: GAN Lab By competing against each other, they can train each other to make the random points a straight line or a circle. As you will notice, the computer requires thousands of experiments to learn such simple drawings. Imagine that you gave real human photographs instead of the above example line or circle. In 2014, when this system was developed, it produced a very low resolution with black and white colors. In last 5 years, the quality of photography that artificial intelligence can synthesize has gradually increased. In these 5 years, machine learning researchers both developed and started to apply GAN techniques in many different fields. For example, computers can now produce cartoon characters and anime characters. The PokeGAN project designs new Pokemon characters, looking at old characters. The CycleGAN project turns draft drawings into photographs. It can learn the styles of painters and turn photographs into paintings or satellite photographs into maps. The StackGAN project turns text into visual. For example, you write ”a little bird with a short beak, black and blue" and it produces such a bird image. It is not searching it but actually producing it. In fact, these birds do not exist. What caused to such great progress in such a short time? Because it's not just about the advancement of technology. Shows the amount of data generated on the Internet. Anything you notice? Since 2010, we are experiencing an explosion in the data we produce. This is one of the main reasons why GAN technique has been increasingly successful in machine learning since 2014. In the last 5 years, we have increased the number and variety of sample data by uploading the photos and videos we take to the internet and social media platforms. Well, how we are producing so much data? The reason is very simple. For instance, FaceApp that ages people's photo. Only one such application allows hundreds of millions of photos a week to be uploaded. Let's hope these are only used for machine learning. Because we accepted the terms and conditions without reading it to use the application , we give all the rights of those photographs to Yaroslav Goncharov who developed the application. FaceApp is just one of the applications that uses machine learning and GAN technique. Nowadays, producer neural networks perceive images as a set of styles. It can learn the pose, posture, hair, face shape, eyes, skin color of a person separately and produce new photos. A Style-Based Generator Architecture for Generative Adversarial Networks Before FaceApp, which was used by almost everyone, there was FakeApp that some people know. FakeApp can combine videos of celebrities with GAN techniques and make them do things they never did. Deepfake Videos In order to produce Deepfake videos, the machines need to learn again with the GAN technique. Although the examples in link are impressive, they are actually in the process of combining two existing images. A few months ago, with a technique developed in Samsung's artificial intelligence laboratories, it became possible to produce video from a single photo. Few-Shot Adversarial Learning of Realistic Neural Talking Head Models Normally, we said that a lot of data is needed in machine learning. It is enough to give the computer one visual to learn. The computer combines this with basic facial gestures to produce a simple video. Probably in the near future we will be able to produce automatic videos from the selfies we take with our mobile phone. Or we can turn on our camera and play a photo like a puppet with our own facial expression. That's all for now! Make sure you are following me on social media and if you find it useful please share with your friends. See you in next post soon Hackers! Reverse Python Stay Connected!

Stories, News & Mentions about ADVERSA

Adversa’s Stories on HackerNoon