Vegvisits

vegvisits

Finally, ELIZA: A Platform, Not a ChatBot!

Between the Digital Marketing course and the coaching program I’m doing atm, I haven’t had the time to sit down and write a proper article. Instead, here’s a medley of thoughts about Love, The Mind, and Being that have been bouncing around in my head. What Is Love? , rings a remix of the ‘90s band Haddaway’s single on my Instagram feed. What is love? Baby don’t hurt me don’t hurt me no more So, what love? Is it when a baby doesn’t hurt you? is The love we hear about in songs or see in movies isn’t love. It’s an attachment. Attachment and exploitation of a love object. What can the object give me? Attention, time, sex… but in the real world, we soon find the other person can’t do that for us, and the fear creeps in—the fear of losing the object that provides the illusion of love. Instead of facing the reality of unfulfilled expectations, we simultaneously try to fix ourselves so that the person finds us loveable and adjusts the person to fit our preferred criteria. , then we’d be happy; we would feel loved. We get tricked into believing love can be outsourced. So, the fear of losing the love object that provides the illusion of love begins to enter. When we don’t love ourselves, looking for love on the outside always gives us anxiety. “Will they like me?” is the focus, whilst the real question is do I like myself? If they were just a little more to our liking . . Love is a state, not a feeling. Love is independent of love objects. Love is free. It is always there, patiently waiting to be uncovered. It’s a state you began in and will die in. Love is an energy that cannot be created or destroyed. You don’t need to generate love, find love, or get love. It can only be gently uncovered behind the layers of conditioning put upon your heart. The paradox is that no one can give you love - you cannot get love; you can only LOVE. This is an important distinction ❤️ Stuff Like This ; your mind whispers when you dare to dream beyond the confines you’ve been placed within. But what if it does happen? What if it’s right there, waiting for you to see through the lies of your mind? What if all sorts of things happen to all sorts of people, including you? Your mind is just telling you a story based on past experiences. It pretends to grasp the entirety of everything: how the world operates, what other people are like, and what you are like. (Seriously, ask your mind anything, and it’ll come up with a response! - how silly is that? It’ll never say, gosh, I don’t know, and leave it at that. It’ll begin wondering and try to figure out a response to soothe itself that it does know everything). It claims to have all the answers when, in reality, it’s barely scratching the surface. Stuff like this doesn’t happen to people like me We cling to the illusion of self-knowledge, yet the truth is we don’t fully understand why we are the way we are. Sure, we concoct explanations and reasons—neatly logical delusions—but at the core, we simply exist as we are, inexplicable even to ourselves. And the mind? It can’t accept this mystery. Here’s another annoying feature of the mind Have you ever noticed how, when you ignore a certain thought or narrative, your mind scrambles to find something else to occupy you with? It will use to reel you back in. When you feel restless, it’s your mind bouncing around, desperate to divert your focus away from the present back to itself. The mind thrives on your attention; it feeds off it. And we feed it generously, distracted from the now, lost in thoughts of past hurts and stories. anything The more intense the emotion tied to a particular memory, the more your mind returns to that story. It never goes back to the happiest moment of your life over and over again as much as it revisits the moments of hurt, unfairness, and sadness.   These negative experiences have a stronger gravitational pull for evolutionary reasons, but it doesn’t mean we should leave it at that. Happiness only exists in the present moment. I know it’s such a cliche, but seriously - where is your past, right now? Where is it? It’s in your fucking mind, obstructing you from being here, right now, as you are. The only way these stories stay alive is if you keep feeding them your attention. Oh, yes, you do. When we say we don’t have control over our attention, what we really mean is that we haven’t trained a skill of controlled attention. But I don’t control my attention… The good news is that attention is a trainable skill,  but the bad thing is that it takes effort. It requires overcoming the automaticity of your mind. It’s tough! I’m going through this training right now with the coaching course I’m doing, and it’s painfully obvious how automated every aspect of my being is. One of the practices we’re supposed to do daily is a 15-minute contemplation exercise. I won’t go into the details, but my mind would literally do nothing else but concentrate. It brings stuff I haven’t thought of in , to escape focusing for 15 minutes. considerable years anything In fact, I sat down to do the exercise but ended up editing this article! This is the state of affairs of my mind, and most minds, I’d argue, is that 15 minutes out of 16 hours that we are awake is too much for the mind to handle. We have close to 0 control of that freak inside our heads. If this isn’t scary, I don’t know what is 🤷‍♀️ Poem The happiness of being I lay on the grass on a sunny Sunday afternoon Looking at the sky, expecting something new When a thought suddenly arises Maybe I’m not supposed to believe in disguises? The thoughts disguised as truth the Opinions of others disguised as the glue that keeps society together and maybe even predicts the good weather. But seriously - what is a thought? A moment captured then repeated in blocks Built into houses of stories we’re told Forgetting the essence cannot fit in a mold. Society is a construct to keep the flock together But finding happiness requires disconnecting from the fear of separating from others, Even in a stormy weather. The storm inside borne out of separation Holds the key to the kingdom of creation Endless moments of being here and now Forever giving, gifting, leading, leaving The True Self you were never without. . . Thank you for your time & attention. Train your mind because life is as good as the quality of your mind. Also published . here

The Crazy Mind, Artificial Love and Happiness of Being

This is paper is available on arxiv under CC 4.0 DEED license. Authors: (1) Leah Chrestien, Czech Technical University in Prague; (2) Tomá˘s Pevný, Czech Technical University in Prague, and Gen Digital, Inc.; (3) Stefan Edelkamp, Czech Technical University in Prague; (4) Antonín Komenda, Czech Technical University in Prague. Table of Links Abstract & Introduction Preliminaries Conditions on strictly optimally efficient heuristic Loss functions Related Work Experiments Conclusion and References Appendix Abstract In imitation learning for planning, parameters of heuristic functions are optimized against a set of solved problem instances. This work revisits the necessary and sufficient conditions of strictly optimally efficient heuristics for forward search algorithms, mainly A* and greedy best-first search, which expand only states on the returned optimal path. It then proposes a family of loss functions based on ranking tailored for a given variant of the forward search algorithm. Furthermore, from a learning theory point of view, it discusses why optimizing cost-to-goal ∗ is unnecessarily difficult. The experimental comparison on a diverse set of problems unequivocally supports the derived theory. h 1 Introduction Automated planning finds a sequence of actions that will reach a goal in a model of the environment provided by the user. It is considered to be one of the core problems in Artificial Intelligence and it is behind some of its successful applications [27, 39, 46]. Early analysis of planning tasks [31] indicated that optimizing the heuristic function steering the search for a given problem domain towards the goal can dramatically improve the performance of the search. Automated optimization of the heuristic function therefore becomes a central request in improving the performance of planners [6, 13]. Learning in planning means optimizing heuristic functions from plans of already solved problems and their instances. This definition includes the selection of proper heuristics in a set of pattern databases [12, 17, 22, 33], a selection of a planner from a portfolio [25], learning planning operators from instances [32, 56], learning for macro-operators and entanglements [9, 29], and learning heuristic functions by general function approximators (e.g. neural networks) [5, 16, 19, 45]. The majority of research [16, 19, 45, 53] optimizes the heuristic function to estimate the cost-togoal, as it is well known that it is an optimal heuristic for many best-first heuristic search algorithms including A* or IDA* [21, 28]. These works optimize the heuristic function to solve regression problems. But even a true cost-to-goal h∗ does not guarantee that the forward search will find the optimal solution while expanding the minimal number of states. This paper defines a stricter version of optimally efficiency [10, 24] as follows. Forward search (or its heuristic) is called strictly optimally efficient iff it expands only states on one optimal solution path returned by the search, i.e., if this optimal solution path has l + 1 states, the search will expand only l states. This work focuses exclusively on a forward search with merit functions. It presents theoretical arguments as to why the order (rank) of states provided by the heuristic function is more important than the precise estimate of the cost-to-goal and it formalizes the necessary and sufficient condition for strictly optimally efficient search. It also argues why learning to rank is a simpler problem than regression estimating the cost-to-goal from a statistical learning theory point of view. The main motivation of the paper is similar to that in [38], but the derived theory and implementation are much simpler. The sufficiency of optimizing the rank has been already recognized in [59] for a beam-search and in [18] for greedy best-first search (GBFS), though, unlike this work, it cannot guarantee strict optimal efficiency. We emphasize that this work neither deals with nor questions the existence of a strictly optimally efficient heuristic function with the forward search. It is assumed that the space of possible heuristic functions is sufficiently large, such that there exist one that is sufficiently good. The theoretical results are supported by experimental comparisons to the prior art on eight domains. The contributions of this paper are as follows. 1. We state necessary and sufficient conditions for strictly optimally efficient heuristic functions for forward search algorithms in deterministic planning. 2. We show that when optimizing a heuristic function, it is better to solve the ranking problem. 3. We argue, why learning to rank is easier than regression used in learning the cost-to-goal. 4. We instantiate it for A* and GBFS searches leading to two slightly different loss functions. 5. We experimentally demonstrate on eight problems (three grid and five PDDL) that optimizing rank is always better.

Optimize Planning Heuristics to Rank, not to Estimate Cost-to-Goal: Abstract & Intro

This paper is available on arxiv under CC 4.0 license. Authors: (1) Ali Ghanbari, Dept. of Computer Science, Iowa State University; (2) Deepak-George Thomas, Dept. of Computer Science, Iowa State University; (3) Muhammad Arbab Arshad, Dept. of Computer Science, Iowa State University; (4) Hridesh Rajan, Dept. of Computer Science, Iowa State University. Table of Links Abstract & Introduction Background Motivating Example Proposed Approach Supported DNN Bugs Evaluation Threats to Validity Related Work Conclusion, Acknowledgments & References IX. CONCLUSION This paper revisits mutation-based fault localization in the context of DNN and presents a novel DNN fault localization technique, named deepmufl. The technique is based on the idea of mutating a pre-trained DNN model and calculating suspiciousness values according to Metallaxis and MUSE approaches, Ochiai and SBI formulas, and two types of impacts of mutations on the results of test data points. deepmufl is compared to state-of-the-art static and dynamic fault localization systems [11], [8], [34], [12] on a benchmark of 109 model bugs. In this benchmark, while deepmufl is slower than the other tools, it proved to be almost two times more effective than them in terms of the total number of bugs detected and it detects 21 bugs that none of the studied tools were able to detect. We further studied the impact of mutation selection on fault localization time. We observed that we can halve the time taken to perform fault localization by deepmufl, while losing only 7.55% of the previously detected bugs. ACKNOWLEDGMENTS The authors thank Anonymous ASE 2023 Reviewers for their valuable feedback. We also thank Mohammad Wardat for his instructions on querying StackOverflow. This material is based upon work supported by the National Science Foundation (NSF) under the grant #2127309 to the Computing Research Association for the CIFellows Project. This work is also partially supported by the NSF grants #2223812, #2120448, and #1934884. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF. REFERENCES [1] IEEE Standard Classification for Software Anomalies, 2010. [2] A. McPeak, “What’s the true cost of a software bug?” https://smartbear. com/blog/software-bug-cost/, 2017, accessed 08/10/23. [3] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015. [4] M. J. Islam, G. Nguyen, R. Pan, and H. Rajan, “A comprehensive study on deep learning bug characteristics,” in ESEC/FSE, 2019, pp. 510–520. [5] M. J. Islam, R. Pan, G. Nguyen, and H. Rajan, “Repairing deep neural networks: Fix patterns and challenges,” in ICSE, 2020, pp. 1135–1146. [6] Y. Zhang, Y. Chen, S.-C. Cheung, Y. Xiong, and L. Zhang, “An empirical study on tensorflow program bugs,” in ISSTA, 2018, pp. 129–140. [7] N. Humbatova, G. Jahangirova, G. Bavota, V. Riccio, A. Stocco, and P. Tonella, “Taxonomy of real faults in deep learning systems,” in ICSE, 2020, pp. 1110–1121. [8] M. Wardat, B. D. Cruz, W. Le, and H. Rajan, “Deepdiagnosis: automatically diagnosing faults and recommending actionable fixes in deep learning programs,” in ICSE. IEEE, 2022, pp. 561–572. [9] K. Pei, Y. Cao, J. Yang, and S. Jana, “Deepxplore: Automated whitebox testing of deep learning systems,” in SOSP, 2017, pp. 1–18. [10] J. Kim, R. Feldt, and S. Yoo, “Guiding deep learning system testing using surprise adequacy,” in ICSE, 2019, pp. 1039–1049. [11] M. Wardat, W. Le, and H. Rajan, “Deeplocalize: fault localization for deep neural networks,” in ICSE, 2021, pp. 251–262. [12] A. Nikanjam, H. B. Braiek, M. M. Morovati, and F. Khomh, “Automatic fault detection for deep learning programs using graph transformations,” TOSEM, vol. 31, no. 1, pp. 1–27, 2021. [13] M. Usman, D. Gopinath, Y. Sun, Y. Noller, and C. S. Pas˘ areanu, “Nn ˘ repair: Constraint-based repair of neural network classifiers,” in CAV, 2021, pp. 3–25. [14] X. Zhang, J. Zhai, S. Ma, and C. Shen, “Autotrainer: An automatic dnn training problem detection and repair system,” in ICSE, 2021, pp. 359–371. [15] W. E. Wong, R. Gao, Y. Li, R. Abreu, and F. Wotawa, “A survey on software fault localization,” TSE, vol. 42, no. 8, pp. 707–740, 2016. [16] M. Papadakis and Y. Le Traon, “Using mutants to locate” unknown” faults,” in ICST, 2012, pp. 691–700. [17] S. Moon, Y. Kim, M. Kim, and S. Yoo, “Ask the mutants: Mutating faulty programs for fault localization,” in ICST, 2014, pp. 153–162. [18] R. A. DeMillo, R. J. Lipton, and F. G. Sayward, “Hints on test data selection: Help for the practicing programmer,” IEEE Computer, vol. 11, pp. 34–41, 1978. [19] R. Abreu, P. Zoeteweij, and A. J. Van Gemund, “On the accuracy of spectrum-based fault localization,” in TAICPART-MUTATION, 2007, pp. 89–98. [20] V. Dallmeier, C. Lindig, and A. Zeller, “Lightweight Bug Localization with AMPLE,” in ISAADD, 2005, pp. 99–104. [21] J. A. Jones, M. J. Harrold, and J. Stasko, “Visualization of test information to assist fault localization,” in ICSE, 2002, pp. 467–477. [22] L. Naish, H. J. Lee, and K. Ramamohanarao, “A model for spectra-based software diagnosis,” TOSEM, vol. 20, no. 3, pp. 1–32, 2011. [23] S. Yoo, “Evolving human competitive spectra-based fault localisation techniques,” in SBSE, 2012, pp. 244–258. [24] X. Xie, F.-C. Kuo, T. Y. Chen, S. Yoo, and M. Harman, “Provably optimal and human-competitive results in sbse for spectrum based fault localisation,” in SBSE, 2013, pp. 224–238. [25] S. Ma, Y. Liu, W.-C. Lee, X. Zhang, and A. Grama, “Mode: automated neural network model debugging via state differential analysis and input selection,” in ESEC/FSE, 2018, pp. 175–186. [26] H. F. Eniser, S. Gerasimou, and A. Sen, “Deepfault: Fault localization for deep neural networks,” in FASE, 2019, pp. 171–191. [27] N. Humbatova, G. Jahangirova, and P. Tonella, “Deepcrime: mutation testing of deep learning systems based on real faults,” in ISSTA, 2021, pp. 67–78. [28] Q. Hu, L. Ma, X. Xie, B. Yu, Y. Liu, and J. Zhao, “Deepmutation++: A mutation testing framework for deep learning systems,” in ASE, 2019, pp. 1158–1161. [29] L. Ma, F. Zhang, J. Sun, M. Xue, B. Li, F. Juefei-Xu, C. Xie, L. Li, Y. Liu, J. Zhao et al., “Deepmutation: Mutation testing of deep learning systems,” in ISSRE, 2018, pp. 100–111. [30] M. Papadakis and Y. Le Traon, “Metallaxis-fl: mutation-based fault localization,” STVR, vol. 25, no. 5-7, pp. 605–628, 2015. [31] F. Chollet et al., “Keras,” https://keras.io, 2015. [32] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and X. Zheng, “Tensorflow: A system for large-scale machine learning,” in OSDI, 2016, p. 265–283. [33] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, highperformance deep learning library,” in NIPS, 2019, pp. 8024–8035. [34] E. Schoop, F. Huang, and B. Hartmann, “Umlaut: Debugging deep learning programs using program structure and model behavior,” in CHI, 2021, pp. 1–16. [35] W. E. Wong and A. P. Mathur, “Reducing the cost of mutation testing: An empirical study,” JSS, pp. 185–196, 1995. [36] A. Ghanbari, D.-G. Thomas, M. A. Arshad, and H. Rajan, “Mutationbased fault localization of deep neural networks,” https://github.com/ ali-ghanbari/deepmufl-ase-2023, 2023. 37] J. H. Andrews, L. C. Briand, and Y. Labiche, “Is mutation an appropriate tool for testing experiments?” in ICSE, 2005, pp. 402–411. [38] M. Papadakis, M. Kintis, J. Zhang, Y. Jia, Y. Le Traon, and M. Harman, “Mutation testing advances: an analysis and survey,” in Advances in Computers, 2019, vol. 112, pp. 275–378. [39] V. Debroy and W. E. Wong, “Using mutation to automatically suggest fixes for faulty programs,” in ICST, 2010, pp. 65–74. [40] A. Ghanbari, S. Benton, and L. Zhang, “Practical program repair via bytecode mutation,” in ISSTA, 2019, pp. 19–30. [41] G. Fraser and A. Arcuri, “Achieving scalable mutation-based generation of whole test suites,” ESE, pp. 783–812, 2015. [42] F. C. M. Souza, M. Papadakis, Y. Le Traon, and M. E. Delamaro, “Strong mutation-based test data generation using hill climbing,” in IWSBST, 2016, pp. 45–54. [43] D. Shin, S. Yoo, M. Papadakis, and D.-H. Bae, “Empirical evaluation of mutation-based test case prioritization techniques,” STVR, p. e1695, 2019. [44] J. P. Galeotti, C. A. Furia, E. May, G. Fraser, and A. Zeller, “Inferring loop invariants by mutation, dynamic analysis, and static checking,” TSE, pp. 1019–1037, 2015. [45] A. Groce, I. Ahmed, C. Jensen, and P. E. McKenney, “How verified is my code? falsification-driven verification (t),” in ASE, 2015, pp. 737– 748. [46] B. Liblit, M. Naik, A. X. Zheng, A. Aiken, and M. I. Jordan, “Scalable statistical bug isolation,” ACM SIGPLAN Notices, vol. 40, no. 6, pp. 15–26, 2005. [47] R. Abreu, P. Zoeteweij, and A. J. Van Gemund, “An evaluation of similarity coefficients for software fault localization,” in PRDC, 2006, pp. 39–46. [48] Wikipedia contributors, “Hierarchical data format — Wikipedia, the free encyclopedia,” 2022, accessed 08/10/23. [49] H. Coles, T. Laurent, C. Henard, M. Papadakis, and A. Ventresque, “Pit: a practical mutation testing tool for java,” in ISSTA, 2016, pp. 449–452. [50] P. Ammann and J. Offutt, Introduction to software testing. Cambridge University Press, 2016. [51] R. Just, F. Schweiggert, and G. M. Kapfhammer, “Major: An efficient and extensible tool for mutation analysis in a java compiler,” in ASE, 2011, pp. 612–615. [52] Y.-S. Ma, J. Offutt, and Y. R. Kwon, “Mujava: an automated class mutation system,” STVR, pp. 97–133, 2005. [53] D. Schuler and A. Zeller, “Javalanche: Efficient mutation testing for java,” in ESEC/FSE, 2009, pp. 297–298. [54] “Junit,” http://junit.org/, 2019, accessed 08/10/23. [55] “Testng documentation,” https://testng.org/doc/documentation-main. html, 2017, accessed 08/10/23. [56] M. M. Morovati, A. Nikanjam, F. Khomh, Z. Ming et al., “Bugs in machine learning-based systems: A faultload benchmark,” arXiv, 2022. [57] X. Li and L. Zhang, “Transforming programs and tests in tandem for fault localization,” Proceedings of the ACM on Programming Languages, vol. 1, no. OOPSLA, pp. 1–30, 2017. [58] C. Parnin and A. Orso, “Are automated debugging techniques actually helping programmers?” in ISSTA, 2011, pp. 199–209. [59] P. S. Kochhar, X. Xia, D. Lo, and S. Li, “Practitioners’ expectations on automated fault localization,” in ISSTA, 2016, pp. 165–176. [60] D. Shin and D.-H. Bae, “A theoretical framework for understanding mutation-based testing methods,” in ICST, 2016, pp. 299–308. [61] A. V. Pizzoleto, F. C. Ferrari, J. Offutt, L. Fernandes, and M. Ribeiro, “A systematic literature review of techniques and metrics to reduce the cost of mutation testing,” Journal of Systems and Software, vol. 157, p. 110388, 2019. [62] J. Zhang, “Scalability studies on selective mutation testing,” in ICSE, vol. 2, 2015, pp. 851–854. [63] A. J. Offutt, J. Pan, K. Tewary, and T. Zhang, “An experimental evaluation of data flow and mutation testing,” Software: Practice and Experience, vol. 26, no. 2, pp. 165–176, 1996. [64] scikit-learn Contributors, “scikit-learn: Machine learning in python,” 2020, accessed 08/10/23. [Online]. Available: https://scikit-learn.org/stable/ [65] R. Heckel, “Graph transformation in a nutshell,” ENTCS, vol. 148, no. 1, pp. 187–198, 2006. [66] J. Cao, M. Li, X. Chen, M. Wen, Y. Tian, B. Wu, and S.-C. Cheung, “Deepfd: Automated fault diagnosis and localization for deep learning programs,” in ICSE, 2022, p. 573–585. [67] Y. Ishimoto, M. Kondo, N. Ubayashi, and Y. Kamei, “Pafl: Probabilistic automaton-based fault localization for recurrent neural networks,” IST, vol. 155, p. 107117, 2023. [68] Y. Sun, H. Chockler, X. Huang, and D. Kroening, “Explaining image classifiers using statistical fault localization,” in ECCV, 2020, pp. 391–406.

Mutation-based Fault Localization of Deep Neural Networks: Conclusion, Acknowledgments & References

This paper is available on arxiv under CC 4.0 license. Authors: (1) Ali Ghanbari, Dept. of Computer Science, Iowa State University; (2) Deepak-George Thomas, Dept. of Computer Science, Iowa State University; (3) Muhammad Arbab Arshad, Dept. of Computer Science, Iowa State University; (4) Hridesh Rajan, Dept. of Computer Science, Iowa State University. Table of Links Abstract & Introduction Background Motivating Example Proposed Approach Supported DNN Bugs Evaluation Threats to Validity Related Work Conclusion, Acknowledgments & References Abstract Deep neural networks (DNNs) are susceptible to bugs, just like other types of software systems. A significant uptick in using DNN, and its applications in wide-ranging areas, including safety-critical systems, warrant extensive research on software engineering tools for improving the reliability of DNNbased systems. One such tool that has gained significant attention in the recent years is DNN fault localization. This paper revisits mutation-based fault localization in the context of DNN models and proposes a novel technique, named deepmufl, applicable to a wide range of DNN models. We have implemented deepmufl and have evaluated its effectiveness using 109 bugs obtained from StackOverflow. Our results show that deepmufl detects 53/109 of the bugs by ranking the buggy layer in top-1 position, outperforming state-of-the-art static and dynamic DNN fault localization systems that are also designed to target the class of bugs supported by deepmufl. Moreover, we observed that we can halve the fault localization time for a pre-trained model using mutation selection, yet losing only 7.55% of the bugs localized in top-1 position. —Deep Neural Network, Mutation, Fault Localization Index Terms I. INTRODUCTION Software bugs [1] are a common and costly problem in modern software systems, costing the global economy billions of dollars annually [2]. Recently, data-driven solutions have gained significant attention for their ability to efficiently and cost-effectively solve complex problems. With the advent of powerful computing hardware and an abundance of data, the use of deep learning [3], which is based on deep neural networks (DNNs), has become practical. Despite their increasing popularity and success stories, DNN models, like any other software, may contain bugs [4], [5], [6], [7], which can undermine their safety and reliability in various applications. Detecting DNN bugs is not easier than detecting bugs in traditional programs, i.e., programs without any data-driven component in them, as DNNs depend on the properties of the training data and numerous hyperparameters [8]. Mitigating DNN bugs has been the subject of fervent research in recent years, and various techniques have been proposed for testing [9], [10], fault localization [11], [12], and repair [13], [14] of DNN models. Fault localization in the context of traditional programs has been extensively studied [15], with one well-known approach being mutation-based fault localization (MBFL) [16], [17]. This approach is based on mutation analysis [18], which is mainly used to assess the quality of a test suite by measuring the ratio of artificially introduced bugs that it can detect. MBFL improves upon the more traditional, lightweight spectrum-based fault localization [19], [20], [21], [22], [23], [24] by uniquely capturing the relationship between individual statements in the program and the observed failures. While both spectrum-based fault localization [25], [26] and mutation analysis [27], [28], [29] have been studied in the context of DNNs, to the best of our knowledge, MBFL for DNNs has not been explored by the research community, yet the existing MBFL approaches are not directly applicable to DNN models. This paper revisits the idea of MBFL in the context of DNNs. Specifically, we design, implement, and evaluate a technique, named deepmufl, to conduct MBFL in pre-trained DNN models. The basic idea behind deepmufl is derived from its traditional MBFL counterparts, namely, Metallaxis [30] and MUSE [17], that are based on measuring the impact of mutations on passing and failing test cases (see §II for more details). In summary, given a pre-trained model and a set of data points, deepmufl separates the data points into two sets of “passing” and “failing” data points (test cases), depending on whether the output of the model matches the ground-truth. deepmufl then localizes the bug in two phases, namely mutation generation phase and mutation testing/execution phase. In mutation generation phase, it uses 79 mutators, a.k.a. mutation operators, to systematically mutate the model, e.g., by replacing activation function of a layer, so as to generate a pool of mutants, i.e., model variants with seeded bugs. In mutation testing phase, deepmufl feeds each of the mutants with passing and failing data points and compares the output to the output of the original model to record the number of passing and failing test cases that are impacted by the injected bugs. In this paper, we study two types of impacts: type 1 impact, a` la MUSE, which tracks only fail to pass and pass to fail, and type 2 impact, like Metallaxis, which tracks changes in the actual output values. deepmufl uses these numbers to calculate suspiciousness values for each layer according to MUSE, as well as two variants of Metallaxis formulas. The layers are then sorted in descending order of their suspiciousness values for the developer to inspect. We have implemented deepmufl on top of Keras [31], and it supports three types of DNN models for regression, as well as classification tasks that must be written using Sequential API of Keras: fully-connected DNN, convolutional neural network (CNN), and recurrent neural network (RNN). Extending deepmufl to other libraries, e.g., TensorFlow [32] and PyTorch [33], as well as potentially to other model architectures, e.g., functional model architecture in Keras, is a matter of investing engineering effort on the development of new mutators tailored to such libraries and models. Since the current implementation of deepmufl operates on pre-trained models, its scope is limited to model bugs [7], i.e., bugs related to activation function, layer properties, model properties, and bugs due to missing/redundant/wrong layers (see §VI). We have evaluated deepmufl using a diverse set of 109 Keras bugs obtained from StackOverflow. These bugs are representatives of the above-mentioned model bugs, in that our dataset contains examples of each bug sub-category at different layers of the models suited for different tasks. For example, concerning the sub-category wrong activation function model bug, we have bugs in regression and classification fullyconnected DNN, CNN, and RNN models that have wrong activation function of different types (e.g., ReLU, softmax, etc.) at different layers. For 53 of the bugs, deepmufl, using its MUSE configuration, pinpoints the buggy layer by ranking it in top-1 position. We have compared deepmufl’s effectiveness to that of state-of-the-art static and dynamic DNN fault localization systems Neuralint [12], DeepLocalize [11], DeepDiagnosis [8], and UMLAUT [34] that are also designed to detect model bugs. Our results show that, in our bug dataset, deepmufl, in its MUSE configuration, is 77% more effective than DeepDiagnosis, which detects 30 of the bugs. Despite this advantage of deepmufl in terms of effectiveness, since it operates on a pre-trained model, it is slower than state-of-the-art DNN fault localization tools from an end-user’s perspective. However, this is mitigated, to some extent, by the fact that similar to traditional programs, one can perform mutation selection [35] to curtail the mutation testing time: we observed that by randomly selecting 50% of the mutants for testing, we can still find 49 of the bugs in top-1 position, yet we halve the fault localization time after training the model. In summary, this paper makes the following contributions. We develop MBFL for DNN and implement it in a novel tool, named deepmufl, that can be uniformly applied to a wide range of DNN model types. • Technique: We compare deepmufl to state-of-the-art static and dynamic fault localization approaches and observed: • Study: – In four configurations, deepmufl outperforms other approaches in terms of the number of bugs that appear in top-1 position and it detects 21 bugs that none of the studied techniques were able to detect. – We can halve the fault localization time for a pretrained model by random mutation selection without significant loss of effectiveness. We have created the largest curated dataset of model bugs, comprising 109 Keras models ranging from regression to classification and fully-connected DNN to CNN and RNN. • Bug Dataset: In the next section, we review concepts of DNNs, mutation analysis, and MBFL. In §III, we present a motivating example and discusses how deepmufl works under the hood. In §IV, we present technical details of the proposed approach, before discussing the scope of deepmufl in §V. In §VI, we present the results of our experiments with deepmufl and state-of-the-art DNN fault localization tools from different aspects. We discuss threats to validity in §VII and conclude the paper in §IX. Paper organization. The source code of deepmufl and the data associated with our experiments are publicly available [36]. Data availability.

Mutation-based Fault Localization of Deep Neural Networks: Abstract & Introduction

is the secret sauce that not only amplifies skills but also the of your skills. Writing your external knowledge It clarifies your thinking, lights the path for your team, and bridges the gap between engineering and business. Writing can do more than create code; it can persuade, inform, encourage, and inspire. It's worth practicing in its own right. Clarify Your Thoughts Ever been stuck in a mental maze, trying to wrangle a problem like an uncooperative octopus? Yeah, we've all been there. Coding can feel like an endless loop of confusion at times. However, writing can be your trusty problem-solving sidekick. Think of writing as a kind of dialogue you're having with yourself, like rubber ducking, but on paper (or screen). It's a way for your brain to untangle the neurons and make sense of the chaos. Writing forces you to slow down, take a breath, and really think through your ideas. It's like parking your racing thoughts in neat little garages. It helps you spot inconsistencies, gaps, and dead ends in your thinking. You ever re-read something you've written and thought, "What was I even trying to say here?" Yup. That's the point. Writing down your thoughts crystallizes them, making them more tangible and manageable. It's like turning fog into ice cubes. You can handle them, rearrange them, even toss them out if they're not doing you any good. A Lifeline for Teammates In complex code, comments serve as lighthouses, guiding fellow devs through the intricacies of the implemented logic. These written notes explain the ' ' behind certain code choices, offer context for future developers, and sometimes, serve as warnings against potential pitfalls. Well-written ( ) documentation, code, and comments reduce the learning curve for new team members and ensure that the code's original intent is preserved even as the codebase evolves. why and maintained Sigh - remember the telephone game? Sitting in a circle, each person whispers the message to the next, and by the time it reaches the last person, the message no longer even remotely resembles the original. The same can happen with code. Sure, good code is self-documenting. But, when there's an opportunity to add clarity, we should take it. Writing docs and commenting code is like planting trees. Sure, you might not enjoy the shade, but someone else surely will. The trouble with this analogy is in the forest of code, the trees grow quickly... and the next developer who revisits the code revisiting your own code in six months, wondering what on earth you were thinking. For yourself your team, always leave a note. could be you and Bridge the Engineering/Business Gap Writing has the power to break down the walls between the engineering world, often viewed as a cryptic land of code, and the business domain, which thrives on clear, concise communication. For developers, the ability to articulate complex ideas through writing can be a game-changer. You might be a pro at creating elegant solutions, but if you can't convey that elegance in a way that everyone, from the project manager to the sales team, can understand, your skills might not get the recognition they deserve. - George Bernard Shaw "The single biggest problem in communication is the illusion that it has taken place." Shaw got it right. Communication is key. But how can you, as a developer, use writing to bridge this gap? We need to be able to articulate technical constraints to stakeholders, propose solutions to business teams, or explain the implications of certain technological decisions. Imagine how useful it would be for a developer to craft concise, jargon-free, and persuasive documents. They'd be able to seamlessly connect the worlds of engineering and business, ensuring that both sides are aligned in their goals! Persuade, Inform, Encourage, Inspire Outside the confines of code, developers have stories to tell, experiences to share, and knowledge to convey. Whether it's writing a tech blog, penning an op-ed about ethical tech, or drafting a proposal for a new project, the act of writing amplifies a developer's voice. Well-written content can persuade stakeholders, inform the community, encourage peers, and inspire the next generation of coders. In essence, through writing, developers can transcend their immediate roles and make meaningful contributions to the broader tech community. What are you waiting for? Also published here.

How Writing Enhances Software Developers' Skills and Knowledge

is the secret sauce that not only amplifies skills but also the of your skills. Writing your external knowledge It clarifies your thinking, lights the path for your team, and bridges the gap between engineering and business. Writing can do more than create code; it can persuade, inform, encourage, and inspire. It's worth practicing in its own right. Clarify Your Thoughts Ever been stuck in a mental maze, trying to wrangle a problem like an uncooperative octopus? Yeah, we've all been there. Coding can feel like an endless loop of confusion at times. However, writing can be your trusty problem-solving sidekick. Think of writing as a kind of dialogue you're having with yourself, like rubber ducking, but on paper (or screen). It's a way for your brain to untangle the neurons and make sense of the chaos. Writing forces you to slow down, take a breath, and really think through your ideas. It's like parking your racing thoughts in neat little garages. It helps you spot inconsistencies, gaps, and dead ends in your thinking. Have you ever re-read something you've written and thought, "What was I even trying to say here?" Yup. That's the point. Writing down your thoughts crystallizes them, making them more tangible and manageable. It's like turning fog into ice cubes. You can handle them, rearrange them, even toss them out if they're not doing you any good. A Lifeline for Teammates In complex code, comments serve as lighthouses, guiding fellow devs through the intricacies of the implemented logic. These written notes explain the ' ' behind certain code choices, offer context for future developers, and sometimes, serve as warnings against potential pitfalls. Well-written ( ) documentation, code, and comments reduce the learning curve for new team members and ensure that the code's original intent is preserved even as the codebase evolves. why and maintained Sigh - remember the telephone game? Sitting in a circle, each person whispers the message to the next, and by the time it reaches the last person, the message no longer even remotely resembles the original. The same can happen with code. Sure, good code is self-documenting. But, when there's an opportunity to add clarity, we should take it. Writing docs and commenting code is like planting trees. Sure, you might not enjoy the shade, but someone else surely will. The trouble with this analogy is in the forest of code, the trees grow quickly... and the next developer who revisits the code revisiting your own code in six months, wondering what on earth you were thinking. For yourself your team, always leave a note. could be you and Bridge the Engineering/Business Gap Writing has the power to break down the walls between the engineering world, often viewed as a cryptic land of code, and the business domain, which thrives on clear, concise communication. For developers, the ability to articulate complex ideas through writing can be a game-changer. You might be a pro at creating elegant solutions, but if you can't convey that elegance in a way that everyone, from the project manager to the sales team, can understand, your skills might not get the recognition they deserve. - George Bernard Shaw "The single biggest problem in communication is the illusion that it has taken place." Shaw got it right. Communication is key. But how can you, as a developer, use writing to bridge this gap? We need to be able to articulate technical constraints to stakeholders, propose solutions to business teams, or explain the implications of certain technological decisions. Imagine how useful it would be for a developer to craft concise, jargon-free, and persuasive documents. They'd be able to seamlessly connect the worlds of engineering and business, ensuring that both sides are aligned in their goals! Persuade, Inform, Encourage, Inspire Outside the confines of code, developers have stories to tell, experiences to share, and knowledge to convey. Whether it's writing a tech blog, penning an op-ed about ethical tech, or drafting a proposal for a new project, the act of writing amplifies a developer's voice. Well-written content can persuade stakeholders, inform the community, encourage peers, and inspire the next generation of coders. In essence, through writing, developers can transcend their immediate roles and make meaningful contributions to the broader tech community. What are you waiting for? Also published . here

Coding and Beyond: The Multifaceted Benefits of Writing for Developers

The Grand Inquisitor

Reconceiving Education with the Metaverse

Repensar la Educación con el Metaverso

Repenser l'éducation avec le métaverse

मेटावर्स के साथ शिक्षा की पुनः प्राप्ति

メタバースで教育を再考する

Reconcebendo a Educação com o Metaverso

Nhận thức lại giáo dục với Metaverse

用元宇宙重新构想教育

The total perspective vortex Containerizing the best code in the world means nothing if that container does not have the resources to do its job. The idea for this story started while . It became clear that returning accurate responses to inquiries was only the first part of the solution. writing about effective container probes kubelet The second part is the subject of this article: , but exploring the entire landscape of resource allocation, from a single container in a Kubernetes pod to near-infinite capacity in the cloud. allocating resources to containers Kubernetes resource management: a recap “How to ensure containers always have the resources they need?” Consider this section a recap for those just starting with Kubernetes resource management. The official documentation on and namespace limit ranges is still mandatory reading in the space. If you are already familiar with the subject, skip to the next section. Kubernetes resource management for containers Kubernetes resource management primarily covers , and . A container specification may define resource requests and resource limits for each resource type. CPU, memory local ephemeral storage A is the minimum resource amount required for the container runtime. The node scheduler only schedules a pod on a node if that node can meet that minimum request for all containers in the pod. resource request A is the maximum amount of resources the kubelet should allocate to a container during its lifetime. When a container has the resource limit set but not a resource request, that is equivalent to setting the resource request to the same value as the resource limit. resource limit Kubernetes allows software developers and system administrators to collaborate on setting minimum reservations and maximum limits for resources like CPU, memory, and temporary storage. A container may have any combination of resource requests and limits, with the different combinations of CPU and memory reservations defining the quality of service for the parent pod as follows: Every container in the pod has both resource request and limit set, and for each container, request and limit have the same value: Guaranteed. Resource request or limit not set in at least one of the containers in the pod: BestEffort. All other pods that are not in the two previous buckets: Burstable. It is helpful to look at what these categories mean when the decides to schedule a pod on a worker node: kube-scheduler “Guaranteed” means the scheduler only assigns the pod to worker nodes with enough resources to meet the requests of all containers in the pod. “Burstable” QoS means the scheduler looks for a worker node with enough memory to meet the resource requests of all containers in the pod. or anything above the resource request, for that matter. The scheduler does not care whether the worker node can meet the resource limits of containers in the pod “BestEffort” means the scheduler will first meet the resource requests of every pod with a “Guaranteed” or “Burstable” QoS before scheduling a “BestEffort” pod. The scheduler continuously revisits that assessment, so if a “Guaranteed” or “Burstable” pod needs those same resources. it may still mark a “BestEffort” pod for eviction : Who should define container resources? Decision time We must first understand that while . resource requests protect the container resource limits protect the cluster The two main parties in that decision are and . That “negotiation” amongst them is somewhat challenging because it is spread over time: product developers system administrators spend most of the time establishing resource boundaries in isolation from deployment to production environments. Product developers S often deal with specifications with insufficient documentation on how changes in preset requests and limits affect the workloads. ystem administrators Kubernetes workload Product developers have greater visibility into the needs of the workloads and their behavior under different resource allocations. In general, they should be able to identify the following thresholds ahead of shipment: resource request. This value is the resource utilization required to load the runtime and be able to answer probe requests from the kubelet comfortably. to identify troubled containers. Absolute minimum This article shows several PromQL queries resource limit. This value is the point where the performance of the container starts to hit external limitations, such as disk I/O or bottlenecks in calls to 3rd party services. Maximum usable I also encourage a look at ’s blog post titled “ ”, examining important trade-offs related to the definition of resource requests and limits. Natan Yellin You can’t have both high utilization and high reliability Ideally, product developers should also establish and document how container performance varies within those boundaries. That information enables system architects to plan initial capacity before deployment and system administrators to adjust resource requests in production environments. Quotas, ranges, and the tragedy of the commons have visibility into cluster capacity and how much of that capacity is still available. As someone who occasionally wears an SRE hat, I appreciate workloads in the “Guaranteed” QoS bucket. It takes a special kind of confidence (and lots of testing) for a development organization to set fixed resource boundaries on all containers in a pod and trust that pod to work reliably within those boundaries. System administrators “Burstable” workloads fall somewhat lower in that spectrum of trust. When wearing my SRE hat, my best hope is that all containers in a burstable pod have those “absolute minimum” resource requests mentioned in the previous section. When it comes to resource limits in a burstable workload, my opinion depends on the development philosophy behind those limits: (such as a Java Virtual Machine launched with minimum and maximum memory limits.) This kind of arrangement (resource limit set higher than the resource request) means the development team spent time identifying the usable operational range of the workload and has a good grip on managing those resources. (good) Autoscaling within the workload . This design philosophy reminds me of the “ ,” where everyone takes a bit more of a shared resource to anticipate its eventual exhaustion, accelerating the exhaustion of the shared resource. That philosophy is particularly disastrous when the development team underestimates the resource requests. In that situation, the workload continuously operates in the burstable zone of its operational range, making it susceptible to CPU throttling, memory exhaustion, and termination by the kubelet in the worker node. (bad) Protection against eventual bugs tragedy of the commons And that finally leaves us with “BestEffort” workloads, a perennial source of worries due to their unfettered ability to exhaust cluster resources and their tenuous scheduling status in the face of other pods requesting those same resources. When dealing with these types of workloads, a system administrator must take a moment to consider their options: 1. Use to set default resource requests and limits for containers and pods in a namespace. I think the blanket assignment of identical resource limits for all workloads across the namespace, combined with the lack of input from the product developers, makes it an unrealistic proposition. limit ranges 2. Use to cap the combined resources for all pods in a namespace. This approach is a more promising solution. It reduces the “blast radius” of an eventual container going haywire to that namespace, with the advantage of avoiding the waste of multiplying the resource safety buffer by the number of pods in the namespace. resource quotas 3. Take a deep breath and hope for relief from the technologies covered in the next section. pod autoscaling Pod autoscaling: Beyond static sizing Many readers are probably ready to bring up the usual workarounds for dealing with unruly workloads: Horizontal pod autoscaling (HPA) Vertical pod autoscaling (VPA) These popular autoscaling techniques can dynamically allocate capacity within a cluster based on resource metrics; HPA can alter the number of pod replicas in a deployment, while VPA can change the resource requests and limits for a pod — Note that until this is added to Kubernetes, VPA changes to a pod result in the pod being restarted. Kubernetes Enhancement Proposal While capable of automatically extending capacity, these components are not a free pass to forego the precise definition of the operational ranges for their workloads. For instance, HPA will not take action on a metric where containers in the pod do not have the resource request for that metric. HPA is also ill-suited to scenarios where a deployment relies on a specific number of replicas for the pod in the cluster. Turning to VPA is better for workloads that require a fixed number of replicas or where the workload does not ship with preset limits. There is, however, a relatively long . Additionally, if using limit ranges, the considerations may get quite involved. list of limitations Ultimately, the effectiveness of these tools in autoscaling workloads depends on the profiling effort spent on defining well-researched resource requests and limits during development. Here, I must give VPA extra credit for its ability to “learn” the operational resource ranges for a deployment. Auto Pilot operators: SRE in a box? introduce the notion of custom resources to manage applications and workloads. Conceptually, an operator continuously watches a custom resource and maps its contents to Kubernetes operators workload resources. The calls for five different levels of autonomy, with the highest being level 5, or “Auto Pilot.” An operator at that stage of maturity should, amongst other kinds of autonomous behavior, apply horizontal or vertical scaling to the workloads it manages. Operator Maturity Model There is a natural extra cost in developing an operator, including designing and documenting custom resource definitions. There is also the additional runtime cost of the operator pod. That is why I think the value of operators starts to offset its costs only at level 4, or “Deep Insights,” which entails generating metrics, issuing alerts, workload analysis, and others. Below level 4, operators do little to alleviate the continuous cycle of product developers attempting to predict the future and system administrators trying to bend those predictions into reality. Knative: Elastic capacity with zero waste According to its website, Knative “is an Open-Source Enterprise-level solution to build Serverless and Event-Driven Applications.” From the perspective of this article, we are interested in Knative’s Serving module. That module contains the Knative Pod Autoscaler (KPA,) which mimics HPA’s ability to vary the number of pod replicas to meet demand, with two key distinctions: Scale replicas based on the volume of external requests Scale replicas down to zero when there are no external requests Unlike HPA, which reacts to resource utilization when it crosses preset boundaries, KPA makes its decisions based on concurrency and requests-per-second targets. As a rule of thumb, HPA is better suited for long-running, slow-starting processes where resource demands do not change abruptly. In contrast, KPA favors transient processes that can boot quickly to meet immediate demand. And last but not least, serverless or not, KPA works with containers, so resource limits still matter. Cluster autoscaling: Pushing the limits Once you have appropriately tuned all workload resources and pod autoscalers, it is time to look deeper into the toolbox and reach for the Kubernetes cluster autoscaler. The cluster autoscaler can add (or remove) worker nodes based on resource demands. The autoscaler respects configurable lower and upper boundaries to ensure minimum capacity and some level of cost controls to avoid unruly workloads overstretching the cluster. Keep in mind that response times for a cluster autoscaler are decidedly sluggish compared to pod autoscalers, with a worker node taking several minutes to become available to the Kube-scheduler. Slow stretching. Suppose you need a cluster autoscaler to work with peak demands. In that case, you must resort to deploying a sacrificial buffer of “pause pods” (described in the .) When new workloads need those resources, the Kube-scheduler immediately evicts the pause pods to free up resources. The cluster later scales up the number of nodes to reallocate the evicted pods. cluster autoscaler documentation Note that Kubernetes also play a role in how autoscalers do their work, making tainted worker nodes off-limits for the placement of pods without tolerance for those taints. taints and tolerations Serverless: Limitless capacity…with limits For the ultimate level of autoscaling, it is time to abandon the boundaries of a cluster — even Kubernetes — and look at serverless capabilities at the cloud provider level. has limits Major cloud providers can spin up container images on demand based on multiple events, such as web requests, database operations, and timers. Compared to a single cluster, the resource capacity of a cloud provider is virtually unlimited, and the cost of running a container for a few moments is almost negligible compared to running the same workload inside a cluster. Ironically, despite an ultra-low-cost per invocation, serverless approaches make you switch from worrying about resource limits to : worrying about cost overruns Container limits still matter: Cloud providers require you to indicate the number of CPUs and the amount of memory for each container invocation. Budget limits: Where a bug in a test suite may exhaust cluster capacity, . that same bug in a serverless deployment may erase your entire department’s budget overnight You may find several variants of serverless offerings, such as handing a small function to the cloud engine rather than an entire container image or letting the cloud provider use one of your managed clusters to run the containers (trading scale for security.) For added convenience, many of these offerings can start from a Git repository containing a Dockerfile, which they use to build and cache the container image before scheduling workloads referencing that image. Conclusion We covered the entire spectrum of resource allocation for a workload, from tuning a single container to hyperscaling a workload outside Kubernetes. The article started with a recap of Kubernetes resource management and its core concepts of . We then built upon those static limits into the realm of : HPA, VPA, and KPA. resource requests and limits pod autoscalers From pod autoscalers’ ability to efficiently utilize a cluster’s maximum capacity, we moved on to increase that capacity with . We then completed the with (and costs) in the form of offerings across cloud providers. cluster autoscaling total perspective vortex virtually unlimited resources serverless Except for VPA, these techniques still build upon . For example, HPA does not take action based on resources where the limits are not specified, while cloud providers outright ask for the CPU and memory resources assigned to each container run. reasonable effort to determine a container’s resource consumption during development activities With the landscape of resource scaling out of the way, my next article on this topic will cover concrete techniques to identify resource requests and limits during development and keep a watchful eye on them in production. References — , by Andre Tost Practical Kubernetes Top Ten Challenges — Part 6: Sizing and Footprint Optimization , by Nathan Yellin You can’t have both high utilization and high reliability Vertical Pod Autoscaler deep dive, limitations and real-world examples Vertical Pod Autoscaling: The Definitive Guide Kubernetes Workload Resources (Managed Knative) IBM Code Engine Also published here.

Stories, News & Mentions about VEGVISITS

Vegvisits’s Stories on HackerNoon