Simula is on a mission to build the world's first Linux VR Computer, a portable VR headset dedicated to office computing and knowledge work

SimulaVR

simulavr

AVR Microcontroller Testing Procedures

I heard about AI-generated art for the first time in a Wall Street Journal podcast in 2021. In the podcast, the host was playing AI-generated pieces of classical music. These pieces emulated the style of classical composers, and the precision with which they did that was rather impressive. A listener uninitiated to classical music would probably have a hard time differentiating them from the actual works of those composers. This was also the first time I took seriously. Listening to those pieces made me wonder – what if AI gets so good at making music that those renowned composers eventually get buried under piles of AI-generated classical music? I have always believed in judging a work of art for what it is instead of finding its aesthetic validity in the legacy of a musician or how critics received a piece of music. If AI-generated music is going to get better, I will most likely say, Sorry, Chopin. generative AI (GenAI) Last week, a colleague showed me OpenAI Sora, a text-to-video model that generates remarkably detailed, realistic videos from text prompts. The first video they showcased on the website featured a woman (with five fingers, not four) walking through the streets of Tokyo. The texture of her skin, the blemishes and spots on her face, the wrinkles, and her hair – everything about this woman in focus is more realistic than (or at least, as much as) subjects in those videos that were once ‘really shot’ by professional videographers. Are we entering the era of the hyperreal? In his book Simulacra and Simulation, Jean Baudrillard first defined the concept of hyperreal as the blurring of the lines that separate reality from representation. This blurring is caused by the creation of representations (signs) that do not refer to anything that exists in reality. Hyperreality results in our inability to distinguish what is simulated from what is real. In his text, Baudrillard was alluding to the process by which value gets created in consumerist paradigms. But with GenAI, his idea is being stretched to its literal limits. On the OpenAI Sora page are over 30 video clips, each containing chunks of time that never happened, all in great detail, ready for us to witness with our eyes. What do we see when we see ‘historical footage of California during the gold rush’ generated by Sora? In truth, there is nothing historical about this footage, and neither is it footage in the true sense. The historical effect is rendered by employing visual aesthetics that elicit a sense of historicity in films and documentaries – warm tones, vintage overlay, grainy effects, and a lower frame rate. Yet, the prompt results in a very believable "footage." I am no historian, so I cannot say if the clip gets historical details right in how it renders the look of the buildings and the attire of the people on horses. But the geographical features look rather compelling. But this is not the crux of the issue here. Such visual effects have been achieved by studios using CGI for decades now. But these endeavors were applied to a specific end and within particular contexts – like creating a realistic game, a movie sequence, or making a documentary more tangible, if you will. The content was vetted by experts who could verify its correctness and authenticity. But now that this technology (text-to-video models) will democratize access to such capabilities, the consequences will be very different. Content will be affected by this democratization of GenAI long before it starts weaving symphonies that draw tears from our eyes or write Nobel-winning novels. A new distinction between content and art Content and art have coexisted ever since the advent of media – writing and painting, followed by print and photography, then audio and video. Content was usually short-lived and created for a specific function, such as entertaining, delivering news, capturing memories, recording events, or even selling things. Art served a purpose in the human being’s socio-cultural experience, and artworks survived well beyond their time. However, both content and art had to be created before GenAI entered the picture. higher Until now, art – or at least, what is deemed art in ‘high culture’ has remained largely untouched by GenAI capabilities. While GenAI may be able to emulate Mozart’s musical style, the aesthetic and cultural value of that style was rooted in the time and context in which Mozart was creating music. The same goes for literary works, which, in effect, created a paradigm shift in the ways of thinking about the world, human beings, society, and the small and big questions underpinning our existence during a period in history. The story of the content is somewhat different. Until some point in the 2000s, content creation required specialized apparatus – sophisticated video cameras or DSLRs and teams of people were needed to create audio-visual content, for example. But with social media and the proliferation of smartphones, content creation took a new turn. Everyone was now a creator, snipping pictures (or videos) of their dogs and cats doing funny stuff, creating video memes, or making comic faces with Instagram filters. Compilations of these snippets of human existence made their way to YouTube in droves, and content flooded depthless Instagram, Facebook, and TikTok feeds. Yet, few pieces of content warranted the need to question the reality of the subject of these videos. With the democratization of GenAI, we will find ourselves asking this question more often. In fact, while scrolling through the comments section of the Apple Airpods Pro 2 launch live stream in 2023, I found a user wondering whether the woman on screen, who was presenting the features of the new Airpods, was a real person. However, the industry is attempting to solve this problem with standards like the C2PA, which will embed details like the creator and mode of creation within media files. This will make it possible to differentiate AI-generated content from ‘real’ content. Nonetheless, AI-generated content will coexist with the content that is created by people. This will add another level of distinction between content and art – which is that art be created, but content be generated. must can The ability to generate content will undoubtedly have far-reaching economic consequences and leave a profound impact on many professions. But it will also alter our relationship with content and how it informs our worldview. The desert of the real? What if generated content is more compelling, more convincing, more entertaining, and more appealing to the viewer? Given how most societies today are inclined to promote value-creating systems, generated content will likely overtake content created by people in such a scenario – which means that content platforms will be saturated with AI-generated content that will show places that never existed, people who never walked on the planet, events that never occurred, or cute dog stuff that no real dog ever did. This is quite likely to happen – AI thought leader Nina Schick says 90% of online content could be AI-generated. But the proliferation of AI-generated content could go two ways. According to neuroscientists, knowing an artwork as authentic activates reward pathways in the brain of viewers. Forged or fake art, on the other hand, does not lead to the same effect. If this result is extrapolated to AI-generated content, one would be led to conclude that AI-generated content will not be as fulfilling as that which is created by real people recording real things – given that content platforms enforce rules that enable viewers to identify AI-generated content. Will AI-generated dog videos be as effective in reducing our stress levels as the of real dogs doing real things? If not, then the adoption of text-to-video models will be limited to certain areas, like making clips for purchasable stock video collections. Or generating elements of videos, like backgrounds or animated overlays. footage In this scenario, we will see the return of content that is created rather than generated – and social media platforms would be the first sites where this shift will unfold. But there is another factor that must be taken into account here. Research on Instagram usage patterns has shown that upward comparison (comparing oneself to those who people perceive as superior to them) through content consumption has detrimental effects on users' mental health. If AI-generated content breaks through the barriers of creative possibilities, it will present a version of the world that is to its real counterpart. If this happens, will AI-generated content immerse us in a hyperreal world, framing our existing one into a dull and dry reality? superior Probably not – at least not right away. Current text-to-video models still have limitations. For instance, the wolf pups in Sora-generated clips sometimes emerge out of nothing, and bodies sometimes magically pass through other bodies. In another clip, a plastic chair sometimes behaves like a sheet of paper and changes shape through the clip. OpenAI claims that giving the model foresight into multiple frames enables them to retain the visual identity of a subject even when it goes out of view for a few moments. But this doesn't make the model infallible, as is evident in the videos on OpenAI’s Sora page. Moreover, the generated clips are still only 60 seconds long, and it will probably be a while before we see full-length feature films and documentaries generated by such models. However, these limitations will likely be eliminated in upcoming generations of multimodal models – and soon enough, considering that the technology has been in development for a little over half a decade. When that time comes, the experience of consuming content or simply navigating through cyberspace will take a new turn. For the better or worse? Only time will tell. Looking towards a brighter future Generation and creation remain very distinct ideas, even in the era of GenAI. This distinction is rooted in the etymology of these words – create comes from the Latin , which means to form out of nothing. On the other hand, generate comes from the Latin (meaning to produce), which in turn comes from (meaning a type or a family). In both mathematics and linguistics, generation pertains to a process of applying a set of rules to an input to produce something. This is precisely what Generative Pre-Trained Transformers (GPTs) do. creare generō genus As far-fetched as the meaning of creation may seem in the era of content, GenAI is compelling us to exercise our higher cognitive abilities as it abstracts the heavy lifting of content generation from the creators. Creation, at its heart, remains a process of making something out of nothing. It is this idea of creation that has ushered era-defining epochs in the course of our cultural, social, and technological history. generated With GenAI solutions undertaking the less cognitive aspects of content creation, each of us will be empowered to create more sophisticated things, just better and faster. But where this paradigm of GenAI-augmented creation takes us will be determined collectively by content-powered platforms, regulators, content provenance standards and their endorsement, and our inevitable neurological responses to new forms of content.

Democratization of Text-to-video Models Will Launch Us Into a Hyperreal Cyberspace. Are We Ready?

Authors:
(1) Omid Davoodi, Carleton University, School of Computer Science;
(2) Shayan Mohammadizadehsamakosh, Sharif University of Technology, Department of Computer Engineering;
(3) Majid Komeili, Carleton University, School of Computer Science. Table of Links Abstract and Intro Background Information Methodology Prototype Interpretability Prototype-query Similarity Interpretability of the Decision-Making Process The Effects of Low Prototype Counts Discussions The Effects of Low Prototype Counts During preparation for our experiments, we noticed an unexpected phenomenon for prototype similarity. In some dataset/method combinations, it was possible to achieve high classification accuracies of more than 80% with only one prototype per class. For example, TesNet was able to achieve about 88% accuracy when trained on the ImageNet subset using only 7 prototypes. In this instance, the similarity of a prototype with its activation was very low. This sheds light on some of the limitations of these methods. Part-prototype-based classifiers are, in the end, discriminative models. This is in contrast to how they are perceived from an interpretability standpoint. When humans look at the explanations given for the decision, they see the similarity between the prototypes and the activated regions. However such similarity is only useful to the model itself as long as it can utilize it to discriminate effectively between the classes. As long as a query sample is closer to the prototype of the correct class than all other classes, the model makes the correct decision, regardless of the actual similarity between the query sample and the prototype. When designing a model, setting the number of prototypes to a small value favors interpretability, but, on the other hand, it may adversely affect the similarity of the prototypes and their corresponding activations. Even when the number of prototypes is relatively small, to truly understand the model, it is essential to examine the similarity between prototypes and their corresponding activations (as in experiment 2), as well as the interpretability of the individual prototypes (as in experiment 1). This paper is available on arxiv under CC 4.0 license. Authors: (1) Omid Davoodi, Carleton University, School of Computer Science; (2) Shayan Mohammadizadehsamakosh, Sharif University of Technology, Department of Computer Engineering; (3) Majid Komeili, Carleton University, School of Computer Science. Authors: Authors: (1) Omid Davoodi, Carleton University, School of Computer Science; (2) Shayan Mohammadizadehsamakosh, Sharif University of Technology, Department of Computer Engineering; (3) Majid Komeili, Carleton University, School of Computer Science. Table of Links Abstract and Intro Abstract and Intro Background Information Background Information Methodology Methodology Prototype Interpretability Prototype Interpretability Prototype-query Similarity Prototype-query Similarity Interpretability of the Decision-Making Process Interpretability of the Decision-Making Process The Effects of Low Prototype Counts The Effects of Low Prototype Counts Discussions Discussions The Effects of Low Prototype Counts During preparation for our experiments, we noticed an unexpected phenomenon for prototype similarity. In some dataset/method combinations, it was possible to achieve high classification accuracies of more than 80% with only one prototype per class. For example, TesNet was able to achieve about 88% accuracy when trained on the ImageNet subset using only 7 prototypes. In this instance, the similarity of a prototype with its activation was very low. This sheds light on some of the limitations of these methods. Part-prototype-based classifiers are, in the end, discriminative models. This is in contrast to how they are perceived from an interpretability standpoint. When humans look at the explanations given for the decision, they see the similarity between the prototypes and the activated regions. However such similarity is only useful to the model itself as long as it can utilize it to discriminate effectively between the classes. As long as a query sample is closer to the prototype of the correct class than all other classes, the model makes the correct decision, regardless of the actual similarity between the query sample and the prototype. When designing a model, setting the number of prototypes to a small value favors interpretability, but, on the other hand, it may adversely affect the similarity of the prototypes and their corresponding activations. Even when the number of prototypes is relatively small, to truly understand the model, it is essential to examine the similarity between prototypes and their corresponding activations (as in experiment 2), as well as the interpretability of the individual prototypes (as in experiment 1). This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

The Effects of Low Prototype Counts on ML Model Interpretability and Similarity

Authors:
(1) Omid Davoodi, Carleton University, School of Computer Science;
(2) Shayan Mohammadizadehsamakosh, Sharif University of Technology, Department of Computer Engineering;
(3) Majid Komeili, Carleton University, School of Computer Science. Table of Links Abstract and Intro Background Information Methodology Prototype Interpretability Prototype-query Similarity Interpretability of the Decision-Making Process The Effects of Low Prototype Counts Discussions Prototype-query Similarity Our second set of experiments has the goal of finding out whether the regions activated by a prototype contain the same concept as the prototype itself. Prototype classifiers, in general, use the closest prototype to a query in their operation. Part-prototype classifiers follow the same idea by finding the closest prototype patch to another patch in the input image. For this to be interpretable, the notion of closeness used by the method should match that of a human. Otherwise, the explanation itself will seem nonsensical. Figure 5 shows examples of both good and bad similarity between the query patch and the prototype. As before, the first experiment was purely comparative. Each fine-grained task consisted of two pairs of images. Each pair consisted of a prototype and its activation on a query image. The prototypes picked were the most important prototypes when trying to classify an image. The idea was to avoid prototypes that were not instrumental in the final decision as they were more likely to not be present in the query image itself. The annotators were then told to select the prototype/activation pair that were conceptually more similar to each other. Figure 6 shows examples of this task where annotators are expected to select pair-2 because the corresponding images are more similar than the pair-1 images. This works well for the part-prototype classification methods as they all have an explanation generation procedure that utilizes their prototypes. On the other hand, ACE is not a classifier. It is an unsupervised concept learning method. Nevertheless, we thought that ACE by itself also had a notion of similarity baked into the method. It picks multiple patches from potentially different images to represent a concept. The assumption is that these patches are similar because they share that concept. Here, we decided to simply include two patches from the same concept in the pairs that represented ACE. The annotators were presented with batches of 10 fine-grained tasks. Like before, 2 out of those 10 tasks contained validation data. Also similar to previous experiments, annotators had to obtain qualification by completing 7 tasks perfectly before being allowed to work on the main experiment. The experiment was repeated 3 times and the results were aggregated. The instructions shown to the annotators are shown in Figure 6. As with prototype interpretability experiments, we also devised a more absolute version of the similarity experiment by adding the options "Both" and "None" to the same experimental layout. The instructions for this other experiment were as follows: • Each set contains two pairs of images. In each one, select the pair of images where the marked regions in those images are conceptually closer to each other than the other pair. • For example if in pair 1, the marked regions are conceptually very different (for example, one is of a face and the other is of the paws of a cat), do not select that pair. On the other hand, if in pair 2, the marked regions are visually and conceptually very similar, select pair 2. • If both pairs have similar regions, select Both. If none of the pairs has similar regions, select None. In total, 1156 pairs were compared and annotated by humans for the comparative study of prototype-query similarity. Table 5 shows the percentage each method was chosen against others. As can be seen, TesNet were chosen 71.88% of times when compared to other methods. At the second place was ProtoPNet with 64.47% preference. This shows that apart from TesNet, the other methods show comparatively worse prototype-query similarity compared to ProtoPNet itself. This has an interesting implication for the interpretability of these methods compared to the baseline. Table 6 shows the results of the same experiments but on a method-by-method basis. Each element of the table shows the results of the pairwise comparison between the two methods, with the method shown in the first column being the one selected out of the two. For example, the intersection between the row "ACE" and the column "ProtoPool" shows the percentage of times (24.17%) ACE was chosen against ProtoPool in the experiments. While there is more variation when looked at with this much detail, the general trends hold true. For the absolute experiments, 1984 pairs were compared and annotated by humans in total. Table 7 shows the percentage of times each method’s prototypes and query sample activations were seen to be close conceptually. While the overall rankings seem to have mostly remained the same, the lower overall percentages show that none of these methods is doing well in this regard. The best results, TesNet at 42.98% and ProtoPNet at 40.41% are still quite low. This is in agreement with the results previously reported in[12], [14]. Finally, Table 8 shows the results of the same experiments but on a method-by-method basis. Each row shows the percentage of times a method (shown on the first column) was selected either solely or as part of the "Both" option when paired with another method shown on the top row. Note that the sum of mirrored cells does not necessarily add up to 100% due to the existence of "Both" and "None" options. The consistently low similarity rates across different methods point to a general failure of all models in achieving a good similarity between the activated regions of the query samples and the prototypes. This paper is available on arxiv under CC 4.0 license. Authors: (1) Omid Davoodi, Carleton University, School of Computer Science; (2) Shayan Mohammadizadehsamakosh, Sharif University of Technology, Department of Computer Engineering; (3) Majid Komeili, Carleton University, School of Computer Science. Authors: Authors: (1) Omid Davoodi, Carleton University, School of Computer Science; (2) Shayan Mohammadizadehsamakosh, Sharif University of Technology, Department of Computer Engineering; (3) Majid Komeili, Carleton University, School of Computer Science. Table of Links Abstract and Intro Abstract and Intro Background Information Background Information Methodology Methodology Prototype Interpretability Prototype Interpretability Prototype-query Similarity Prototype-query Similarity Interpretability of the Decision-Making Process Interpretability of the Decision-Making Process The Effects of Low Prototype Counts The Effects of Low Prototype Counts Discussions Discussions Prototype-query Similarity Our second set of experiments has the goal of finding out whether the regions activated by a prototype contain the same concept as the prototype itself. Prototype classifiers, in general, use the closest prototype to a query in their operation. Part-prototype classifiers follow the same idea by finding the closest prototype patch to another patch in the input image. For this to be interpretable, the notion of closeness used by the method should match that of a human. Otherwise, the explanation itself will seem nonsensical. Figure 5 shows examples of both good and bad similarity between the query patch and the prototype. As before, the first experiment was purely comparative. Each fine-grained task consisted of two pairs of images. Each pair consisted of a prototype and its activation on a query image. The prototypes picked were the most important prototypes when trying to classify an image. The idea was to avoid prototypes that were not instrumental in the final decision as they were more likely to not be present in the query image itself. The annotators were then told to select the prototype/activation pair that were conceptually more similar to each other. Figure 6 shows examples of this task where annotators are expected to select pair-2 because the corresponding images are more similar than the pair-1 images. This works well for the part-prototype classification methods as they all have an explanation generation procedure that utilizes their prototypes. On the other hand, ACE is not a classifier. It is an unsupervised concept learning method. Nevertheless, we thought that ACE by itself also had a notion of similarity baked into the method. It picks multiple patches from potentially different images to represent a concept. The assumption is that these patches are similar because they share that concept. Here, we decided to simply include two patches from the same concept in the pairs that represented ACE. The annotators were presented with batches of 10 fine-grained tasks. Like before, 2 out of those 10 tasks contained validation data. Also similar to previous experiments, annotators had to obtain qualification by completing 7 tasks perfectly before being allowed to work on the main experiment. The experiment was repeated 3 times and the results were aggregated. The instructions shown to the annotators are shown in Figure 6. As with prototype interpretability experiments, we also devised a more absolute version of the similarity experiment by adding the options "Both" and "None" to the same experimental layout. The instructions for this other experiment were as follows: • Each set contains two pairs of images. In each one, select the pair of images where the marked regions in those images are conceptually closer to each other than the other pair. • For example if in pair 1, the marked regions are conceptually very different (for example, one is of a face and the other is of the paws of a cat), do not select that pair. On the other hand, if in pair 2, the marked regions are visually and conceptually very similar, select pair 2. • If both pairs have similar regions, select Both. If none of the pairs has similar regions, select None. In total, 1156 pairs were compared and annotated by humans for the comparative study of prototype-query similarity. Table 5 shows the percentage each method was chosen against others. As can be seen, TesNet were chosen 71.88% of times when compared to other methods. At the second place was ProtoPNet with 64.47% preference. This shows that apart from TesNet, the other methods show comparatively worse prototype-query similarity compared to ProtoPNet itself. This has an interesting implication for the interpretability of these methods compared to the baseline. Table 6 shows the results of the same experiments but on a method-by-method basis. Each element of the table shows the results of the pairwise comparison between the two methods, with the method shown in the first column being the one selected out of the two. For example, the intersection between the row "ACE" and the column "ProtoPool" shows the percentage of times (24.17%) ACE was chosen against ProtoPool in the experiments. While there is more variation when looked at with this much detail, the general trends hold true. For the absolute experiments, 1984 pairs were compared and annotated by humans in total. Table 7 shows the percentage of times each method’s prototypes and query sample activations were seen to be close conceptually. While the overall rankings seem to have mostly remained the same, the lower overall percentages show that none of these methods is doing well in this regard. The best results, TesNet at 42.98% and ProtoPNet at 40.41% are still quite low. This is in agreement with the results previously reported in[12], [14]. Finally, Table 8 shows the results of the same experiments but on a method-by-method basis. Each row shows the percentage of times a method (shown on the first column) was selected either solely or as part of the "Both" option when paired with another method shown on the top row. Note that the sum of mirrored cells does not necessarily add up to 100% due to the existence of "Both" and "None" options. The consistently low similarity rates across different methods point to a general failure of all models in achieving a good similarity between the activated regions of the query samples and the prototypes. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Analysis of Prototype-Query Similarity Rankings

Author:
(1) Mohammad Akhlaghi, Centro de Estudios de F´ısica del Cosmos de Aragon (CEFCA), Plaza San Juan 1, 44001, Teruel, Spain {mohammad@akhlaghi.org}. Table of Links Abstract and Intro Simulating the Exposure Map Acknowledgment and References Appendix APPENDIX A. SOFTWARE ACKNOWLEDGEMENT This research was done with the following free software programs and libraries: 1.23, Bzip2 1.0.8, CFITSIO 4.1.0, CMake 3.24.0, cURL 7.84.0, Dash 0.5.11-057cd65, Discoteq flock 0.4.0, Expat 2.4.1, File 5.42, Fontconfig 2.14.0, FreeType 2.11.0, Git 2.37.1, GNU Astronomy Utilities 0.20.72-08b0 (Akhlaghi & Ichikawa 2015; ?), GNU Autoconf 2.71, GNU Automake 1.16.5, GNU AWK 5.1.1, GNU Bash 5.2-rc2, GNU Binutils 2.39, GNU Bison 3.8.2, GNU Compiler Collection (GCC) 12.1.0, GNU Coreutils 9.1, GNU Diffutils 3.8, GNU Findutils 4.9.0, GNU gettext 0.21, GNU gperf 3.1, GNU Grep 3.7, GNU Gzip 1.12, GNU Integer Set Library 0.24, GNU libiconv 1.17, GNU Libtool 2.4.7, GNU libunistring 1.0, GNU M4 1.4.19, GNU Make 4.3, GNU Multiple Precision Arithmetic Library 6.2.1, GNU Multiple Precision Complex library, GNU Multiple Precision Floating-Point Reliably 4.1.0, GNU Nano 6.4, GNU NCURSES 6.3, GNU Readline 8.2- rc2, GNU Scientific Library 2.7, GNU Sed 4.8, GNU Tar 1.34, GNU Texinfo 6.8, GNU Wget 1.21.2, GNU Which 2.21, GPL Ghostscript 9.56.1, Help2man , Less 590, Libffi 3.4.2, Libgit2 1.3.0, libICE 1.0.10, Libidn 1.38, Libjpeg 9e, Libpaper 1.1.28, Libpng 1.6.37, libpthread-stubs (Xorg) 0.4, libSM 1.2.3, Libtiff 4.4.0, libXau (Xorg) 1.0.9, libxcb (Xorg) 1.15, libXdmcp (Xorg) 1.1.3, libXext 1.3.4, Libxml2 2.9.12, libXt 1.2.1, Lzip 1.23, OpenSSL 3.0.5, PatchELF 0.13, Perl 5.36.0, pkgconfig 0.29.2, podlators 4.14, Python 3.10.6, utilLinux 2.38.1, util-macros (Xorg) 1.19.3, WCSLIB 7.11, X11 library 1.8, XCB-proto (Xorg) 1.15, xorgproto 2022.1, xtrans (Xorg) 1.4.0, XZ Utils 5.2.5 and Zlib 1.2.11. The LATEX source of the paper was compiled to make the PDF using the following packages: courier 61719 (revision), epsf 2.7.4, etoolbox 2.5k, helvetic 61719 (revision), lineno 5.3, pgf 3.1.10, pgfplots 1.18.1, revtex4- 1 4.1s, tex 3.141592653, textcase 1.04 and ulem 53365 (revision). We are very grateful to all their creators for freely providing this necessary infrastructure. This research (and many other projects) would not be possible without them. This paper is available on arxiv under CC 4.0 DEED license. Author: (1) Mohammad Akhlaghi, Centro de Estudios de F´ısica del Cosmos de Aragon (CEFCA), Plaza San Juan 1, 44001, Teruel, Spain {mohammad@akhlaghi.org}. Author: Author: (1) Mohammad Akhlaghi, Centro de Estudios de F´ısica del Cosmos de Aragon (CEFCA), Plaza San Juan 1, 44001, Teruel, Spain {mohammad@akhlaghi.org}. Table of Links Abstract and Intro Abstract and Intro Simulating the Exposure Map Simulating the Exposure Map Acknowledgment and References Acknowledgment and References Appendix Appendix APPENDIX A. SOFTWARE ACKNOWLEDGEMENT This research was done with the following free software programs and libraries: 1.23, Bzip2 1.0.8, CFITSIO 4.1.0, CMake 3.24.0, cURL 7.84.0, Dash 0.5.11-057cd65, Discoteq flock 0.4.0, Expat 2.4.1, File 5.42, Fontconfig 2.14.0, FreeType 2.11.0, Git 2.37.1, GNU Astronomy Utilities 0.20.72-08b0 (Akhlaghi & Ichikawa 2015; ?), GNU Autoconf 2.71, GNU Automake 1.16.5, GNU AWK 5.1.1, GNU Bash 5.2-rc2, GNU Binutils 2.39, GNU Bison 3.8.2, GNU Compiler Collection (GCC) 12.1.0, GNU Coreutils 9.1, GNU Diffutils 3.8, GNU Findutils 4.9.0, GNU gettext 0.21, GNU gperf 3.1, GNU Grep 3.7, GNU Gzip 1.12, GNU Integer Set Library 0.24, GNU libiconv 1.17, GNU Libtool 2.4.7, GNU libunistring 1.0, GNU M4 1.4.19, GNU Make 4.3, GNU Multiple Precision Arithmetic Library 6.2.1, GNU Multiple Precision Complex library, GNU Multiple Precision Floating-Point Reliably 4.1.0, GNU Nano 6.4, GNU NCURSES 6.3, GNU Readline 8.2- rc2, GNU Scientific Library 2.7, GNU Sed 4.8, GNU Tar 1.34, GNU Texinfo 6.8, GNU Wget 1.21.2, GNU Which 2.21, GPL Ghostscript 9.56.1, Help2man , Less 590, Libffi 3.4.2, Libgit2 1.3.0, libICE 1.0.10, Libidn 1.38, Libjpeg 9e, Libpaper 1.1.28, Libpng 1.6.37, libpthread-stubs (Xorg) 0.4, libSM 1.2.3, Libtiff 4.4.0, libXau (Xorg) 1.0.9, libxcb (Xorg) 1.15, libXdmcp (Xorg) 1.1.3, libXext 1.3.4, Libxml2 2.9.12, libXt 1.2.1, Lzip 1.23, OpenSSL 3.0.5, PatchELF 0.13, Perl 5.36.0, pkgconfig 0.29.2, podlators 4.14, Python 3.10.6, utilLinux 2.38.1, util-macros (Xorg) 1.19.3, WCSLIB 7.11, X11 library 1.8, XCB-proto (Xorg) 1.15, xorgproto 2022.1, xtrans (Xorg) 1.4.0, XZ Utils 5.2.5 and Zlib 1.2.11. The LATEX source of the paper was compiled to make the PDF using the following packages: courier 61719 (revision), epsf 2.7.4, etoolbox 2.5k, helvetic 61719 (revision), lineno 5.3, pgf 3.1.10, pgfplots 1.18.1, revtex4- 1 4.1s, tex 3.141592653, textcase 1.04 and ulem 53365 (revision). We are very grateful to all their creators for freely providing this necessary infrastructure. This research (and many other projects) would not be possible without them. This paper is available on arxiv under CC 4.0 DEED license. This paper is available on arxiv under CC 4.0 DEED license. available on arxiv

Gnuastro: Simulating The Exposure Map of a Pointing Pattern - Appendix

Author:
(1) Mohammad Akhlaghi, Centro de Estudios de F´ısica del Cosmos de Aragon (CEFCA), Plaza San Juan 1, 44001, Teruel, Spain {mohammad@akhlaghi.org}. Table of Links Abstract and Intro Simulating the Exposure Map Acknowledgment and References Appendix 3. ACKNOWLEDGEMENT The workflow of this research note was developed in the reproducible framework of Maneage (Managing data lineage, Akhlaghi et al. 2021, latest Maneage commit 8161194, from 22 May 2023). This note is created from the Git commit 4176d29 that is hosted on Codeberg[4] and is archived on SoftwareHeritage for longevity. The analysis of this research note was done using GNU Astronomy Utilities (Gnuastro, ascl.net/1801.009) version 0.20.72-08b0. Work on Gnuastro has been funded by the Japanese Ministry of Education, Culture, Sports, Science, and Technology (MEXT) scholarship and its Grant-in-Aid for Scientific Research (21244012, 24253003), the European Research Council (ERC) advanced grant 339659-MUSICOS, the Spanish Ministry of Economy and Competitiveness (MINECO, grant number AYA2016-76219-P) and the NextGenerationEU grant through the Recovery and Resilience Facility project ICTS-MRR-2021- 03-CEFCA. I also acknowledge the financial support provided by the Governments of Spain and Aragon through their general budgets and ´ the Fondo de Inversiones de Teruel, and the Spanish Ministry of Science and Innovation (MCIN/AEI/10.13039/501100011033 y FEDER, Una manera de hacer Europa) with grant PID2021- 124918NA-C43. REFERENCES Akhlaghi, M., & Ichikawa, T. 2015, ApJS, 220, 1, doi: 10.1088/0067-0049/220/1/1 Akhlaghi, M., Infante-Sainz, R., Roukema, B. F., et al. 2021, CiSE, 23, 82, doi: 10.1109/MCSE.2021.3072860 Calabretta, M. R., & Greisen, E. W. 2002, A&A, 395, 1077, doi: 10.1051/0004-6361:20021327 Illingworth, G. D., Magee, D., Oesch, P. A., et al. 2013, ApJS, 209, 6, doi: 10.1088/0067-0049/209/1/6 Trujillo, I., D’Onofrio, M., Zaritsky, D., et al. 2021, A&A, 654, A40, doi: 10.1051/0004-6361/202141603 This paper is available on arxiv under CC 4.0 DEED license. [4] https://codeberg.org/gnuastro/paper-pointing-simulate Author: (1) Mohammad Akhlaghi, Centro de Estudios de F´ısica del Cosmos de Aragon (CEFCA), Plaza San Juan 1, 44001, Teruel, Spain {mohammad@akhlaghi.org}. Author: Author: (1) Mohammad Akhlaghi, Centro de Estudios de F´ısica del Cosmos de Aragon (CEFCA), Plaza San Juan 1, 44001, Teruel, Spain {mohammad@akhlaghi.org}. Table of Links Abstract and Intro Abstract and Intro Simulating the Exposure Map Simulating the Exposure Map Acknowledgment and References Acknowledgment and References Appendix Appendix 3. ACKNOWLEDGEMENT The workflow of this research note was developed in the reproducible framework of Maneage (Managing data lineage, Akhlaghi et al. 2021, latest Maneage commit 8161194, from 22 May 2023). This note is created from the Git commit 4176d29 that is hosted on Codeberg[4] and is archived on SoftwareHeritage for longevity. The analysis of this research note was done using GNU Astronomy Utilities (Gnuastro, ascl.net/1801.009) version 0.20.72-08b0. Work on Gnuastro has been funded by the Japanese Ministry of Education, Culture, Sports, Science, and Technology (MEXT) scholarship and its Grant-in-Aid for Scientific Research (21244012, 24253003), the European Research Council (ERC) advanced grant 339659-MUSICOS, the Spanish Ministry of Economy and Competitiveness (MINECO, grant number AYA2016-76219-P) and the NextGenerationEU grant through the Recovery and Resilience Facility project ICTS-MRR-2021- 03-CEFCA. I also acknowledge the financial support provided by the Governments of Spain and Aragon through their general budgets and ´ the Fondo de Inversiones de Teruel, and the Spanish Ministry of Science and Innovation (MCIN/AEI/10.13039/501100011033 y FEDER, Una manera de hacer Europa) with grant PID2021- 124918NA-C43. REFERENCES Akhlaghi, M., & Ichikawa, T. 2015, ApJS, 220, 1, doi: 10.1088/0067-0049/220/1/1 Akhlaghi, M., Infante-Sainz, R., Roukema, B. F., et al. 2021, CiSE, 23, 82, doi: 10.1109/MCSE.2021.3072860 Calabretta, M. R., & Greisen, E. W. 2002, A&A, 395, 1077, doi: 10.1051/0004-6361:20021327 Illingworth, G. D., Magee, D., Oesch, P. A., et al. 2013, ApJS, 209, 6, doi: 10.1088/0067-0049/209/1/6 Trujillo, I., D’Onofrio, M., Zaritsky, D., et al. 2021, A&A, 654, A40, doi: 10.1051/0004-6361/202141603 This paper is available on arxiv under CC 4.0 DEED license. This paper is available on arxiv under CC 4.0 DEED license. available on arxiv [4] https://codeberg.org/gnuastro/paper-pointing-simulate

Gnuastro: Simulating The Exposure Map of a Pointing Pattern - Acknowledgment and References

Author:
(1) Mohammad Akhlaghi, Centro de Estudios de F´ısica del Cosmos de Aragon (CEFCA), Plaza San Juan 1, 44001, Teruel, Spain {mohammad@akhlaghi.org}. Table of Links Abstract and Intro Simulating the Exposure Map Acknowledgment and References Appendix 2. SIMULATING THE EXPOSURE MAP Within Gnuastro, the executable in charge of simulating the exposure map of a given list of pointings is called astscript-pointing- -simulate. As with all Gnuastro features, it has a complete documentation in Gnuastro’s manual[2] which is available in many formats (on the command-line, as web pages, PDF and etc). A complete and dedicated tutorial is also available within the tutorials chapter[3] of the manual. Three of the images produced in the tutorial (as of version 0.21) are displayed in Figure 1. The left image shows the exposure map that is produced from a five-point, “+”-shaped set of pointings on the sky, with one pointing in the center and four on the outer extents. The middle image uses one of the hooks that are available within the script to produce a noisy image and not an exposure map. This hook can also be used to insert simulated/real objects. The right image shows how another hook can be used to account for trimming of the outer edges (which sometimes do not get exposed enough due to vignetting). This hook can also be used to insert any random area of bad pixels or to simulate persistence in near infra-red detectors (these can be large). For its processing, this script needs two input files: 1) a table containing the RA and Dec of each pointing, 2) a FITS image from the camera that the dither pattern is meant for. The image has to contain the celestial world coordinate system (WCS, see Calabretta & Greisen 2002) headers. Having a WCS is very important, especially for cameras with a wide field of view, because distortions can become visible/significant. The tutorial and dedicated manual for this script are available in the Gnuastro manual and will always be up to date with the running version; therefore for the practical details, we encourage readers to follow the Gnuastro manual. The future changes in the script that break with this research note will be clearly designated in a “changes after publication” subsection of the section describing this script. This paper is available on arxiv under CC 4.0 DEED license. [2] https://www.gnu.org/software/gnuastro/manual [3] https://www.gnu.org/software/gnuastro/manual/html node/Tutorials.html Author: (1) Mohammad Akhlaghi, Centro de Estudios de F´ısica del Cosmos de Aragon (CEFCA), Plaza San Juan 1, 44001, Teruel, Spain {mohammad@akhlaghi.org}. Author: Author: (1) Mohammad Akhlaghi, Centro de Estudios de F´ısica del Cosmos de Aragon (CEFCA), Plaza San Juan 1, 44001, Teruel, Spain {mohammad@akhlaghi.org}. Table of Links Abstract and Intro Abstract and Intro Simulating the Exposure Map Simulating the Exposure Map Acknowledgment and References Acknowledgment and References Appendix Appendix 2. SIMULATING THE EXPOSURE MAP Within Gnuastro, the executable in charge of simulating the exposure map of a given list of pointings is called astscript-pointing- -simulate. As with all Gnuastro features, it has a complete documentation in Gnuastro’s manual[2] which is available in many formats (on the command-line, as web pages, PDF and etc). A complete and dedicated tutorial is also available within the tutorials chapter[3] of the manual. Three of the images produced in the tutorial (as of version 0.21) are displayed in Figure 1. The left image shows the exposure map that is produced from a five-point, “+”-shaped set of pointings on the sky, with one pointing in the center and four on the outer extents. The middle image uses one of the hooks that are available within the script to produce a noisy image and not an exposure map. This hook can also be used to insert simulated/real objects. The right image shows how another hook can be used to account for trimming of the outer edges (which sometimes do not get exposed enough due to vignetting). This hook can also be used to insert any random area of bad pixels or to simulate persistence in near infra-red detectors (these can be large). For its processing, this script needs two input files: 1) a table containing the RA and Dec of each pointing, 2) a FITS image from the camera that the dither pattern is meant for. The image has to contain the celestial world coordinate system (WCS, see Calabretta & Greisen 2002) headers. Having a WCS is very important, especially for cameras with a wide field of view, because distortions can become visible/significant. The tutorial and dedicated manual for this script are available in the Gnuastro manual and will always be up to date with the running version; therefore for the practical details, we encourage readers to follow the Gnuastro manual. The future changes in the script that break with this research note will be clearly designated in a “changes after publication” subsection of the section describing this script. This paper is available on arxiv under CC 4.0 DEED license. This paper is available on arxiv under CC 4.0 DEED license. available on arxiv [2] https://www.gnu.org/software/gnuastro/manual https://www.gnu.org/software/gnuastro/manual [3] https://www.gnu.org/software/gnuastro/manual/html node/Tutorials.html https://www.gnu.org/software/gnuastro/manual/html

Gnuastro: Simulating The Exposure Map of a Pointing Pattern - Simulating the Exposure Map

Author:
(1) Mohammad Akhlaghi, Centro de Estudios de F´ısica del Cosmos de Aragon (CEFCA), Plaza San Juan 1, 44001, Teruel, Spain {mohammad@akhlaghi.org}. Table of Links Abstract and Intro Simulating the Exposure Map Acknowledgment and References Appendix ABSTRACT The pointing pattern is an integral part of designing one’s observation strategy for a certain scientific goal. But accounting for the particular science case or instrument artifacts (like distortion, vignetting or large areas of bad pixels) can make it hard to predict how the exposure map of the final stack will be. To help address this problem, Gnuastro 0.21 includes a new executable program called astscript-pointing-simulate that is fully described in the Gnuastro manual and comes with a complete tutorial. The figures of this research note are reproducible with Maneage, on the Git commit 4176d29. Keywords: Astronomical methods (1043), Field of view (534), Direct imaging (387), Astronomical techniques (1684), Astronomy software (1855) 1. INTRODUCTION Astronomical images are often composed of many single exposures. After completing the observations, when the science topic does not depend on the time of observation, we stack those single exposures into one deep stacked image (also known as coadd). Designing the strategy to take those single exposures is therefore a very important aspect of planning an astronomical observation. Each exposure has a pointing (the location on the sky that it was targeting). Subsequent exposures are usually taken with different pointings. When the next pointing is ver near (a small fraction of the field of view) to the previous one, it is known as a dither (literally/generally meaning trembling or vibration). But when the distance between subsequent pointings is large (and issues like re-focusing become necessary), the pointing is known as an offset. These terms are sometimes used interchangably by some. When we only have dithers, most of the central part of the final stack has a fixed depth. Only a thin border becomes shallower (conveying a sense of vibration!). For example see Figures 3 and 4 of Illingworth et al. (2013) which show the exposures that went into the XDF survey. These types of images (where each pixel contains the number of exposures, or time, that were used in it) are known as exposure maps. However, it can happen that only offsets are used. For example see Figure 1 of Trujillo et al. (2021), which shows the exposure map for the LIGHTS survey. The dithering pattern therefore is strongly defined by the science case (high-level purpose of the observation) and your telescope’s field of view. For example in the HUDF, CANDELS and other proposals (that constitute the XDF), the scientific target was high redshift (distant) galaxies. When counting the pixels that they cover in relation to the number of pixels in each exposure, these targets are very small objects. Such that within that small footprint (of just 1 arcminute!) we have thousands of such small objects (which may also be stars or nearby galaxies of similar observed size to the high redshift galaxies). Therefore for the scientific goal of the XDF, the very small dithers were sufficient to avoid instrument artifacts. However, the LIGHTS survey is focused on the halos of large nearby galaxies (that can be more than 10 arcminutes wide); thus covering a large fraction of the field of view of the Large Binocular Telescope that was used. In order to do robust calibration of the images (for example to estimate the flat field pattern, or the sky background level) it was necessary to have offsets. This enabled the galaxy and its halo to cover very different pixels from one exposure to the next. In other cases, the pointings include both offsets and dithers (around each offset). To find the ideal dither pattern for the particular scientific goal, it therefore helps to be able to simulate various strategies. In the next section, an installed script is introduced for this purpose (to simulate the exposure map of a series of pointings) in GNU Astronomy Utilities (Gnuastro Akhlaghi & Ichikawa 2015) version[1] 0.21 and later. This paper is available on arxiv under CC 4.0 DEED license. [1] This paper is published shortly before the release of Gnuastro 0.21. If Gnuastro 0.21 is not yet available, please use the latest test (alpha) release. Author: (1) Mohammad Akhlaghi, Centro de Estudios de F´ısica del Cosmos de Aragon (CEFCA), Plaza San Juan 1, 44001, Teruel, Spain {mohammad@akhlaghi.org}. Author: Author: (1) Mohammad Akhlaghi, Centro de Estudios de F´ısica del Cosmos de Aragon (CEFCA), Plaza San Juan 1, 44001, Teruel, Spain {mohammad@akhlaghi.org}. Table of Links Abstract and Intro Abstract and Intro Simulating the Exposure Map Simulating the Exposure Map Acknowledgment and References Acknowledgment and References Appendix Appendix ABSTRACT The pointing pattern is an integral part of designing one’s observation strategy for a certain scientific goal. But accounting for the particular science case or instrument artifacts (like distortion, vignetting or large areas of bad pixels) can make it hard to predict how the exposure map of the final stack will be. To help address this problem, Gnuastro 0.21 includes a new executable program called astscript-pointing-simulate that is fully described in the Gnuastro manual and comes with a complete tutorial. The figures of this research note are reproducible with Maneage, on the Git commit 4176d29. Keywords : Astronomical methods (1043), Field of view (534), Direct imaging (387), Astronomical techniques (1684), Astronomy software (1855) Keywords 1. INTRODUCTION Astronomical images are often composed of many single exposures. After completing the observations, when the science topic does not depend on the time of observation, we stack those single exposures into one deep stacked image (also known as coadd). Designing the strategy to take those single exposures is therefore a very important aspect of planning an astronomical observation. Each exposure has a pointing (the location on the sky that it was targeting). Subsequent exposures are usually taken with different pointings. When the next pointing is ver near (a small fraction of the field of view) to the previous one, it is known as a dither (literally/generally meaning trembling or vibration). But when the distance between subsequent pointings is large (and issues like re-focusing become necessary), the pointing is known as an offset. These terms are sometimes used interchangably by some. When we only have dithers, most of the central part of the final stack has a fixed depth. Only a thin border becomes shallower (conveying a sense of vibration!). For example see Figures 3 and 4 of Illingworth et al. (2013) which show the exposures that went into the XDF survey. These types of images (where each pixel contains the number of exposures, or time, that were used in it) are known as exposure maps. However, it can happen that only offsets are used. For example see Figure 1 of Trujillo et al. (2021), which shows the exposure map for the LIGHTS survey. The dithering pattern therefore is strongly defined by the science case (high-level purpose of the observation) and your telescope’s field of view. For example in the HUDF, CANDELS and other proposals (that constitute the XDF), the scientific target was high redshift (distant) galaxies. When counting the pixels that they cover in relation to the number of pixels in each exposure, these targets are very small objects. Such that within that small footprint (of just 1 arcminute!) we have thousands of such small objects (which may also be stars or nearby galaxies of similar observed size to the high redshift galaxies). Therefore for the scientific goal of the XDF, the very small dithers were sufficient to avoid instrument artifacts. However, the LIGHTS survey is focused on the halos of large nearby galaxies (that can be more than 10 arcminutes wide); thus covering a large fraction of the field of view of the Large Binocular Telescope that was used. In order to do robust calibration of the images (for example to estimate the flat field pattern, or the sky background level) it was necessary to have offsets. This enabled the galaxy and its halo to cover very different pixels from one exposure to the next. In other cases, the pointings include both offsets and dithers (around each offset). To find the ideal dither pattern for the particular scientific goal, it therefore helps to be able to simulate various strategies. In the next section, an installed script is introduced for this purpose (to simulate the exposure map of a series of pointings) in GNU Astronomy Utilities (Gnuastro Akhlaghi & Ichikawa 2015) version[1] 0.21 and later. This paper is available on arxiv under CC 4.0 DEED license. This paper is available on arxiv under CC 4.0 DEED license. available on arxiv [1] This paper is published shortly before the release of Gnuastro 0.21. If Gnuastro 0.21 is not yet available, please use the latest test (alpha) release.

Gnuastro: Simulating The Exposure Map of a Pointing Pattern - Abstract and Intro

I'd like to share my experiences using evilginx2 and gophish for red teaming phishing simulations. While there are numerous guides available, I aim to highlight the nuances and practical insights that aren't typically covered in documentation and tutorials. Approaches When setting up a phishing campaign for red team purposes, you have three options: you can clone the page and only collect data
the next level is not only cloning the page but also verifying the captured data against a legitimate page
The last and most sophisticated method involves a man-in-the-middle attack, where your page is a proxy between the victim and the legitimate site. Evilginx2 The third method is done by evilginx2, which, despite its name, does not have nginx built in, but the functionality is the same, except that evilginx2 has built-in filters and functions like inject_js that make the attacker's life more comfortable. Gophish Gophish is a tool that allows you to manage groups and users, dynamically create emails, and send them at will. Integration A few weeks ago, Evilginx got an update 3.3, which introduced integration with Gophish, a framework for phishing campaigns. There have been several such projects in the past, such as gophish[1], but in this text, I will focus on the official evilginx by K. Gretzky. More info about the update with the demo can be found here: https://breakdev.org/evilginx-3-3-go-phish/ Both tools are in Golang, which is a relatively easy-to-understand language. Experience with Evilginx2 Malicious proxy with evilginx has several drawbacks compared to just cloning the page. Firstly, the setup is much more complicated. When you clone a page, you download the necessary files via developer tools and add the password-storing feature via some backend. Evilginx setup is more complicated. Setup needs a configuration file and a phishlet that works as a template for the phishing proxy. Setting up and running evilginx2 in the production version took me over a week compared to just cloning. The advantage is that evilginx scales much better. Once you know how to create them, you can change phishing scenarios at will because adding new phishlet is trivial. I should mention the state of the project in which evilginx is. The documentation is concise and not very useful if you have no experience with evilginx2 already. Some features and behaviors are not documented, and you need to read and debug the source code to find the behavior. The project has special financing where you can pay 400$ for a course of excellent quality, but it replaces the documentation to some extent, which doesn't seem like a good way. Either it's an open-source project with all the parts, or it's not open-source; this way, it's open-code, but you are forced into buying the expensive course. It doesn't help that the project has many open issues and bugs that nobody addresses. If you're serious about phishing, you'll probably have to edit the source code and use your version because the author doesn't put much effort into finding new PRs. For example, the PR request I made to fix the POST of a GET request three weeks[2] ago has still not been merged. The project is plagued by old libraries that are long unused. Of course, I don't want the author to handle everything himself, but sometimes, you have to give up control of the project to benefit the project. And more moderators and maintainers who want to help would certainly be found. But not to be only negative, this project is still the best of its kind. If you learn all its quirks and modify it to your purpose, it will become a tool allowing you to do campaigns of all kinds, with a level of automation that is impossible if you write the tooling yourself. Experience with Gophish Gophish isn't updated very often either, but I haven't encountered any bugs, so it doesn't really bother me. Although gophish has an API, it lacks some functionality that would be useful, so K.Gretzky had to add this API[3] to make the integration with evilginx work, anyway this approach is not sustainable and I think gophish would benefit from an API that would allow event reporting. The Gophish email template community is active, and you don't need to invest much time in a replica of the o365 email because there is a good chance that someone has already created the template. The UI is very good, and many features, like a timeline of events, will be appreciated by clients in the report. Bypassing anti measures If you send emails within a known service such as o365 or Gmail, it is good to first build an instance where you collect crawler IP addresses which you then block on the production instance and domain, thus minimizing the risk of being blacklisted, for example, outlook. Of course, this is the easier part. The harder part is to have an SMPT server with a good reputation that allows you to send phishing emails in bulk, so I do not recommend building your own SMTP server but rather use some email as a service with its own domain that has a good server reputation by default, for example, Gmail business. Summary While evilginx2 offers powerful features and scalability, its complex setup and poor documentation are significant hurdles.
Gophish provides robust campaign management tools but also suffers from limited API functionality.
Both tools require considerable customization and understanding to effectively use in red team operations.
Mastering both tools offers significant rewards. Thank you for reading. Feedback is appreciated! References [1]: https://github.com/fin3ss3g0d/evilgophish [2]: https://github.com/kgretzky/evilginx2/pull/1047 [3]: https://github.com/kgretzky/gophish I'd like to share my experiences using evilginx2 and gophish for red teaming phishing simulations. While there are numerous guides available, I aim to highlight the nuances and practical insights that aren't typically covered in documentation and tutorials. Approaches When setting up a phishing campaign for red team purposes, you have three options: you can clone the page and only collect data the next level is not only cloning the page but also verifying the captured data against a legitimate page The last and most sophisticated method involves a man-in-the-middle attack, where your page is a proxy between the victim and the legitimate site. you can clone the page and only collect data the next level is not only cloning the page but also verifying the captured data against a legitimate page The last and most sophisticated method involves a man-in-the-middle attack, where your page is a proxy between the victim and the legitimate site. Evilginx2 The third method is done by evilginx2 , which, despite its name, does not have nginx built in, but the functionality is the same, except that evilginx2 has built-in filters and functions like inject_js that make the attacker's life more comfortable. evilginx2 evilginx2 nginx evilginx2 inject_js Gophish Gophish is a tool that allows you to manage groups and users, dynamically create emails, and send them at will. Gophish Gophish Integration A few weeks ago, Evilginx got an update 3.3 , which introduced integration with Gophish, a framework for phishing campaigns. There have been several such projects in the past, such as gophish[1], but in this text, I will focus on the official evilginx by K. Gretzky. 3.3 More info about the update with the demo can be found here: https://breakdev.org/evilginx-3-3-go-phish/ https://breakdev.org/evilginx-3-3-go-phish/ Both tools are in Golang, which is a relatively easy-to-understand language. Experience with Evilginx2 Malicious proxy with evilginx has several drawbacks compared to just cloning the page. Firstly, the setup is much more complicated. When you clone a page, you download the necessary files via developer tools and add the password-storing feature via some backend. Evilginx setup is more complicated. Setup needs a configuration file and a phishlet that works as a template for the phishing proxy. Setting up and running evilginx2 in the production version took me over a week compared to just cloning. The advantage is that evilginx scales much better. Once you know how to create them, you can change phishing scenarios at will because adding new phishlet is trivial. I should mention the state of the project in which evilginx is. The documentation is concise and not very useful if you have no experience with evilginx2 already. Some features and behaviors are not documented, and you need to read and debug the source code to find the behavior. The project has special financing where you can pay 400$ for a course of excellent quality, but it replaces the documentation to some extent, which doesn't seem like a good way. Either it's an open-source project with all the parts, or it's not open-source; this way, it's open-code, but you are forced into buying the expensive course. It doesn't help that the project has many open issues and bugs that nobody addresses. If you're serious about phishing, you'll probably have to edit the source code and use your version because the author doesn't put much effort into finding new PRs. For example, the PR request I made to fix the POST of a GET request three weeks[2] ago has still not been merged. The project is plagued by old libraries that are long unused. Of course, I don't want the author to handle everything himself, but sometimes, you have to give up control of the project to benefit the project. And more moderators and maintainers who want to help would certainly be found. But not to be only negative, this project is still the best of its kind. If you learn all its quirks and modify it to your purpose, it will become a tool allowing you to do campaigns of all kinds, with a level of automation that is impossible if you write the tooling yourself. Experience with Gophish Gophish isn't updated very often either, but I haven't encountered any bugs, so it doesn't really bother me. Although gophish has an API, it lacks some functionality that would be useful, so K.Gretzky had to add this API[3] to make the integration with evilginx work, anyway this approach is not sustainable and I think gophish would benefit from an API that would allow event reporting. The Gophish email template community is active, and you don't need to invest much time in a replica of the o365 email because there is a good chance that someone has already created the template. The UI is very good, and many features, like a timeline of events, will be appreciated by clients in the report. Bypassing anti measures If you send emails within a known service such as o365 or Gmail, it is good to first build an instance where you collect crawler IP addresses which you then block on the production instance and domain, thus minimizing the risk of being blacklisted, for example, outlook. Of course, this is the easier part. The harder part is to have an SMPT server with a good reputation that allows you to send phishing emails in bulk, so I do not recommend building your own SMTP server but rather use some email as a service with its own domain that has a good server reputation by default, for example, Gmail business. Summary While evilginx2 offers powerful features and scalability, its complex setup and poor documentation are significant hurdles. Gophish provides robust campaign management tools but also suffers from limited API functionality. Both tools require considerable customization and understanding to effectively use in red team operations. Mastering both tools offers significant rewards. While evilginx2 offers powerful features and scalability, its complex setup and poor documentation are significant hurdles. Gophish provides robust campaign management tools but also suffers from limited API functionality. Both tools require considerable customization and understanding to effectively use in red team operations. Mastering both tools offers significant rewards. Thank you for reading. Feedback is appreciated! References [1]: https://github.com/fin3ss3g0d/evilgophish https://github.com/fin3ss3g0d/evilgophish [2]: https://github.com/kgretzky/evilginx2/pull/1047 https://github.com/kgretzky/evilginx2/pull/1047 [3]: https://github.com/kgretzky/gophish https://github.com/kgretzky/gophish

Red Team Phishing Simulations: Using Evilginx2 and GoPhish

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Jihoon Chung;
(2)  Zhenyu (James) Kong. Table of Links Abstract & Introduction
Review of Related Work
Proposed Research Methodology
Numerical Case Studies
Real-World Simulation Case Studies
Conclusion
References V. REAL-WORLD SIMULATION CASE STUDIES An assembly operation from an actual auto-body assembly process is used as a real-world case study. Fig. 5 describes a car’s floor pan, which is the assembled product from this process. The assembled product consists of four parts, including the right bracket, left bracket, right floor pan, and left floor pan. Fig. 6 shows the process assembly procedure consisting of three stations. During the assembly process, the parts are held by fixtures, which are the KCCs in this process [7]. KPCs are measured from four points, namely, M1, M2, M3, and M4, respectively, as shown in Fig. 5. Every part has a designated location for measuring the KPCs. For instance, part 1 has M1, part 2 has M2, part 3 has M3, and part 4 has M4. These measurements can be taken at each station once the relevant part has been assembled in the preceding stations. For example, M3 on part 3 cannot be measured in station 1 because part 3 has not yet been assembled at station 1 [7]. In the designated location, KPCs are measured in three directions (X, Y, and Z). In this assembly process, there exists a total of 33 KCCs, which are dimensional errors of fixture locators [8]. The fault pattern matrix Φ is established based on the literature [7], [11], [47] and provided in Appendix F. Since the number of measurements (12) is less than that of KCCs (33), it causes an underdetermined system in the fault quality linear model. Therefore, sparse estimation is required to identify process faults. In this assembly process, two lists of correlated KCCs exist; the rest are assumed to be independent [8]. The first list comprises six KCCs, including KCC8, KCC9, KCC10, KCC11, KCC12, and KCC13. The second list is composed of KCC31, KCC32, and KCC33. The correlation structures of the two groups are described (for simplification, the lower triangular terms are removed) as follows [8]: To provide nonstationary process faults along the KPCs samples collected over time, two groups are provided. In the first group, the first correlated list is provided as process faults. For the process faults of the second group, all the correlated lists and one independent KCCs are included. Therefore, the case study represents the situation when the process faults evolve over time since the variation of certain KCCs will be propagated to other KCCs if the process faults are not mitigated immediately during the process. 50 KPCs samples are generated from each group. The variances of process faults, non-process faults, and noise of KPCs are determined to be the same value as in Section IV. In addition, performance evaluation measures and benchmark methods used in Section IV are still utilized in Section V. Table III shows the performance comparison in AUC and NMSE by varying the correlation coefficients. For convenience, the correlation coefficients k1 and k2 are equal and are selected from {0.1, 0.3, 0.6, 0.9, 0.95} in the case studies. Specifically, the AUC of the proposed method is close to 1.0 at every correlations levels, representing the perfect classification between the process faults and non-process faults in terms of estimated variance. The effectiveness of the proposed method is significant in NMSE when a high correlation exists. Specifically, the NMSE of the proposed method is less than 10% of that of all the benchmark methods when the correlation coefficient is 0.9 and 0.95. To demonstrate the generalizability of the proposed method, the correlated KCCs are randomly selected in further case studies. Assuming there exist two correlated lists with the size of three, respectively. Therefore, six KCCs are randomly selected from 33 KCCs for two correlated lists in each trial. As in the previous study, one correlated list is provided as process faults in the first group, while two correlated lists and two independent KCCs are used as process faults in the remaining second group. Fig. 7 shows the performances of all the methods in AUC and NMSE results in ten different spatial correlations (i.e., k1 = k2). From Fig. 7 (a), it is prominent that the proposed method successfully differentiates between process faults and non-process faults. Specifically, the AUC of the proposed method is nearly 1.0, while those of most benchmark methods are below 0.9 in the highly correlated case studies. In addition, Fig. 7 (b) represents that the proposed method provides valuable information to the practitioners to determine when to stop the operations to maintain the process when a high correlation exists between KCCs. All the benchmark methods illustrate similar trends compared to Section IV since the methods cannot consider both the spatial correlation of KCCs and the nonstationary process faults. This paper is available on arxiv under CC 4.0 license. Authors: (1) Jihoon Chung; (2)  Zhenyu (James) Kong. This paper is available on arxiv under CC 4.0 license. Authors: (1) Jihoon Chung; (2)  Zhenyu (James) Kong. Table of Links Abstract & Introduction Review of Related Work Proposed Research Methodology Numerical Case Studies Real-World Simulation Case Studies Conclusion References Abstract & Introduction Abstract & Introduction Review of Related Work Review of Related Work Proposed Research Methodology Proposed Research Methodology Numerical Case Studies Numerical Case Studies Real-World Simulation Case Studies Real-World Simulation Case Studies Conclusion Conclusion References References V. REAL-WORLD SIMULATION CASE STUDIES An assembly operation from an actual auto-body assembly process is used as a real-world case study. Fig. 5 describes a car’s floor pan, which is the assembled product from this process. The assembled product consists of four parts, including the right bracket, left bracket, right floor pan, and left floor pan. Fig. 6 shows the process assembly procedure consisting of three stations. During the assembly process, the parts are held by fixtures, which are the KCCs in this process [7]. KPCs are measured from four points, namely, M1, M2, M3, and M4, respectively, as shown in Fig. 5. Every part has a designated location for measuring the KPCs. For instance, part 1 has M1, part 2 has M2, part 3 has M3, and part 4 has M4. These measurements can be taken at each station once the relevant part has been assembled in the preceding stations. For example, M3 on part 3 cannot be measured in station 1 because part 3 has not yet been assembled at station 1 [7]. In the designated location, KPCs are measured in three directions (X, Y, and Z). In this assembly process, there exists a total of 33 KCCs, which are dimensional errors of fixture locators [8]. The fault pattern matrix Φ is established based on the literature [7], [11], [47] and provided in Appendix F. Since the number of measurements (12) is less than that of KCCs (33), it causes an underdetermined system in the fault quality linear model. Therefore, sparse estimation is required to identify process faults. In this assembly process, two lists of correlated KCCs exist; the rest are assumed to be independent [8]. The first list comprises six KCCs, including KCC8, KCC9, KCC10, KCC11, KCC12, and KCC13. The second list is composed of KCC31, KCC32, and KCC33. The correlation structures of the two groups are described (for simplification, the lower triangular terms are removed) as follows [8]: To provide nonstationary process faults along the KPCs samples collected over time, two groups are provided. In the first group, the first correlated list is provided as process faults. For the process faults of the second group, all the correlated lists and one independent KCCs are included. Therefore, the case study represents the situation when the process faults evolve over time since the variation of certain KCCs will be propagated to other KCCs if the process faults are not mitigated immediately during the process. 50 KPCs samples are generated from each group. The variances of process faults, non-process faults, and noise of KPCs are determined to be the same value as in Section IV. In addition, performance evaluation measures and benchmark methods used in Section IV are still utilized in Section V. Table III shows the performance comparison in AUC and NMSE by varying the correlation coefficients. For convenience, the correlation coefficients k1 and k2 are equal and are selected from {0.1, 0.3, 0.6, 0.9, 0.95} in the case studies. Specifically, the AUC of the proposed method is close to 1.0 at every correlations levels, representing the perfect classification between the process faults and non-process faults in terms of estimated variance. The effectiveness of the proposed method is significant in NMSE when a high correlation exists. Specifically, the NMSE of the proposed method is less than 10% of that of all the benchmark methods when the correlation coefficient is 0.9 and 0.95. To demonstrate the generalizability of the proposed method, the correlated KCCs are randomly selected in further case studies. Assuming there exist two correlated lists with the size of three, respectively. Therefore, six KCCs are randomly selected from 33 KCCs for two correlated lists in each trial. As in the previous study, one correlated list is provided as process faults in the first group, while two correlated lists and two independent KCCs are used as process faults in the remaining second group. Fig. 7 shows the performances of all the methods in AUC and NMSE results in ten different spatial correlations (i.e., k1 = k2). From Fig. 7 (a), it is prominent that the proposed method successfully differentiates between process faults and non-process faults. Specifically, the AUC of the proposed method is nearly 1.0, while those of most benchmark methods are below 0.9 in the highly correlated case studies. In addition, Fig. 7 (b) represents that the proposed method provides valuable information to the practitioners to determine when to stop the operations to maintain the process when a high correlation exists between KCCs. All the benchmark methods illustrate similar trends compared to Section IV since the methods cannot consider both the spatial correlation of KCCs and the nonstationary process faults.

Sparse Bayesian Learning: Real-World Simulation Case Studies

The Noonification: Using pgvector To Locate Similarities In Enterprise Data (3/22/2024)

Using pgvector To Locate Similarities In Enterprise Data

এন্টারপ্রাইজ ডেটাতে মিল খুঁজে পেতে pgvector ব্যবহার করা

Verwenden von pgvector zum Auffinden von Ähnlichkeiten in Unternehmensdaten

Uso de pgvector para localizar similitudes en datos empresariales

Utilisation de pgvector pour localiser les similitudes dans les données d'entreprise

एंटरप्राइज़ डेटा में समानताएँ खोजने के लिए pgvector का उपयोग करना

pgvector を使用して企業データ内の類似点を特定する

pgVector를 사용하여 기업 데이터의 유사점 찾기

Usando pgvector para localizar semelhanças em dados corporativos

Использование pgvector для поиска сходств в корпоративных данных

Kurumsal Verilerdeki Benzerlikleri Bulmak İçin pgvector'u Kullanma

Sử dụng pgvector để xác định điểm tương đồng trong dữ liệu doanh nghiệp

使用 pgvector 查找企业数据中的相似之处

Comprehending the laws and boundaries of nature enhances our skills as programmers. In recent years, a fascinating analogy has emerged, drawing parallels between the universe and a concept familiar to computer scientists: the discrete event simulator. The universe operates much like a massive simulation coordinated without a central clock ticking. Let's dive in... Understanding the Universe as a Discrete Event Simulator Reality operates in discrete increments of time and space. Rather than a continuous flow, events unfold at specific points in time, triggering state transitions within the cosmic fabric. This concept aligns with the principles of , a computational method used to model systems where events occur at distinct moments. discrete-event simulation Time advances in discrete steps, with events driving changes in system states with discrete units of time. The universe appears to operate on a fundamental level where time and space are quantized, meaning they consist of indivisible units. indivisible Researchers think that Planck units encapsulate this discretization of spacetime. Physicists named Planck units after the physicist . Max Planck Planck Units and the Fabric of Reality represent the smallest possible length, time, mass, and other physical quantities within the framework of quantum mechanics. Planck units Units are derived from fundamental constants such as the , Planck's constant, and the . speed of light gravitational constant The represents the smallest possible length scale, approximately . Planck length 1.616 × 10^-35 meters The , approximately , marks the smallest possible time interval, beyond which time loses its meaning. Planck time 5.391 × 10^-44 seconds Planck time is the time it takes for light to travel a distance equal to the Planck length in a vacuum. This minimal duration represents the shortest meaningful interval of time that can be defined within our current understanding of physics. At durations shorter than the Planck time, the concept of time itself becomes ambiguous, and the laws of physics as we know them break down. You can't break plank units in the same way you cannot break . fundamental particles Ancient Greeks created the notion of indivisible particles, which they termed "atoms." The term "atom" originates from the Greek word "atomos," where "a-" means "not" or "un-" and "tomos" means "cut" or "divided." Atoms are not the least divisible particle but the brilliant idea remains. Implications for Computer Simulations The analogy between the universe and a discrete-event simulator has profound implications for computer simulations and scientific inquiry. https://hackernoon.com/what-is-wrong-with-software-uh8j3y7k?embedable=true Once you acknowledge the between the two realms, you can gain new insights into the nature of reality and develop more efficient simulation techniques. parallels The concept of the universe as a discrete event simulator challenges traditional notions of computation. In the past, computer scientists had a lot of trouble dealing with infinities, problems, , and dealing with divisible units. precision floats For example, comparing two dates was always a problem since we usually represent them as arbitrary divisible points in time. Navigating non-infinitely divisible money amounts, distances, and other measurements has sparked numerous renowned challenges like the , the , the , etc. Genesis mission Mars Climate Orbiter Ariane 5 Flight 501 Conclusion The universe as a discrete event simulator offers a compelling framework for understanding the fundamental principles governing reality. Planck units set a barrier on where to stop diving the measurements. I am not telling you to model timestamps with plank units precision. I am asking you to change your framework and acknowledge we are not dealing with arbitrary precision problems but with finite precision ones. We should leave problems related to infinities only to math domains, not real-world simulators.

Warning: The Universe's Event Simulator Is a Fierce Adversary for Coders!

Writing Effective Integration Tests in Spring: Organized Testing Strategies for HTTP Request Mocking

Assessing the Interpretability of ML Models from a Human Perspective

Authors:
(1) Omid Davoodi, Carleton University, School of Computer Science;
(2) Shayan Mohammadizadehsamakosh, Sharif University of Technology, Department of Computer Engineering;
(3) Majid Komeili, Carleton University, School of Computer Science. Table of Links Abstract and Intro Background Information Methodology Prototype Interpretability Prototype-query Similarity Interpretability of the Decision-Making Process The Effects of Low Prototype Counts Discussions Interpretability of the Decision-Making Process Our final experiment was designed to assess the interpretability of the decision-making process. In general, part-prototype methods offer a set of explanations for a decision and then use those concepts for a final classification step, which is usually a fully connected layer. We designed our framework for cases where the final decision is made using such a final layer over all of the prototype activations and so we tried to stay as faithful as possible to the actual decision-making process. This meant that we had to make sure we included the activation scores and rankings of the prototypes in the experiment design. ACE is not a classification method and there is no final decision to make. As a result, ACE, alongside ProtoTree, is not included in this experiment. In the case of ProtoTree, the method uses a decision-tree structure to offer its explanations. This is not compatible with our experiment design. Moreover, the agreement and distinction tests designed by HIVE[12] are adequate for evaluating the interpretability here as they use the same tree structure as the method itself. In the end, the authors of HIVE found that the tree structure was interpretable by humans. This cannot be said for the remaining five methods. A challenge in interpretability is to show enough information to humans so that they can understand the process without overwhelming them. For example, we cannot expect a human to look at the entire 2000 prototypes and their activations in order to understand how ProtoPNet classifies a query image from the CUB dataset. Many part-prototype models seem to have opted to use the 10 most activated prototypes for a particular query image as a good proxy for the entire model9, 18, 20. 10 prototypes are much more manageable for a human to understand than 2000, but we have to make sure that they are also a good proxy for the model itself. If the model, using those same 10 prototypes, is able to classify the query image and come to the same decision as before (regardless of whether that original decision was correct or not), then those 10 prototypes are a good proxy for the decision-making process of the model for that particular query image. This implies that the model selectively utilizes a limited number of prototypes from its extensive pool to categorize images. This is unfortunately not always the case. For some query images, the model might use more than 10 prototypes to come up with its final decision. In those cases, we can argue that the interpretability of the method will suffer, as it relies on a higher number of prototypes, which, in turn, can overwhelm humans. As a result, finding out the model’s accuracy with only a limited number of prototypes can be a good indicator of the interpretability of the decision-making process. We performed experiments where we only kept the top 1, 3, 5, 10, 15, 20, 30, 50, and top 100 prototypes for each method on the CUB dataset, and For each reduced model, we calculated the percentage of its decisions that agree with the decisions of the model that uses the complete prototype set. The results of these can be seen in Figure 7. As can be seen, not all methods are equal in this regard. Some like ProtoPNet and Deformable ProtoPNet can achieve the same decisions using relatively few prototypes. Others like ProtoPool utilize a large amount of prototypes to classify most of the query images. Only 12% of the decisions of ProtoPool remain the same when only the 10 top prototypes are kept. This observed low percentage is concerning especially as it only goes up to 39% after using 100 prototypes out of the 202 prototypes this model had for the CUB dataset. After this step, we classified the same query samples utilized in the previous experiments using our models but only kept the ones where the decision of the model did not change after keeping only the 10 top activated prototypes. For each of these query images that passed this filter, we then gathered the top 10 prototypes and their activation regions on the query image into an explanation set. The prototypes in this explanation set were sorted from the highest activation to the lowest. As the activation values themselves might be different for each query image and might not mean anything to a human, we used a confidence system where the top activated prototype has a relative confidence level of 100% and every other prototype has its confidence level scaled by its activation value compared to the activation value of the top prototype. We then included the class label associated with that prototype in the explanation set. In the case of ProtoPool which doesn’t have a single class associated with each prototype, we picked the 3 top classes associated with that prototype. These classes were obfuscated with numbers so that the role of prior human knowledge could be reduced. Examples of these explanation sets can be found in Figure 8. We then asked our human participants on Amazon Mechanical Turk to guess the class that was picked for that query image by our classifier model. Our instructions for the AMT participants were as follows: • Given a photo of an animal or a car (Query), an AI is predicting the species or the model of the subject of the photo using prototypes it has learned from previously seen photos. Specifically for each prototype, the AI identifies a region in the photo (Query) that looks the most similar to a region (marked with a yellow border) in the prototype and rates its relative confidence in their similarity. Each prototype is associated with one, or multiple animal species/car models. The AI will pick the species/type based on how similar and frequent the prototypes are to the query. • For each query, we show explanations on how the AI reasons the species of the animals or the model of the car. Looking at the species/model numbers for each query-prototype pair, guess the species/model number that you would think the AI predicted for the query. • Guessing randomly will get you low overall accuracy depending on the number of options. You will only get rewards if your performance is sufficiently higher than random guesses. • Remember that you can zoom-in/out and pan the image using the buttons below the image. The idea behind this experiment is similar to that of the distinction task from HIVE. In order for a human to understand the decision-making process of an AI system, they should be able to correctly predict the decision of the AI system from the same information it uses. In total, 607 explanation sets were shown to and annotated by the participants. Table 9 shows the percentage of query samples in which humans correctly predicted the predictions of the models. Apart from ProtoPool, human participants seem to be able to understand the underlying principles behind the final decision-making process of the rest of the methods. Prototypes in ProtoPool can belong to multiple classes and this causes severe issues with understanding how the method makes its decisions. The stark contrast between ProtoPool and the others shows that this evaluation criteria is a useful way of determining problems in the interpretability of Prototype-based methods. This paper is available on arxiv under CC 4.0 license. Authors: (1) Omid Davoodi, Carleton University, School of Computer Science; (2) Shayan Mohammadizadehsamakosh, Sharif University of Technology, Department of Computer Engineering; (3) Majid Komeili, Carleton University, School of Computer Science. Authors: Authors: (1) Omid Davoodi, Carleton University, School of Computer Science; (2) Shayan Mohammadizadehsamakosh, Sharif University of Technology, Department of Computer Engineering; (3) Majid Komeili, Carleton University, School of Computer Science. Table of Links Abstract and Intro Abstract and Intro Background Information Background Information Methodology Methodology Prototype Interpretability Prototype Interpretability Prototype-query Similarity Prototype-query Similarity Interpretability of the Decision-Making Process Interpretability of the Decision-Making Process The Effects of Low Prototype Counts The Effects of Low Prototype Counts Discussions Discussions Interpretability of the Decision-Making Process Our final experiment was designed to assess the interpretability of the decision-making process. In general, part-prototype methods offer a set of explanations for a decision and then use those concepts for a final classification step, which is usually a fully connected layer. We designed our framework for cases where the final decision is made using such a final layer over all of the prototype activations and so we tried to stay as faithful as possible to the actual decision-making process. This meant that we had to make sure we included the activation scores and rankings of the prototypes in the experiment design. ACE is not a classification method and there is no final decision to make. As a result, ACE, alongside ProtoTree, is not included in this experiment. In the case of ProtoTree, the method uses a decision-tree structure to offer its explanations. This is not compatible with our experiment design. Moreover, the agreement and distinction tests designed by HIVE[12] are adequate for evaluating the interpretability here as they use the same tree structure as the method itself. In the end, the authors of HIVE found that the tree structure was interpretable by humans. This cannot be said for the remaining five methods. A challenge in interpretability is to show enough information to humans so that they can understand the process without overwhelming them. For example, we cannot expect a human to look at the entire 2000 prototypes and their activations in order to understand how ProtoPNet classifies a query image from the CUB dataset. Many part-prototype models seem to have opted to use the 10 most activated prototypes for a particular query image as a good proxy for the entire model9, 18, 20. 10 prototypes are much more manageable for a human to understand than 2000, but we have to make sure that they are also a good proxy for the model itself. If the model, using those same 10 prototypes, is able to classify the query image and come to the same decision as before (regardless of whether that original decision was correct or not), then those 10 prototypes are a good proxy for the decision-making process of the model for that particular query image. This implies that the model selectively utilizes a limited number of prototypes from its extensive pool to categorize images. This is unfortunately not always the case. For some query images, the model might use more than 10 prototypes to come up with its final decision. In those cases, we can argue that the interpretability of the method will suffer, as it relies on a higher number of prototypes, which, in turn, can overwhelm humans. As a result, finding out the model’s accuracy with only a limited number of prototypes can be a good indicator of the interpretability of the decision-making process. We performed experiments where we only kept the top 1, 3, 5, 10, 15, 20, 30, 50, and top 100 prototypes for each method on the CUB dataset, and For each reduced model, we calculated the percentage of its decisions that agree with the decisions of the model that uses the complete prototype set. The results of these can be seen in Figure 7. As can be seen, not all methods are equal in this regard. Some like ProtoPNet and Deformable ProtoPNet can achieve the same decisions using relatively few prototypes. Others like ProtoPool utilize a large amount of prototypes to classify most of the query images. Only 12% of the decisions of ProtoPool remain the same when only the 10 top prototypes are kept. This observed low percentage is concerning especially as it only goes up to 39% after using 100 prototypes out of the 202 prototypes this model had for the CUB dataset. After this step, we classified the same query samples utilized in the previous experiments using our models but only kept the ones where the decision of the model did not change after keeping only the 10 top activated prototypes. For each of these query images that passed this filter, we then gathered the top 10 prototypes and their activation regions on the query image into an explanation set. The prototypes in this explanation set were sorted from the highest activation to the lowest. As the activation values themselves might be different for each query image and might not mean anything to a human, we used a confidence system where the top activated prototype has a relative confidence level of 100% and every other prototype has its confidence level scaled by its activation value compared to the activation value of the top prototype. We then included the class label associated with that prototype in the explanation set. In the case of ProtoPool which doesn’t have a single class associated with each prototype, we picked the 3 top classes associated with that prototype. These classes were obfuscated with numbers so that the role of prior human knowledge could be reduced. Examples of these explanation sets can be found in Figure 8. We then asked our human participants on Amazon Mechanical Turk to guess the class that was picked for that query image by our classifier model. Our instructions for the AMT participants were as follows: • Given a photo of an animal or a car (Query), an AI is predicting the species or the model of the subject of the photo using prototypes it has learned from previously seen photos. Specifically for each prototype, the AI identifies a region in the photo (Query) that looks the most similar to a region (marked with a yellow border) in the prototype and rates its relative confidence in their similarity. Each prototype is associated with one, or multiple animal species/car models. The AI will pick the species/type based on how similar and frequent the prototypes are to the query. • For each query, we show explanations on how the AI reasons the species of the animals or the model of the car. Looking at the species/model numbers for each query-prototype pair, guess the species/model number that you would think the AI predicted for the query. • Guessing randomly will get you low overall accuracy depending on the number of options. You will only get rewards if your performance is sufficiently higher than random guesses. • Remember that you can zoom-in/out and pan the image using the buttons below the image. The idea behind this experiment is similar to that of the distinction task from HIVE. In order for a human to understand the decision-making process of an AI system, they should be able to correctly predict the decision of the AI system from the same information it uses. In total, 607 explanation sets were shown to and annotated by the participants. Table 9 shows the percentage of query samples in which humans correctly predicted the predictions of the models. Apart from ProtoPool, human participants seem to be able to understand the underlying principles behind the final decision-making process of the rest of the methods. Prototypes in ProtoPool can belong to multiple classes and this causes severe issues with understanding how the method makes its decisions. The stark contrast between ProtoPool and the others shows that this evaluation criteria is a useful way of determining problems in the interpretability of Prototype-based methods. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Human Understanding of AI Decisions

Authors:
(1) Omid Davoodi, Carleton University, School of Computer Science;
(2) Shayan Mohammadizadehsamakosh, Sharif University of Technology, Department of Computer Engineering;
(3) Majid Komeili, Carleton University, School of Computer Science. Table of Links Abstract and Intro Background Information Methodology Prototype Interpretability Prototype-query Similarity Interpretability of the Decision-Making Process The Effects of Low Prototype Counts Discussions Prototype Interpretability Our first set of experiments had the goal of finding out the interpretability of the prototypes created by each method. Many part prototype methods claim that their prototypes are intrinsically interpretable. In that case, humans should be able to understand the concept that is encoded within those prototypes. If a prototype represents a large area of an image without any coherent concept understandable to humans, it is not an interpretable prototype. In the first experiment, each person was shown a pair of prototypes (concept images in the case of ACE) on the same dataset coming from different methods. The annotators had to choose the prototype they found more interpretable out of the two. Every person had to complete 10 such pairs each time. Out of those 10, 2 were validation pairs handpicked by us to ensure answer quality. An example of such a pair and the instructions given to each person are show in Figure 4. Only people who had finished a qualification task consisting of 7 such pairs with perfect scores were allowed into the main study. Moreover, each 10-pair set of query samples was given to three different people to gauge overall agreement and find individuals who might have chosen randomly. Additionally, this experiment was repeated 3 times and the results were aggregated. This experiment was a purely comparative study between different methods. This ensured that people had a frame of reference when deciding which method worked better, and this is a common practice in many studies of subjective nature. But they are not without problems. One of the more important problems with this experiment was that it could not answer whether the prototypes from these methods were interpretable or not. It could only answer whether the prototypes of a particular method were more interpretable on average than the ones from another. To remedy this, we conducted another set of experiments. In the new absolute experiments the annotators were given the option to also choose "Both" and "None" if they thought that both or none of the images in the pair were interpretable. The reason we did not simply show them a single image asking for interpretability was to keep the layout of the experiments very similar. This not only simplified the logistics of creating query samples for the new experiment, but also gave us the opportunity to see whether the rankings of the methods seen in the comparative experiments remained the same when the new options were added. The assumption was that if a very similar experiment with those options added resulted in a drastic change in the rankings, it would reveal problems in our methodology that were hidden beforehand. Instructions given to each person were as follows: • For each pair of images, pick the one with the best highlighted patch. A good patch contains relevant and identifying regions/parts of the subject, and only such regions/parts. • If both images have good or bad patches, select "Both" or "None" respectively. • Imagine that someone points to the marked region and says: "This is an image of a bird/car/lion/etc. because of this part“. How much would you agree with that statement? It is important to note that new image pairs were created for this experiment. This was again to see if a new set of data changes the rankings considerably, revealing any potential hidden problems with the statistical significance of the results. A new qualification task and a validation set were also created. The rest of the setup was almost identical to the comparative experiment. Each time, the annotator was given 10 prototype image pairs and told to pick the best option for each pair, with two out of those 10 pairs coming from the validation set. The experiment was repeated 4 times and the results were aggregated. In total, 1104 pairs were compared and annotated by humans for the comparative study. Table 1 shows the percentage of times each method was chosen against others. As can be seen, prototypes selected by TesNet were chosen 73.26% of the time when compared to prototypes selected by other methods. In the second place was ProtoTree with 63.64% preference. It is also interesting to note the relative weakness of SPARROW and Deformable ProtoPNet. Table 2 shows the results of the same experiments but on a method-by-method basis. Each element of the table shows the results of the pairwise comparison between the two methods, with the method shown in the first column being the one selected out of the two. For example, the intersection between the row "ACE" and the column "ProtoPool" shows the percentage of times (42.86%) ACE was chosen against ProtoPool in the experiments. While there is more variation when looked at with this much detail, the general trends hold true. For the absolute experiments, 1304 pairs were compared and annotated by humans in total. Table 3 shows the percentage of times each method’s prototypes were chosen in general. As the options for "Both" and "None" also existed in this experiment, the results here show the percentage of prototypes from each method that the annotators found acceptable. While the overall rankings seem to have mostly remained the same, the higher percentages overall show that while some methods might lose in the comparative study, their prototypes are still interpretable, even if a bit less so than the more successful methods. These results also show that 95.18% of the prototypes created by TesNet and 89.56% of the ones by ProtoTree are seen as interpretable. These results match our own observations, where it seems that TesNet and ProtoTree offer consistently good prototypes in all of the datasets. Like before, Table 4 shows the results of the same experiments but on a method-by-method basis. Each row shows the percentage of times a method (shown on the first column) was selected either solely or as part of the "Both" option when paired with another method shown on the top row. Note that this time, the sum of mirrored cells does not necessarily add up to 100% due to the existence of "Both" and "None" options. The results show a fine-grained consistency in prototype interpretability rates across all methods. This paper is available on arxiv under CC 4.0 license. Authors: (1) Omid Davoodi, Carleton University, School of Computer Science; (2) Shayan Mohammadizadehsamakosh, Sharif University of Technology, Department of Computer Engineering; (3) Majid Komeili, Carleton University, School of Computer Science. Authors: Authors: (1) Omid Davoodi, Carleton University, School of Computer Science; (2) Shayan Mohammadizadehsamakosh, Sharif University of Technology, Department of Computer Engineering; (3) Majid Komeili, Carleton University, School of Computer Science. Table of Links Abstract and Intro Abstract and Intro Background Information Background Information Methodology Methodology Prototype Interpretability Prototype Interpretability Prototype-query Similarity Prototype-query Similarity Interpretability of the Decision-Making Process Interpretability of the Decision-Making Process The Effects of Low Prototype Counts The Effects of Low Prototype Counts Discussions Discussions Prototype Interpretability Our first set of experiments had the goal of finding out the interpretability of the prototypes created by each method. Many part prototype methods claim that their prototypes are intrinsically interpretable. In that case, humans should be able to understand the concept that is encoded within those prototypes. If a prototype represents a large area of an image without any coherent concept understandable to humans, it is not an interpretable prototype. In the first experiment, each person was shown a pair of prototypes (concept images in the case of ACE) on the same dataset coming from different methods. The annotators had to choose the prototype they found more interpretable out of the two. Every person had to complete 10 such pairs each time. Out of those 10, 2 were validation pairs handpicked by us to ensure answer quality. An example of such a pair and the instructions given to each person are show in Figure 4. Only people who had finished a qualification task consisting of 7 such pairs with perfect scores were allowed into the main study. Moreover, each 10-pair set of query samples was given to three different people to gauge overall agreement and find individuals who might have chosen randomly. Additionally, this experiment was repeated 3 times and the results were aggregated. This experiment was a purely comparative study between different methods. This ensured that people had a frame of reference when deciding which method worked better, and this is a common practice in many studies of subjective nature. But they are not without problems. One of the more important problems with this experiment was that it could not answer whether the prototypes from these methods were interpretable or not. It could only answer whether the prototypes of a particular method were more interpretable on average than the ones from another. To remedy this, we conducted another set of experiments. In the new absolute experiments the annotators were given the option to also choose "Both" and "None" if they thought that both or none of the images in the pair were interpretable. The reason we did not simply show them a single image asking for interpretability was to keep the layout of the experiments very similar. This not only simplified the logistics of creating query samples for the new experiment, but also gave us the opportunity to see whether the rankings of the methods seen in the comparative experiments remained the same when the new options were added. The assumption was that if a very similar experiment with those options added resulted in a drastic change in the rankings, it would reveal problems in our methodology that were hidden beforehand. Instructions given to each person were as follows: • For each pair of images, pick the one with the best highlighted patch. A good patch contains relevant and identifying regions/parts of the subject, and only such regions/parts. • If both images have good or bad patches, select "Both" or "None" respectively. • Imagine that someone points to the marked region and says: "This is an image of a bird/car/lion/etc. because of this part“. How much would you agree with that statement? It is important to note that new image pairs were created for this experiment. This was again to see if a new set of data changes the rankings considerably, revealing any potential hidden problems with the statistical significance of the results. A new qualification task and a validation set were also created. The rest of the setup was almost identical to the comparative experiment. Each time, the annotator was given 10 prototype image pairs and told to pick the best option for each pair, with two out of those 10 pairs coming from the validation set. The experiment was repeated 4 times and the results were aggregated. In total, 1104 pairs were compared and annotated by humans for the comparative study. Table 1 shows the percentage of times each method was chosen against others. As can be seen, prototypes selected by TesNet were chosen 73.26% of the time when compared to prototypes selected by other methods. In the second place was ProtoTree with 63.64% preference. It is also interesting to note the relative weakness of SPARROW and Deformable ProtoPNet. Table 2 shows the results of the same experiments but on a method-by-method basis. Each element of the table shows the results of the pairwise comparison between the two methods, with the method shown in the first column being the one selected out of the two. For example, the intersection between the row "ACE" and the column "ProtoPool" shows the percentage of times (42.86%) ACE was chosen against ProtoPool in the experiments. While there is more variation when looked at with this much detail, the general trends hold true. For the absolute experiments, 1304 pairs were compared and annotated by humans in total. Table 3 shows the percentage of times each method’s prototypes were chosen in general. As the options for "Both" and "None" also existed in this experiment, the results here show the percentage of prototypes from each method that the annotators found acceptable. While the overall rankings seem to have mostly remained the same, the higher percentages overall show that while some methods might lose in the comparative study, their prototypes are still interpretable, even if a bit less so than the more successful methods. These results also show that 95.18% of the prototypes created by TesNet and 89.56% of the ones by ProtoTree are seen as interpretable. These results match our own observations, where it seems that TesNet and ProtoTree offer consistently good prototypes in all of the datasets. Like before, Table 4 shows the results of the same experiments but on a method-by-method basis. Each row shows the percentage of times a method (shown on the first column) was selected either solely or as part of the "Both" option when paired with another method shown on the top row. Note that this time, the sum of mirrored cells does not necessarily add up to 100% due to the existence of "Both" and "None" options. The results show a fine-grained consistency in prototype interpretability rates across all methods. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Evaluating Prototype Interpretability

Authors:
(1) Omid Davoodi, Carleton University, School of Computer Science;
(2) Shayan Mohammadizadehsamakosh, Sharif University of Technology, Department of Computer Engineering;
(3) Majid Komeili, Carleton University, School of Computer Science. Table of Links Abstract and Intro Background Information Methodology Prototype Interpretability Prototype-query Similarity Interpretability of the Decision-Making Process The Effects of Low Prototype Counts Discussions Methodology The interpretability of a prototype-based machine learning method as a whole is too broad to be effectively evaluated. As a result, our experiments were designed to measure the human opinion on the individual properties required for an interpretable method. This focus on single aspects allowed us to gain fine-grained insight into how much human users understand and agree with the explanations of different methods. Human annotators were recruited via Amazon Mechanical Turk. To ensure that workers have a good understanding of the task, they were required to first complete a qualification test. Furthermore, to ensure the quality of the responses and reduce noise, we inserted a few validation samples into our tasks and excluded workers who did not pass them. We trained seven methods on three different datasets. These methods include six part-prototype based classification methods (ProtoPNet[9], Deformable ProtoPNet[18], SPARROW[17], ProtoTree[19], TesNet[20], and ProtoPool[21]) and one unsupervised method (ACE22). The unsupervised method (ACE) is normally used to extract concepts from a dataset. Each concept is a part of a training sample and therefore is analogous to the concept of prototypes in part-prototype-based methods. As a result, we decided to include it in our experiments as a method representing unsupervised approaches to this problem and also as a baseline. The datasets on which we trained were CUB-200[23], Stanford Cars[24], and a subset of the Imagenet dataset[25] consisting of 7 classes of different felines and canines. The details for our Imagenet subset are included in the appendix. All datasets were augmented in the same process described in the work by Chen et al[9]. We also used the public code repositories from the authors of the papers of those methods in all cases except SPARROW. In the case of SPARROW, the authors kindly agreed to give us access to their non-public codebase. We used the same hyperparameters provided by these codebases, only changing those values when needed for different datasets. In particular, we used 10 prototypes for each class when training on datasets where the original codebase did not contain hyperparameters. In the special case of ProtoPool, we used 12 prototypes in total for the ImageNet dataset. We trained all classification methods until they reached at least 70% top-1 classification accuracy on the dataset. For Imagenet, our criterion was higher at 90% due to the lower number of classes and the fact that these backbone networks were originally trained on ImageNet in the first place. All methods used ResNet-18 as their backbone network. Something to note is that ProtoPool failed to reach the 70% accuracy level for the Cars dataset, instead only reaching a Top-1 accuracy of 49.53%. The next step was to gather a suite of query samples from the dataset and generate the explanations given by the methods for classifying those samples. For CUB and Cars, we used the first image found in the directory of each separate class. For Imagenet, we used the first 15 images in order to get enough samples for later use. One thing we noticed at this stage was that the usual heatmaps provided by many of these methods to denote the location of a prototype or its activation on a query image tended to occlude a large portion of the image itself. Some, like ProtoPNet, tended to draw a rectangle to denote the general boundary of the prototype or its activation. We found this approach to be quite misleading because there were cases where the rectangles could cover more area than the actual activation site. To address this, we opted for a more fine-grained approach where we created boundaries around the area of the activation heatmap with activation values above 70%. Figure 3 illustrates examples of complex activation sites, failures of the original ProtoPNet region selection approach, and our approach to address the problem. This paper is available on arxiv under CC 4.0 license. Authors: (1) Omid Davoodi, Carleton University, School of Computer Science; (2) Shayan Mohammadizadehsamakosh, Sharif University of Technology, Department of Computer Engineering; (3) Majid Komeili, Carleton University, School of Computer Science. Authors: Authors: (1) Omid Davoodi, Carleton University, School of Computer Science; (2) Shayan Mohammadizadehsamakosh, Sharif University of Technology, Department of Computer Engineering; (3) Majid Komeili, Carleton University, School of Computer Science. Table of Links Abstract and Intro Abstract and Intro Background Information Background Information Methodology Methodology Prototype Interpretability Prototype Interpretability Prototype-query Similarity Prototype-query Similarity Interpretability of the Decision-Making Process Interpretability of the Decision-Making Process The Effects of Low Prototype Counts The Effects of Low Prototype Counts Discussions Discussions Methodology The interpretability of a prototype-based machine learning method as a whole is too broad to be effectively evaluated. As a result, our experiments were designed to measure the human opinion on the individual properties required for an interpretable method. This focus on single aspects allowed us to gain fine-grained insight into how much human users understand and agree with the explanations of different methods. Human annotators were recruited via Amazon Mechanical Turk. To ensure that workers have a good understanding of the task, they were required to first complete a qualification test. Furthermore, to ensure the quality of the responses and reduce noise, we inserted a few validation samples into our tasks and excluded workers who did not pass them. We trained seven methods on three different datasets. These methods include six part-prototype based classification methods (ProtoPNet[9], Deformable ProtoPNet[18], SPARROW[17], ProtoTree[19], TesNet[20], and ProtoPool[21]) and one unsupervised method (ACE22). The unsupervised method (ACE) is normally used to extract concepts from a dataset. Each concept is a part of a training sample and therefore is analogous to the concept of prototypes in part-prototype-based methods. As a result, we decided to include it in our experiments as a method representing unsupervised approaches to this problem and also as a baseline. The datasets on which we trained were CUB-200[23], Stanford Cars[24], and a subset of the Imagenet dataset[25] consisting of 7 classes of different felines and canines. The details for our Imagenet subset are included in the appendix. All datasets were augmented in the same process described in the work by Chen et al[9]. We also used the public code repositories from the authors of the papers of those methods in all cases except SPARROW. In the case of SPARROW, the authors kindly agreed to give us access to their non-public codebase. We used the same hyperparameters provided by these codebases, only changing those values when needed for different datasets. In particular, we used 10 prototypes for each class when training on datasets where the original codebase did not contain hyperparameters. In the special case of ProtoPool, we used 12 prototypes in total for the ImageNet dataset. We trained all classification methods until they reached at least 70% top-1 classification accuracy on the dataset. For Imagenet, our criterion was higher at 90% due to the lower number of classes and the fact that these backbone networks were originally trained on ImageNet in the first place. All methods used ResNet-18 as their backbone network. Something to note is that ProtoPool failed to reach the 70% accuracy level for the Cars dataset, instead only reaching a Top-1 accuracy of 49.53%. The next step was to gather a suite of query samples from the dataset and generate the explanations given by the methods for classifying those samples. For CUB and Cars, we used the first image found in the directory of each separate class. For Imagenet, we used the first 15 images in order to get enough samples for later use. One thing we noticed at this stage was that the usual heatmaps provided by many of these methods to denote the location of a prototype or its activation on a query image tended to occlude a large portion of the image itself. Some, like ProtoPNet, tended to draw a rectangle to denote the general boundary of the prototype or its activation. We found this approach to be quite misleading because there were cases where the rectangles could cover more area than the actual activation site. To address this, we opted for a more fine-grained approach where we created boundaries around the area of the activation heatmap with activation values above 70%. Figure 3 illustrates examples of complex activation sites, failures of the original ProtoPNet region selection approach, and our approach to address the problem. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Stories, News & Mentions about SIMULAVR

Simulavr’s Stories on HackerNoon