Locale is a Control Tower for Ops Observability & Actionability which empowers operations and business teams to leverage their data and set up a simple monitoring and alerting system of record.

LocaleAI

localeai

If you are a two-degree marketplace like Uber, you cater to millions of users requesting a ride through your driver partners accepting and fulfilling those requests. For a three-degree marketplace like Swiggy, there is another static component added (like restaurants or stores), where delivery partners pick up the orders. Borrowing from the quote, “Everything happens somewhere” — all of these events and actions described take place at a specific location! Often, companies end up not leveraging the in their data and running their analyses at the city level. But, cities are too large, geographically diverse and the parameters vary way too much! lat/long component Area level polygons are much more practical but still broad. They don’t have uniform shapes or sizes and are subject to changes very frequently. Even the zones or clusters drawn by the operations teams’ local knowledge require updating and have arbitrary edges. Making sense of your spatial data and deriving precise insights requires these analyses to become more granular and uniform. brings that fine granularity to the table. It works brilliantly in bucketing all your lat-longs into “cells”. These cells can also be clustered to represent a particular neighborhood or area and can be aggregated at different levels. Grid system Hence, this system becomes critical to crunch large spatial data sets to match the supply & demand fragmented across the city. What is meant by grids? In Spatial Data Science, we use grids that are regular polygons repeating over a surface, edge to edge to cover any space without overlaps and gaps — a phenomenon called Each cell can be assigned a unique id for spatial indexing (aggregating the points inside that cell). tessellation . A wide variety of grids of different shapes have been proposed including squares, rectangles, triangles, hexagons or diamonds. The covers the entire surface of the earth. global grid system If you are a hyperlocal, on-demand company, grids as small as can be very useful to run models that work in context to location and in include surge pricing in high demand areas, promotions in low demand areas & distribution models of delivery folks on the ground. 0.5 sq. km real-time. Examples What are the different grid types? There are just three types of grids or that can tessellate: squares, equilateral triangles & hexagons. 1. Square grids: The most common application of square grids occurs in raster datasets and geohashes. For the scope of this piece, we will focus on geohashes. is a hierarchical data structure to transform a 2D spatial point (lat & long) into a short string of alphabets and numbers. They divide the world into a grid of 32 cells with 4 rows and 8 columns. Geohash You can keep splitting up each cell into a grid of 32 cells. Thus, the longer the string of the geohash, the higher the accuracy! You can also easily identify if geohashes are close together if they can have a common prefix. So, the longer the common prefix, the closer they are. For example, the coordinate pair near the tip of the peninsula of Jutland, Denmark produces a slightly shorter hash of (57.64911, 10.40744) u4pruydqqvj. [1] 2. Triangular grids: Triangular grids are not very commonly used. Besides, their unfamiliarity, one of the reasons for that is they have a large perimeter and a small area which means it’s harder to piece them together on the map. Another reason is that each triangle is connected to only three adjacent triangles, which limits the number of options to move and to make connections. (Check the image below) Moreover, while for hexagons and squares, there are always two faces parallel to each other, for triangles, there are two directions in which lines are parallel centered from the axis of movement. Thus, in a way, there are not completely symmetrical. [2] 3. Hexagonal grids: Besides looking appealing, hexagons are more symmetric than geohashes. They are very close to circles in terms of shape to provide a more accurate sampling. [3] As a result, this system has been increasingly adopted by companies like Uber. A hex grid and triangular grids are a dual of each other — putting a dot in the center of each hexagon & connecting them to all the adjacent ones, you get a triangular grid and vice versa! Fun fact: [4] Moving on the vertices of a hexagonal grid is equivalent to playing in the spaces on a triangular grid. Square grids, on the other hand, are a dual of themselves. Why hexagons? At , one question that we get a lot from our customers is, “why are we using hexbins and not geohashes?” Locale Well, the choice depends on your exact use case and you would have to make some tradeoffs, no matter what you use. So, let’s take some parameters and deep dive. This diagram shows the distance of the center of triangles, square & hexagon to its neighbors. Distance from Nearest Cells: A triangle has three kinds of distance (through the edge, vertex and across the center of the edge), a square has two (across the edge & the diagonal) and hexagon only has one — another reason why triangles are not really favored. This property of hexagons makes it very easy to perform analysis and is preferred when your analysis includes aspects of connectivity or movement. [4] All the neighbors in a hexagon form a ring around it with equal radius. The function provides grid cells within distance “k” of an origin index. In the diagram below, here is the of the shaded hexagon and square. kRing 1st kRing Fitting on Curved Surfaces: Hexagons are the densest way to pack circles in tessellation and reduce . (Circles have the least perimeter to area ratio but can’t form a continuous grid). edge effects The more similar a polygon is to a circle, the closer the points near the border area are to the center. Thus, any point inside a hexagon is closer to its center as compared to an equal-area square or triangle. Now, when large areas come into play and the curvature of the earth is important to consider, hexagons are therefore better suited to fit the curvature and suffer less distortion. [5] Explicit Patterns in Data: Hexagons allow any in the data to show easily and explicitly because they break up lines. curvature of the patterns For linear figures like squares and rectangles, this becomes tricky. These shapes draw our attention to the straight, unbroken and parallel lines which hinder the patterns present in the data. Refer to the diagram below. [6] Why geohashes? That brings me to my next question, “When would someone use fishnets or square grids?” Aggregation/ Division of Cells: Different kinds of models require different granularity and that’s where aggregation and division become important. If you need to increase the spatial resolution of a square grid, you just need to divide it into 4. Similarly, to aggregate, you need to combine four grids into one. For hexagons, aggregations and divisions are not uniform at different scales as shown in the image below. The finer cells are only approximately contained in the parent cell. [7] Squares are preferred over hexagons for hierarchical analysis. Combining square grids is fairly simple. No spatial operations are necessary for combing multiple grids built on the same template — you can use matrix algebra. Very Intuitive & Familiar: We also think in terms of “ ”. Up, down, left, right are simple to understand. We have built cities and civilizations on squares and rectangles. Since our primary coordinate system is squared, people find it difficult to work on other systems. squares They are also sometimes used for connectivity analysis as they have eight neighbors (including diagonals). Some Real-Life Examples Hexagons are found widely in nature. For example, honeycombs, graphite, benzene, silicene etc. Chinese Checkers is played on a hex grid and several variants of chess have also been invented for a hex board. The hex grid is a distinguishing feature of the games from many wargame publishers, and a few other games (such as The Settlers of Catan)! [8] About Locale: At Locale, we are building an “ operational ” analytics platform using location data for supply and operations teams in on-demand companies. If you want to delve further, check our website out or get in touch with me on LinkedIn or Twitter . Originally posted here . Read Similar: Geospatial Clustering: Types and Use Cases [ ] Link A Guide to Kickstart with Geospatial Analysis [ ] Link References: If you wish to read more, check out the links to delve further: [1] https://en.wikipedia.org/wiki/Geohash [2] https://boardgames.stackexchange.com/questions/633/why-are-there-fewer-board-games-with-a-triangular-grid [3] https://www.redblobgames.com/grids/hexagons/ [4] http://www-cs-students.stanford.edu/~amitp/game-programming/grids/ [5] https://pro.arcgis.com/en/pro-app/tool-reference/spatial-statistics/h-whyhexagons.htm [4] https://www.sciencedirect.com/science/article/pii/S0304380007001949 [5] http://strimas.com/spatial/hexagonal-grids/ [6] https://gis.stackexchange.com/questions/82362/what-are-the-benefits-of-hexagonal-sampling-polygons [7] https://en.wikipedia.org/wiki/Hex_map [8] https://en.wikipedia.org/wiki/Hexagonal_tiling%5C Previously published at https://blog.locale.ai/spatial-modelling-tidbits-honeycomb-or-fishnets/

Why Do We Use Hexagons And Not Sqaures to Aggregate Location Data

Geo is practically everywhere today! With the prevalence of GPS all around us & the rise of smartphones, location data is being collected in abundance. Legend almost all of the data (~80%) that companies store has a location component. says Our lives are now filled with products and services available at the tap of a phone, getting things delivered in minutes to our doorstep! Today, it is hard to find an app which doesn’t ask for your location permissions. There is a famous in the GIS (Geographic Information Systems) world which says When you place an order from any food delivery app, every event starting from placing of the order — to delivery executives (DE) getting assigned to — the DE picking it up from the restaurant — to him delivering happens at a certain lat-long! quote “everything happens somewhere.” Which brings me to my next point: Why are we obsessed with “location” as a dimension? Firstly, there is no “mobility” without dynamic location. In other words, location is a fundamental aspect whenever there is movement of assets (people, vehicles, cargo, parcel) on the ground. After all, in cities and towns where things change at every sq km in a city, it’s important to add the “context” of those areas. For the on-demand economy that we live in, analyzing location data across all your dimensions (users, stores, partners) becomes in real-time. This is because you have to match your supply with demand, do location-based pricing, promotions or faster and more accurate deliveries at scale! critical And finally, the most successful for consumers are based on sequential activities at a very granular level: Where are you? Where do you need to go? What do you want to do when you get there? This becomes important when you need to acquire customers by targeting ads and retaining and engaging them by their behavior. location-based experiences studying Location is not only about a point on a map. It is about a line. It is about movement. But all this time, the location component has been particularly in business decisions. This is because conventional analytics tools are fantastic for the treasure trove of statistical data that businesses have but somewhere, they fall short of . neglected location-based decisioning You might be wondering: “But, why?” Before we deep-dive into that, let’s understand how geospatial data is different from statistical data. Location is exciting! If played right, it can drive significant revenue improvements (case in point: Uber & Airbnb). Its high usage and real-time nature make it really valuable and sticky. What sets geospatial data apart from statistical data? Just like text, sound, image are different kinds of data, Latitude-Longitude is a different kind of data that can add immense depth, meaning and insights to statistical data in a space-time context. Statistical data comes in a tabular format and usually, is comprised of two elements: values across time. (also called geographic data or location data) is often formatted as points (latitude-longitude coordinates), polygons or polylines. Geospatial data It incorporates the third dimension on top: values across time and dynamic location — which requires a completely different approach and treatment. Computing metrics for static assets involves plotting points on a map and calculating metrics for that. Movement analytics concerns itself with how do we visualize, analyze and optimize how things move on the ground! Two special properties of geospatial data are autocorrelation and non-stationarity, which make it difficult to meet the assumptions and requirements of traditional (nonspatial) statistical methods, like OLS regression. means how similar (correlated) are my measures between nearby observations. It falls in-line with the first law of geography as well- Spatial Autocorrelation “ everything is related to everything else, but near things are more related than distant things ”. is a condition in which a simple “global” model cannot explain spatial relationships between the variables. The nature of the model alters over space and behaves differently in different parts of the study area. Spatial nonstationarity Let’s answer our pending question now: Why are current statistical tools not apt for spatial analysis? Today, the current way of dealing with location data inside companies is broken. Due to the dearth of any location analytics products out there, most of the businesses have no choice but to rely on traditional BI and analytics tools. It’s not puzzling that this is a highly ineffective strategy because these tools are not really meant for geo-analysis. 1. Formats Most companies collect location log data in the form of . The way to analyze the patterns they encompass is to have an infrastructure that can ingest these pings in real-time — something that platforms like Periscope or Tableau don’t cater to. “ pings ” Moreover, location data is present across disparate databases, in different structures. Hence, slicing and dicing variables across tables in real-time comes even more complex. 2. Visualizations All of us know that a statistical dashboard contains bars and charts which sprout from carrying out operations (sum, count, divide, average, etc) on variables. While getting live trend updates through spikes and dips on graphs might be helpful, these charts are work better on aggregated historical data. Adding or dividing lat-longs and creating bars and charts on them is pretty futile. To make sense of these lat-longs, you need to have a map by your side to understand their spatial distribution! Another important aspect that governs all the properties of a geospatial dashboard is a . Not having this layering mechanism which you can use to display multiple types of data points on the map, sort of misses the point. For instance, layer layers help in viewing how my orders (first layer) and partners locations (second layer) are distributed across my area clusters (third layer). Maps are also more insightful to draw inferences than bars and charts if there is movement of components on the ground involved. Hence, real-time geographic analysis when everything is dynamic becomes fundamental. 3. Aggregation Traditional BI tools like and offer the capability to plot points on a map. However, just plotting points on a map is not adequate as billion dots in space is not very intuitive. Moreover, PowerBI Geckoboard Location intelligence is so much more than tracking and plotting points on a map! Strategies like clustering, heat mapping, aggregation, indexing, etc. come in handy to absorb a large number of points. Some tools like and also allow the creation of heat maps — a fantastic way to depict the patterns of metrics. The disadvantage of heatmaps is that they are only a visual representation, thus restricting you to do any sort of analysis on it. Tableau Periscope To know more about why heatmaps don’t work, check out! this A more efficient way to do aggregation is indexing them on or . Once you analyze the pattern of your metrics (such as demand and supply) across grids, you can use the cells as a single unit in your models as well. Indexing also helps to go very granular in your analysis. hexagonal grids geohashes 4. Data Preparation Using and , you can clean your statistical data by taking care of the blanks, spaces, data formats, NAs etc whereas cleaning of GPS data involves snapping it back to the nearest road or correcting for . (You must have observed often while using Google Maps, GPS goes off very far randomly.) It is safe to say that GPS as a technology still has miles to go! Cleaning: Periscope Thoughtspot spatial outliers Platforms like or allow you to merge two tables using an inner join, left join or a right join on the basis of a common identifier. It’s quite difficult to do spatial merges using these platform if, for instance, you have data across three dimensions: users, delivery partners, and stores (sometimes they come in different formats). Merging: Tableau Prep Metabase A involves inserting columns from one feature table of a layer to another in the spatial perspective. spatial join For example. merging a Kormangala area polygon with all the ride pick up points inside it. Enriching of any data implies adding new layers of information and merging it with third-party or external data sources. In the GIS world, we enrich spatial data for a better context of areas in which the points are present. This means adding the environmental layer of roads, traffic, weather, points of interest, demographics, buildings, etc. Enriching: The internal solutions not working out? Some companies, of course, realize this and hack around open-source tools (like or ). But these open source tools come with their own list of constraints and limitations. Kepler.gl QGIS However, the issue doesn’t get resolved here because geospatial data itself comes with a bucket full of challenges. Performing geospatial queries on streaming data become very compute-intensive and legacy technologies (like ArcGIS) provide very little support. The complexity increases with visualizing large geospatial datasets with any sort of interactivity at scale. Sometimes developers also build their own internal tools, but most of the times they are not well suited for all different audiences inside the company. Since the tools are not built in a scalable way, maintaining these suck up a lot of developer bandwidth often! A lot of times there is even a repetition of effort and the wheel keeps getting re-invented over and over again. As Jeff Lawson from Twilio — “It is easier than ever to build software but harder than ever to operate it”. said Why are we building Locale.ai? It all started with a personal problem. As data scientists working with geospatial data, the existing analytics products were futile in our daily workflows. Hence, we had to build our own tools and libraries for our everyday workflows. We then realized data scientists around the globe face similar problems when it comes to location data. As a result, businesses are struggling to attain operational efficiency! At , we plan on solving these problems once and for all. We want to make all of these processes less painful and building a world where it is very easy to get all your geospatial answers in minutes! That’s why we are going back to the drawing board and handcrafting this experience completely from scratch. Locale So, the next time you want to order medicines in case of an emergency, you won’t hopefully read on the screen, “ ” Next time the delivery guys won’t have to stand idle in the scorching heat, cold or rains waiting for the orders to come. Delivery guys are not available. Please check again later. They can be incentivized to move to high demand areas and can earn more money. The push notifications that you get won’t be spam — they will be shot to you at the right place and right time. Read Similar: Bridging Supply-Demand Gaps in Last-Mile Delivery Companies Geospatially [ ] Link Carto vs Kepler vs Locale: Which product to use for geospatial analytics? [ ] Link

Why Do We Need a New Product for Geospatial Analytics for Mobility

How to Run Your Own Local LLM: Updated for 2024 - Version 2

Welcome to this issue of the Activation Function Newsletter. For those of you who are new, it’s pretty simple - Every other week, I introduce you to a new and exciting open-source backend technology (that you’ve probably only kind of heard about ) and explain it to you in 5 minutes or less so you can make better technical decisions moving forward. In this issue, we’ll explore K8sGPT, an open-source tool that combines expert knowledge and Generative AI to help you identify, understand, and troubleshoot your Kubernetes cluster in plain English. Before you dismiss it as “just another ChatGPT wrapper,” let me assure you that it’s not. It’s much more than that. I think it’s a thoughtful implementation of GenAI that has the potential to bring tangible benefits and make your life a bit easier. Alright, Let’s dig in! Quick Facts: K8sGPT was launched at KubeCon Amsterdam 2023. K8sGPT is open-sourced under an Apache-2.0 license. K8sGPT was accepted to CNCF on December 19, 2023, at the Sandbox maturity level. K8sGPT now has over 4K GitHub stars and 50 contributors. Introduction For years, we’ve used ML for APM and Observability to identify patterns, trends, and anomalies (e.g., behavioral changes in applications or infrastructure), anticipate performance issues, generate early warnings, etc. But so far, it hasn’t been great at actually helping us understand and fix issues. The story goes something like this. You get an alert from your ML-enabled monitoring tool and then spend a lot of time looking through and interpreting complex error messages, trying to communicate those to your team, and iterating through remediation steps until you fix the problem. Enters K8sGPT. Its primary function is to help you quickly triage and remedy issues by giving hints and shortcuts in plain English. It does this by leveraging expert knowledge, Natural Language Processing (NLP), and Generative AI. How K8sGPT Works K8sGPT uses contain codified knowledge, which is essentially a series of rules/checks that an experienced Site Reliability Engineer (SRE) might use to diagnose Kubernetes issues such as pod crashes, service failures, and ingress misconfigurations, etc. Analyzers that It was a good decision to avoid using GenAI at this stage to prevent speculative guesses, hallucinations, and, ultimately, a lot of noise. Thought: Once K8sGpT is connected to your Kubernetes cluster through the API Server, the will start scanning your cluster and looking at logs, error messages, configurations, and pod status to identify potential issues. Analyzers : K8sGPT has a set of , but you will be able to write your own analyzers. Technical Note analyzers that are built in It then tries to correlate issues and run additional checks if necessary to add more context and figure out the root cause before packaging the data and sending it to the AI backend provider to get a simple and descriptive text of the issue and potential remediation steps. As you can see, the use of Gen AI is targeted and practical. Of course, the AI can hallucinate and provide incorrect remediation steps, but that’s a minor inconvenience you can spot and ignore easily. Technical Note: Getting Started with K8sGPT The easiest way to get started is using a by running a brew command on Linux/Mac or downloading the Windows Bianary if you’re on Windows. CLI installation Once installed, you’ll need to , and within a few minutes, you’ll be ready to run your first scan. authenticate your chosen AI backend K8sGPT is designed to work with various AI providers, including , , , , , , and . Technical Note: OpenAI Cohere Amazon Bedrock Amazon SageMaker Azure OpenAI Google Gemini LocalAI You can find how to configure different backends . here Once everything is configured, you can start using K8sGPT directly from your CLI using the following set of commands:‍ Usage:
  k8sgpt [command]

Available Commands:
  analyze     This command will find problems within your Kubernetes cluster
  auth        Authenticate with your chosen backend
  cache       For working with the cache the results of an analysis
  completion  Generate the autocompletion script for the specified shell
  filters     Manage filters for analyzing Kubernetes resources
  generate    Generate Key for your chosen backend (opens browser)
  help        Help about any command
  integration Integrate another tool into K8sGPT
  serve       Runs k8sgpt as a server
  version     Print the version number of k8sgpt

Flags:
      --config string        Default config file (default is $HOME/.k8sgpt.yaml)
  -h, --help                 help for k8sgpt
      --kubeconfig string    Path to a kubeconfig. Only required if out-of-cluster.
      --kubecontext string   Kubernetes context to use. Only required if out-of-cluster.

Use "k8sgpt [command] --help" for more information about a command. You’ll soon be able to debug interactively by asking K8sGPT questions about any issue directly in your CLI. As for writing, this feature isn’t live. [Technical Note] K8sGPT Operator K8sGPT can also be installed as an Operator within your Kubernetes Cluster. By using the K8sGPT operator, you’ll be able to scan your cluster for errors continuously. The works by leveraging Kubernetes Custom Resources and producing reports that it stores in your cluster as YAML manifests. K8sGPT Operator You can cache your results locally or like AWS S3. [Technical Note] offload them to a remote location You can also easily customize the analysis and output of the operator to fit your needs and workflow. For example, you specify a certain namespace or resource and a specific output format like JSON. This makes K8sGPT easy to integrate into your CI/CD pipelines and other workflows. Integrations K8sGPT integrates with observability tools like Grafana and Prometheus, but the really cool thing is that it offers an that allows you to extend scanning and troubleshooting capabilities by writing your own plugins that connect it to other tools. integration feature The first one, contributed by the core team, is the plugging, which enables you to generate vulnerability scans as part of your analysis and leverage AI to help you understand and triage Common Vulnerabilities and Exposures (CVEs). Trivy Security & Privacy Alright! I’m sure this is the first or second thing that came to your mind when hearing that K8sGPT sends your data to an external AI-API. Well, the K8sGPT team thought of this and developed a nifty little feature called anonymize, which masks sensitive data like Kubernetes object names and labels before sending it to the AI backend for analysis. All you need to do is use the anonymize flag: k8sgpt analyze --explain --anonymize During the analysis, k8sGPT retrieves sensitive data and masks it before sending it to the AI backend. Once the solution is returned, the masked data is replaced with the actual Kubernetes object names and labels. Where to go from here? K8sGPT is still a nascent project and is quickly evolving thanks to an active community of contributors. As of writing, an exciting new feature on the horizon is the interactive debugging mode that lets you ask questions directly inside your terminal. Other things on the horizon include Kubernetes auto-remediation with K8sGPT, Karpenter, deep integration into AWS services, and new backends into services such as Hugging Face. If you want to dig deeper into K8sGPT, here are a few resources for you: Documentation & Repos - The main code repository k8sgpt - Documentation Website ( ) docs https://docs.k8sgpt.ai - Community-related information community Tutorials K8SGPT - Kubernetes SRE SUPERPOWERS Using AI with Kubernetes | K8sGPT Operator | DEMO Full Tutorial: K8sGPT -- SRE superpowers through AI Talks AI for Kubernetes with ChatGPT and k8sgpt Unleashing the Power of AI in Kubernetes through K8sGPT | Alex Jones Communities Slack: https://k8sgpt.slack.com Who to Follow Alex Jones (Creator of K8sGPT) on and Linkedin X K8sGPT on and LinkedIn X Thomas Schuetz (Maintainer) on and Linkedin X Matthis Holleville (Maintainer) on and Linkedin X It’s a wrap! I hope this gave you a good overview of K8sGPT and how Gen AI might make your life a bit easier when troubleshooting your Kubernetes clusters. If you want to take K8sGPT for a quick spin, I suggest you check out this . Sandbox tutorial Until next time! Also published . here

A Gentle Intro to K8sGPT: Your Kubernetes Debugging Sidekick

Introduction Hi, everyone, my name is Maksim Nechaev. I'm a Senior iOS developer at Snoonu, and also the founder of my own startup. Like many other developers, I've written a lot of apps and features on UIKit. All the global projects I've worked with have been on it. And no wonder because it is the most common set of UI elements if we talk about . Along with UIKit, we should immediately mention layout via constraint. Often, we used SnapKit to simplify this task and write more minimalistic code. iOS development But times were changing, and a new, trendy, and hip SwiftUI framework appeared on the horizon. At first, it was shunned. They looked at it with derision. Maybe it even made sense at first because SwiftUI had huge limitations and shortcomings. We couldn't do 50% of what we could do with UIKit. But years later, SwiftUI has become more powerful, so much so that many companies don't even look at UIKit when starting new projects. Why is that? It's because SwiftUI has a declarative layout style, uses convenient and fast reactive elements, and building any screen complexity takes from 30 minutes to two hours, while on UIKit, you would have spent at least a day. And what is time for business? We all know time is money. So, the first thing SwiftUI is already doing right now is saving money for businesses. Let's go deeper. What else is great about SwiftUI? The ease of creating animations. Yes, yes, if you imagine a cool app with 60+ frames per second that responds smoothly to taps, buttons shrink with animation, screens pop out of the right side of the screen, everything happens in beautiful and pleasing (for UX) animations, all of this is very quick and easy to do inside SwiftUI. You can do a lot of this in UIKit, too, to tell you the truth. But spend a lot more effort and time. I hope I'm doing well in convincing you that SwiftUI is ready for full production development and that you should be learning it right now. That's why, with my first article on this topic, I'd like to touch upon the SwiftUI View lifecycle. How a "view" is created, where it is stored and how it is updated, I would like to talk about it all in this article. Let's get started! SwiftUI View vs UIView Let's start with the fact that UIView is created once in most cases. After that, we simply update its data. A SwiftUI View can be redrawn and created dozens of times within a single screen. This is very well illustrated in the following image. You may notice that when we use UIViewRepresentable, a structure that helps us use UIView inside SwiftUI, we create it once. But further, to update it, we pull updateUIView and update it as many times as needed. SwiftUI View is created many times, while UIView is created once. Storing data in SwiftUI View Let's take a look at ways to store data inside a . After all, you're probably already thinking, how do you store data if the View is constantly being recreated and redrawn? SwiftUI View In total, we can divide it into . three parts . Let and var, the classic use of data. First . @State and @StateObject. Already more SwiftUI approach Second . @Enviroment and @EnviromentObject Third Let Absolutely classic usage; we can create some kind of constant that we pass externally and use inside our structure. struct CustomView: View {
    let text: String

    init(text: String) {
        self.text = text
    }

    var body: some View {
        Text(text)
    }
} The only thing to keep in mind here is that usually, when we create a constant, we assume that it will not change. The situation here is that when is changed and redrawn, the constant can be changed. That is, it can be "bound" to the objects being changed. CustomView text Var This is where it gets interesting. What if we want to change a variable inside a structure? We have always done it in a very simple way, if we talk about classes in UIKit. var text: String Let's imagine that we have made our text a variable. It seems that now we can change it. But no, Swift won't let us do that. Structures are more capricious and rigid than classes, and even more so in SwiftUI. There is only one way to try to change text, and that is to create a mutating function. But unfortunately, we will get an error here, too. It will look like this: "Cannot use mutating member on immutable value.” In general, the idea with an ordinary var in the structure is very doubtful, and you should think ten times why you should do it. After all, SwiftUI has other cool tools for working with values, which we will talk about next. State The situation with the State is fundamentally the opposite. It can and must be used inside structures if we want to work with the value and change it somehow. struct CustomView: View {
    @State private var text: String

    var body: some View {
        Text(text)
    }
    
    func changeText() {
        text = "Hello, World!"
    }
} That is, if we create a structure with some local variables to which we tie local logic, we should use , so that all internal Views that use this value will be updated. Everything will work correctly, but... there is always a "but". What if we want to set the value externally? @State For example, like this struct CustomView: View {
    @State private var text: String

    init(text: String) {
        self.text = text
    }

    var body: some View {
        Text(text)
    }
} Do you think it will work, and in what cases? To be fair, it will work, but only when we set the first value. When we set all subsequent values, only the first one will be displayed inside the . body Here, it is important to realize that does not live as long as View does. That is, when we create inside View, we can change it inside View as many times as we want. The view will be updated and redrawn, but the local value will be stored. But if at the same time we try to set new values for through the initializer, they will be simply ignored. @State @State @State What is the output here? should be used only for saving and updating local data, but not those passed externally to the structure. @State StateObject Now, let's talk about using StateObject. In general, we use to set some kind of ViewModel for our screen. That is, we pair it with a class that obeys the ObservableObject protocol. It works very similarly to , with one exception: it is created after the first initialization of the structure. That is, while could somehow be passed to to put the first value, won't work. It will be deleted from memory after the initializer is called and will be created again. @StateObject @State @State init @StateObject Environment Let's move on to the consideration of . This property modifier is used to access data that is passed down the view hierarchy. Unlike , which is used to control the state of an object within a single view, allows you to access data that is shared by the entire application or a specific part of it. @Environment @StateObject @Environment Let's say you have some value that needs to be available in many places in your interface, such as custom settings or a design theme. Instead of passing this value through all levels of the hierarchy, you can use for easier access. @Environment When you mark a property in your SwiftUI view as ,SwiftUI automatically looks up the value of that property in the nearest matching context and provides it to you. If the value changes, SwiftUI also automatically updates the views that use that value. This makes a very powerful tool for managing data in SwiftUI. @Environment @Environment Example of use: struct ContentView: View {
    @Environment(\.locale) var locale: Locale

    var body: some View {
        Text("Current localization: \(locale.identifier)")
    }
} EnvironmentObject Now, let's break down . This property modifier is used to inject dependencies into SwiftUI views. Unlike , which is designed to access system settings or values defined at the application level, is used to pass and use user data between views. @EnvironmentObject @Environment @EnvironmentObject When you use , you expect the object to be provided by the parent views. This allows you to create more modular and easily scalable applications because your views do not depend on specific implementations of their dependencies. Instead, they dynamically retrieve the necessary objects from their environment. @EnvironmentObject Example usage of : @EnvironmentObject class UserSettings: ObservableObject {
    @Published var score = 0
}

struct ContentView: View {
    @EnvironmentObject var settings: UserSettings

    var body: some View {
        Text("Score: \(settings.score)")
    }
} SwiftUI View Life Cycle Understanding the life cycle of a view in SwiftUI is key to creating efficient and well-performing applications. Let's explore this process in a detailed and interesting way. 1. Initialization ( ) Initialization Every SwiftUI View starts its life journey with initialization. This is when SwiftUI creates an instance of your view and sets up its initial state. It's important to realize that SwiftUI views are structures, meaning they are value types. When you create a view, you are actually creating a "snapshot" of it at this point in time. 2. State Update ( ) State Update After initialization, SwiftUI watches for changes in data that may affect your view. This data can include , , , , and . When one of these values changes, SwiftUI starts the process of updating the view. @State @Binding @ObservedObject @EnvironmentObject @Environment 3. Body Computation ( ) Body Computation In this step, SwiftUI calls the property of your view. is a computed property, and every time SwiftUI detects a change that affects the view, it recomputes . This process allows SwiftUI to determine which parts of the UI need to be updated. body body body 4. Rendering & Layout ( ) Rendering & Layout Once the body of the view has been computed, SwiftUI proceeds to the rendering and layout phase. In this stage, SwiftUI determines exactly how the interface elements should be arranged and rendered. This process involves calculating the dimensions, positions, and other layout aspects for each element on the screen. 5. View Activation ( ) View Activation This is a specific point in the lifecycle that relates to user interaction with the view. For example, it could be the activation of a text box or the appearance of a view on the screen. 6. Deinitialization ( ) Deinitialization The last step in the life cycle of a view is to deinitize it. When a view is removed from the SwiftUI view hierarchy, it is deinitialized. This is an important time to release resources or perform any cleanup. Important Points: : SwiftUI frequently recreates and reuses views. Therefore, it is important to avoid expensive operations in initializers and . Reuse body : SwiftUI is optimized to ensure that only the necessary parts of the interface are updated. This improves the performance and efficiency of the application. Efficiency Conclusions SwiftUI View is very different from the familiar UIView, but that's the point. SwiftUI is different, and that's a good thing. In this article, I tried to convey the essence of working with SwiftUI View as succinctly and clearly as possible because this is a key skill for a SwiftUI developer. You will come across it in every task, which means it's worth understanding it very deeply. You need to be conscious of what you are writing and creating. I wish everyone good luck on the path to becoming powerful developers. If you are reading this article, then you are already on the right path. Thanks for your time, bye everyone! Please respond to this article so I can write more useful material for you.

Understanding the SwiftUI View Lifecycle and Data Management

Authors:
(1) Luyuan Peng, Acoustic Research Laboratory, National University of Singapore;
(2) Hari Vishnu, Acoustic Research Laboratory, National University of Singapore;
(3) Mandar Chitre, Acoustic Research Laboratory, National University of Singapore;
(4) Yuen Min Too, Acoustic Research Laboratory, National University of Singapore;
(5) Bharath Kalyan, Acoustic Research Laboratory, National University of Singapore;
(6) Rajat Mishra, Acoustic Research Laboratory, National University of Singapore. Table of Links I Introduction II Method III Datasets IV Experiments, Acknowledgment, and References IV. EXPERIMENTS We test three model configurations on both the simulator and tank datasets. First, we implement a pose regression model based on GoogLeNet [7] as our baseline model. Secondly, we implement a model using ResNet-50 [9] to evaluate the improvement possible by use of a deeper network. ResNet allows us to have deeper networks with more free parameters, and tries to avoid the problem of overfitting by having residual connections between layers [9]. Thirdly, we implement a pose regression model using an LSTM as an intermediate layer. The results indicate that all three configurations can perform well in both simulated and tank datasets as shown in Table I. The errors are minimal and of comparable magnitude to the noise in the pose recorded by the camera sensors. Fig. 4 shows that the trajectories predicted by the models are very close to the actual trajectories in terms of both position and orientation. With the simulator dataset, which is free of noise, turbidity, light distortion, and other challenges that may be observed in real underwater settings, the ResNet-50 and LSTM-based model performed better than the baseline model. On the other hand, with the tank datasets with some distortions typical of underwater scenarios, ResNet-50 performed worse than the baseline model, whereas the LSTM model did better than baseline with dataset #1 (which mostly featured translation but almost no rotation). This suggests the ResNet-50 architecture may be slightly overfitting despite the regularization, so it is not able to provide better performance in the tank dataset. We also note that data augmentation significantly improves the model performance, so in cases where there is no significant rotation, the use of data from both cameras can be used to bolster performance. Overall, these methods are robust for application in real underwater environments, and show promise for use in open water settings, where they will be tested next. ACKNOWLEDGMENT This research project is supported by A*STAR under its RIE2020 Advanced Manufacturing and Engineering (AME) Industry Alignment Fund - Pre-Positioning (IAF-PP) Grant No. A20H8a0241. REFERENCES [1] J. Gonzalez-Garcıa, A. Gomez-Espinosa, E. Cuan-Urquizo, L. G. ´ Garcıa-Valdovinos, T. Salgado-Jimenez, and J. A. E. Cabello, “Au-tonomous Underwater Vehicles: Localization, Navigation, and Commu- nication for Collaborative Missions,” Applied Sciences, vol. 10, no. 4, p. 1256, Feb. 2020. [2] L. Peng and M. Chitre, ”Regressing Poses from Monocular Images in an Underwater Environment”, 2022 OCEANS - Chennai, in press. [3] A. Kendall, M. Grimes and R. Cipolla, ”PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization,” 2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp. 2938- 2946, doi: 10.1109/ICCV.2015.336. [4] S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, Nov. 1997, doi: 10.1162/neco.1997.9.8.1735. [5] F. Walch, C. Hazirbas, L. Leal-Taixe, T. Sattler, S. Hilsenbeck, and D. Cremers, “Image-based localization using lstms for structured feature correlation,” 2017 IEEE International Conference on Computer Vision (ICCV), 2017. [6] A. Kendall and R. Cipolla, ”Geometric Loss Functions for Camera Pose Regression with Deep Learning,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 6555-6564, doi: 10.1109/CVPR.2017.694. [7] C. Szegedy et al., ”Going deeper with convolutions,” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1-9, doi: 10.1109/CVPR.2015.7298594. [8] A. Chaudhary, R. Mishra, B. Kalyan, and M. Chitre, ”Development of an Underwater Simulator using Unity3D and Robot Operating Systems”, 2021 OCEANS - San Diego, in press. [9] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Luyuan Peng, Acoustic Research Laboratory, National University of Singapore; (2) Hari Vishnu, Acoustic Research Laboratory, National University of Singapore; (3) Mandar Chitre, Acoustic Research Laboratory, National University of Singapore; (4) Yuen Min Too, Acoustic Research Laboratory, National University of Singapore; (5) Bharath Kalyan, Acoustic Research Laboratory, National University of Singapore; (6) Rajat Mishra, Acoustic Research Laboratory, National University of Singapore. Authors: Authors: (1) Luyuan Peng, Acoustic Research Laboratory, National University of Singapore; (2) Hari Vishnu, Acoustic Research Laboratory, National University of Singapore; (3) Mandar Chitre, Acoustic Research Laboratory, National University of Singapore; (4) Yuen Min Too, Acoustic Research Laboratory, National University of Singapore; (5) Bharath Kalyan, Acoustic Research Laboratory, National University of Singapore; (6) Rajat Mishra, Acoustic Research Laboratory, National University of Singapore. Table of Links I Introduction I Introduction II Method II Method III Datasets III Datasets IV Experiments, Acknowledgment, and References IV Experiments, Acknowledgment, and References IV. EXPERIMENTS We test three model configurations on both the simulator and tank datasets. First, we implement a pose regression model based on GoogLeNet [7] as our baseline model. Secondly, we implement a model using ResNet-50 [9] to evaluate the improvement possible by use of a deeper network. ResNet allows us to have deeper networks with more free parameters, and tries to avoid the problem of overfitting by having residual connections between layers [9]. Thirdly, we implement a pose regression model using an LSTM as an intermediate layer. The results indicate that all three configurations can perform well in both simulated and tank datasets as shown in Table I. The errors are minimal and of comparable magnitude to the noise in the pose recorded by the camera sensors. Fig. 4 shows that the trajectories predicted by the models are very close to the actual trajectories in terms of both position and orientation. With the simulator dataset, which is free of noise, turbidity, light distortion, and other challenges that may be observed in real underwater settings, the ResNet-50 and LSTM-based model performed better than the baseline model. On the other hand, with the tank datasets with some distortions typical of underwater scenarios, ResNet-50 performed worse than the baseline model, whereas the LSTM model did better than baseline with dataset #1 (which mostly featured translation but almost no rotation). This suggests the ResNet-50 architecture may be slightly overfitting despite the regularization, so it is not able to provide better performance in the tank dataset. We also note that data augmentation significantly improves the model performance, so in cases where there is no significant rotation, the use of data from both cameras can be used to bolster performance. Overall, these methods are robust for application in real underwater environments, and show promise for use in open water settings, where they will be tested next. ACKNOWLEDGMENT This research project is supported by A*STAR under its RIE2020 Advanced Manufacturing and Engineering (AME) Industry Alignment Fund - Pre-Positioning (IAF-PP) Grant No. A20H8a0241. REFERENCES [1] J. Gonzalez-Garcıa, A. Gomez-Espinosa, E. Cuan-Urquizo, L. G. ´ Garcıa-Valdovinos, T. Salgado-Jimenez, and J. A. E. Cabello, “Au-tonomous Underwater Vehicles: Localization, Navigation, and Commu- nication for Collaborative Missions,” Applied Sciences, vol. 10, no. 4, p. 1256, Feb. 2020. [2] L. Peng and M. Chitre, ”Regressing Poses from Monocular Images in an Underwater Environment”, 2022 OCEANS - Chennai, in press. [3] A. Kendall, M. Grimes and R. Cipolla, ”PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization,” 2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp. 2938- 2946, doi: 10.1109/ICCV.2015.336. [4] S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, Nov. 1997, doi: 10.1162/neco.1997.9.8.1735. [5] F. Walch, C. Hazirbas, L. Leal-Taixe, T. Sattler, S. Hilsenbeck, and D. Cremers, “Image-based localization using lstms for structured feature correlation,” 2017 IEEE International Conference on Computer Vision (ICCV), 2017. [6] A. Kendall and R. Cipolla, ”Geometric Loss Functions for Camera Pose Regression with Deep Learning,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 6555-6564, doi: 10.1109/CVPR.2017.694. [7] C. Szegedy et al., ”Going deeper with convolutions,” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1-9, doi: 10.1109/CVPR.2015.7298594. [8] A. Chaudhary, R. Mishra, B. Kalyan, and M. Chitre, ”Development of an Underwater Simulator using Unity3D and Robot Operating Systems”, 2021 OCEANS - San Diego, in press. [9] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Underwater Visual Localization Using Machine Learning and LSTM: Experiments, and References

Authors:
(1) Luyuan Peng, Acoustic Research Laboratory, National University of Singapore;
(2) Hari Vishnu, Acoustic Research Laboratory, National University of Singapore;
(3) Mandar Chitre, Acoustic Research Laboratory, National University of Singapore;
(4) Yuen Min Too, Acoustic Research Laboratory, National University of Singapore;
(5) Bharath Kalyan, Acoustic Research Laboratory, National University of Singapore;
(6) Rajat Mishra, Acoustic Research Laboratory, National University of Singapore. Table of Links I Introduction II Method III Datasets IV Experiments, Acknowledgment, and References III. DATASETS To train and test our model, we used one dataset collected from an underwater robotics simulator [8] as shown in Fig. 3 as well as two datasets collected from a tank as shown in Fig. 2. In the simulator dataset, we operated the ROV in simulation to perform inspection on a vertical pipe in a spiral motion. The total spatial extent covered by the ROV during the inspection is about 2×4×2 m. 14,400 samples of image-pose pair data were collected. In the first tank dataset, we operated the ROV in a lawnmower path with translations only (Fig.2), and the rotations were minimal. We collected 3,437 data samples. In the second tank dataset, the ROV primarily performed rotation maneuvers at 5 selected points. We collected 4,977 data samples. We augmented the left-camera dataset by adding the right-camera data, and thereby using the geometry of the stereo camera placement to provide more training data. This worked well and yielded better performance. The total spatial extent covered in tank datasets was 0.4×0.6×0.2 m. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Luyuan Peng, Acoustic Research Laboratory, National University of Singapore; (2) Hari Vishnu, Acoustic Research Laboratory, National University of Singapore; (3) Mandar Chitre, Acoustic Research Laboratory, National University of Singapore; (4) Yuen Min Too, Acoustic Research Laboratory, National University of Singapore; (5) Bharath Kalyan, Acoustic Research Laboratory, National University of Singapore; (6) Rajat Mishra, Acoustic Research Laboratory, National University of Singapore. Authors: Authors: (1) Luyuan Peng, Acoustic Research Laboratory, National University of Singapore; (2) Hari Vishnu, Acoustic Research Laboratory, National University of Singapore; (3) Mandar Chitre, Acoustic Research Laboratory, National University of Singapore; (4) Yuen Min Too, Acoustic Research Laboratory, National University of Singapore; (5) Bharath Kalyan, Acoustic Research Laboratory, National University of Singapore; (6) Rajat Mishra, Acoustic Research Laboratory, National University of Singapore. Table of Links I Introduction I Introduction II Method II Method III Datasets III Datasets IV Experiments, Acknowledgment, and References IV Experiments, Acknowledgment, and References III. DATASETS To train and test our model, we used one dataset collected from an underwater robotics simulator [8] as shown in Fig. 3 as well as two datasets collected from a tank as shown in Fig. 2. In the simulator dataset, we operated the ROV in simulation to perform inspection on a vertical pipe in a spiral motion. The total spatial extent covered by the ROV during the inspection is about 2×4×2 m. 14,400 samples of image-pose pair data were collected. In the first tank dataset, we operated the ROV in a lawnmower path with translations only (Fig.2), and the rotations were minimal. We collected 3,437 data samples. In the second tank dataset, the ROV primarily performed rotation maneuvers at 5 selected points. We collected 4,977 data samples. We augmented the left-camera dataset by adding the right-camera data, and thereby using the geometry of the stereo camera placement to provide more training data. This worked well and yielded better performance. The total spatial extent covered in tank datasets was 0.4×0.6×0.2 m. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Underwater Visual Localization Using Machine Learning and LSTM: Datasets

Authors:
(1) Luyuan Peng, Acoustic Research Laboratory, National University of Singapore;
(2) Hari Vishnu, Acoustic Research Laboratory, National University of Singapore;
(3) Mandar Chitre, Acoustic Research Laboratory, National University of Singapore;
(4) Yuen Min Too, Acoustic Research Laboratory, National University of Singapore;
(5) Bharath Kalyan, Acoustic Research Laboratory, National University of Singapore;
(6) Rajat Mishra, Acoustic Research Laboratory, National University of Singapore. Table of Links I Introduction II Method III Datasets IV Experiments, Acknowledgment, and References II. METHOD We implement two different architectures. The first uses pretrained deep convolutional neural networks (DCNN) to extract features from the input images. The resulting feature map is then passed to an affine regressor consisting of dense neural layers to output a 7-dimensional pose vector estimates. The second architecture adds an additional LSTM layer between the DCNN and affine regressor to perform dimensionality reduction by assessing the spatial structure of the image. This is done by four LSTMs which parse the DCNN-output feature map in different directions starting at each of the four image corners to process the spatial trends, and compress it into lower-dimensional information which is easier processed by the affine regressor. The input images used in the training are rescaled to 256×256 pixels before cropping into a 224×224 feature input using centre cropping. To speed up training, the images are normalized by the mean and standard deviation of the images. The poses are also normalized to lie within [-1, 1]. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Luyuan Peng, Acoustic Research Laboratory, National University of Singapore; (2) Hari Vishnu, Acoustic Research Laboratory, National University of Singapore; (3) Mandar Chitre, Acoustic Research Laboratory, National University of Singapore; (4) Yuen Min Too, Acoustic Research Laboratory, National University of Singapore; (5) Bharath Kalyan, Acoustic Research Laboratory, National University of Singapore; (6) Rajat Mishra, Acoustic Research Laboratory, National University of Singapore. Authors: Authors: (1) Luyuan Peng, Acoustic Research Laboratory, National University of Singapore; (2) Hari Vishnu, Acoustic Research Laboratory, National University of Singapore; (3) Mandar Chitre, Acoustic Research Laboratory, National University of Singapore; (4) Yuen Min Too, Acoustic Research Laboratory, National University of Singapore; (5) Bharath Kalyan, Acoustic Research Laboratory, National University of Singapore; (6) Rajat Mishra, Acoustic Research Laboratory, National University of Singapore. Table of Links I Introduction I Introduction II Method II Method III Datasets III Datasets IV Experiments, Acknowledgment, and References IV Experiments, Acknowledgment, and References II. METHOD We implement two different architectures. The first uses pretrained deep convolutional neural networks (DCNN) to extract features from the input images. The resulting feature map is then passed to an affine regressor consisting of dense neural layers to output a 7-dimensional pose vector estimates. The second architecture adds an additional LSTM layer between the DCNN and affine regressor to perform dimensionality reduction by assessing the spatial structure of the image. This is done by four LSTMs which parse the DCNN-output feature map in different directions starting at each of the four image corners to process the spatial trends, and compress it into lower-dimensional information which is easier processed by the affine regressor. The input images used in the training are rescaled to 256×256 pixels before cropping into a 224×224 feature input using centre cropping. To speed up training, the images are normalized by the mean and standard deviation of the images. The poses are also normalized to lie within [-1, 1]. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Underwater Visual Localization Using Machine Learning and LSTM: Method

Authors:
(1) Luyuan Peng, Acoustic Research Laboratory, National University of Singapore;
(2) Hari Vishnu, Acoustic Research Laboratory, National University of Singapore;
(3) Mandar Chitre, Acoustic Research Laboratory, National University of Singapore;
(4) Yuen Min Too, Acoustic Research Laboratory, National University of Singapore;
(5) Bharath Kalyan, Acoustic Research Laboratory, National University of Singapore;
(6) Rajat Mishra, Acoustic Research Laboratory, National University of Singapore. Table of Links I Introduction II Method III Datasets IV Experiments, Acknowledgment, and References I. INTRODUCTION Visual localization is a potential solution for the problem of localization in a known underwater environment for inspection. Inspection missions may often involve operations around marine structures, making acoustic navigation with beacons difficult due to shadowing and multipath [1]. As the vehicle has to operate close to structures, inertial navigation systems, which accumulate errors with time, may not provide sufficient positioning accuracy [1]. In comparison, visual localization using cameras may offer a cost-effective, consistent and accurate alternative in such missions. Previous work has shown that machine learning-based regression methods based on PoseNet [3], can effectively regress a 6-degree-offreedom (DOF) pose from a single 224×224 RGB image with approximately 6 cm position accuracy and 1.7°orientation accuracy when tested on simulated underwater datasets [2]. It was also shown that using a deeper neural network as the extractor may improve the model’s localization accuracy [2]. This work further investigates the effectiveness of such models on underwater datasets and also explores different techniques to further improve localization performance. We have three main contributions: We explore the use of long-short-term memory (LSTM) [4] in the pose regression model to exploit spatial correlation of the image features and to achieve more structured dimensionality reduction [5].


We test the proposed models on underwater datasets collected from a 1.6 m × 1 m × 1 m water-filled tank using a remotely operated vehicle (ROV). The tank offers an environment where we can control lighting and turbidity. The models are able to achieve good accuracy in these datasets, with performance comparable to that obtained with the simulator dataset.


The base dataset consist of images taken from the first camera of a stereo camera mounted on the vehicle. Furthermore, we explore the performance improvement obtained by augmenting the data with additional images from the second camera. Fig. 2 shows some examples of underwater scenes in the tank dataset. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Luyuan Peng, Acoustic Research Laboratory, National University of Singapore; (2) Hari Vishnu, Acoustic Research Laboratory, National University of Singapore; (3) Mandar Chitre, Acoustic Research Laboratory, National University of Singapore; (4) Yuen Min Too, Acoustic Research Laboratory, National University of Singapore; (5) Bharath Kalyan, Acoustic Research Laboratory, National University of Singapore; (6) Rajat Mishra, Acoustic Research Laboratory, National University of Singapore. Authors: Authors: (1) Luyuan Peng, Acoustic Research Laboratory, National University of Singapore; (2) Hari Vishnu, Acoustic Research Laboratory, National University of Singapore; (3) Mandar Chitre, Acoustic Research Laboratory, National University of Singapore; (4) Yuen Min Too, Acoustic Research Laboratory, National University of Singapore; (5) Bharath Kalyan, Acoustic Research Laboratory, National University of Singapore; (6) Rajat Mishra, Acoustic Research Laboratory, National University of Singapore. Table of Links I Introduction I Introduction II Method II Method III Datasets III Datasets IV Experiments, Acknowledgment, and References IV Experiments, Acknowledgment, and References I. INTRODUCTION Visual localization is a potential solution for the problem of localization in a known underwater environment for inspection. Inspection missions may often involve operations around marine structures, making acoustic navigation with beacons difficult due to shadowing and multipath [1]. As the vehicle has to operate close to structures, inertial navigation systems, which accumulate errors with time, may not provide sufficient positioning accuracy [1]. In comparison, visual localization using cameras may offer a cost-effective, consistent and accurate alternative in such missions. Previous work has shown that machine learning-based regression methods based on PoseNet [3], can effectively regress a 6-degree-offreedom (DOF) pose from a single 224×224 RGB image with approximately 6 cm position accuracy and 1.7°orientation accuracy when tested on simulated underwater datasets [2]. It was also shown that using a deeper neural network as the extractor may improve the model’s localization accuracy [2]. This work further investigates the effectiveness of such models on underwater datasets and also explores different techniques to further improve localization performance. We have three main contributions: We explore the use of long-short-term memory (LSTM) [4] in the pose regression model to exploit spatial correlation of the image features and to achieve more structured dimensionality reduction [5]. We test the proposed models on underwater datasets collected from a 1.6 m × 1 m × 1 m water-filled tank using a remotely operated vehicle (ROV). The tank offers an environment where we can control lighting and turbidity. The models are able to achieve good accuracy in these datasets, with performance comparable to that obtained with the simulator dataset. The base dataset consist of images taken from the first camera of a stereo camera mounted on the vehicle. Furthermore, we explore the performance improvement obtained by augmenting the data with additional images from the second camera. Fig. 2 shows some examples of underwater scenes in the tank dataset. We explore the use of long-short-term memory (LSTM) [4] in the pose regression model to exploit spatial correlation of the image features and to achieve more structured dimensionality reduction [5]. We explore the use of long-short-term memory (LSTM) [4] in the pose regression model to exploit spatial correlation of the image features and to achieve more structured dimensionality reduction [5]. We test the proposed models on underwater datasets collected from a 1.6 m × 1 m × 1 m water-filled tank using a remotely operated vehicle (ROV). The tank offers an environment where we can control lighting and turbidity. The models are able to achieve good accuracy in these datasets, with performance comparable to that obtained with the simulator dataset. We test the proposed models on underwater datasets collected from a 1.6 m × 1 m × 1 m water-filled tank using a remotely operated vehicle (ROV). The tank offers an environment where we can control lighting and turbidity. The models are able to achieve good accuracy in these datasets, with performance comparable to that obtained with the simulator dataset. The base dataset consist of images taken from the first camera of a stereo camera mounted on the vehicle. Furthermore, we explore the performance improvement obtained by augmenting the data with additional images from the second camera. Fig. 2 shows some examples of underwater scenes in the tank dataset. The base dataset consist of images taken from the first camera of a stereo camera mounted on the vehicle. Furthermore, we explore the performance improvement obtained by augmenting the data with additional images from the second camera. Fig. 2 shows some examples of underwater scenes in the tank dataset. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Underwater Visual Localization Using Machine Learning and LSTM: Introduction

Evolution of Localization in Swift: From Strings to String Catalogs

সুইফটে স্থানীয়করণের বিবর্তন: স্ট্রিং থেকে স্ট্রিং ক্যাটালগ পর্যন্ত

Entwicklung der Lokalisierung in Swift: Von Strings zu String-Katalogen

Evolución de la localización en Swift: de cadenas a catálogos de cadenas

Evolution de la localisation dans Swift : des chaînes aux catalogues de chaînes

स्विफ्ट में स्थानीयकरण का विकास: स्ट्रिंग्स से स्ट्रिंग कैटलॉग तक

Swift におけるローカリゼーションの進化: 文字列から文字列カタログへ

Swift 지역화의 진화: 문자열에서 문자열 카탈로그로

Evolução da localização em Swift: de strings a catálogos de strings

Эволюция локализации в Swift: от строк к каталогам строк

Swift'de Yerelleştirmenin Evrimi: Dizelerden Dize Kataloglarına

Sự phát triển của bản địa hóa trong Swift: Từ chuỗi đến danh mục chuỗi

Swift 本地化的演变：从字符串到字符串目录

Hi! My name’s Amir. I’m a Web3 Community Manager and a podcast host. To improve the reach of my podcast, 2 months ago, I tried the Rask.ai platform, and have edited over 10 hours of footage since. In this review, I’ll go into the challenges Rask.ai can solve for you, its features, and the pros/cons of the platform. What Is Rask.ai, and Why Did I Buy It? Essentially, Rask.ai is a platform that uses AI for video/audio localization and repurposing. By repurposing, I mean auto reframing to vertical format, and auto-editing long videos to capture the most interesting moments. Rask.ai poses itself as a platform that has all the AI features for content makers in one place, a one-stop-shop of some sort. In one subscription, you get all the AI tools you would need to make your video content better. The main purpose that I had in mind when checking out Rask.ai was to increase the reach of my podcast on TikTok and YouTube. The podcast originally is in Kazakh, but to expand the potential audience and potential advertisers, I was looking for ways to target the English-speaking market as well. Price, Interface, and Functionality The platform is very flexible when it comes to pricing: you have 3 options to choose from, which vary by the price, the amount of minutes, and other available features. Honestly, I think the only metric you should really look at is the amount of minutes, as it will influence your work the most. I’ve picked the second plan, Creator Pro. It includes 100 minutes of edited footage per month and costs me $120. They also offer a plan for $50 and 25 minutes, as well as a corporate plan - $600 and 500 minutes. The biggest plan the platform offers is enterprise, which starts at 2000 minutes with a custom price. Note that all prices are correct if you pay annually - plans with monthly payments will be ~15% more expensive. Last month, I had 2 podcasts instead of one, and I ran out of minutes on my regular plan. However, you can always buy additional minutes at a $3 rate. Setting up the account is very easy (as it should be in 2024): you can use your Google account to log in or create the account with your email. The interface is very clean and isn’t overloaded by different buttons and links. You have FAQ and customer support at the top right corner, accessible from any page, so if you get a bit confused, there’s always that. I can’t say that I’m a tech guru of any kind, but I got used to the website in a matter of minutes. I’ve never worked with a mobile version of the site as I always store all raw footage on my Mac, but if you’re editing a small TikTok/Reels, you can easily do it from your phone. What Does Rask.ai Do, and How Do I Use It? The platform has 2 main features: localization and shorts. Let’s start with the first one, which became a selling point for me. You download a video from your device or paste a link from YouTube or Google Drive, choose the number of speakers, original language, and the language you want to translate the video to, and click “translate.” After that, the magic begins. In a matter of minutes (depending on the length of the video), you get the same video but translated to a different language. The voice-over keeps the voice’s depth, tone, and emotion, and even creates a lip sync, so the video footage is similar to the audio. How do I use that feature? Before discovering Rask.ai, in order to reach the English-speaking market, I was adding EN subtitles to all my podcasts. However, that approach was far from ideal as I was always encountering many barriers: Time-consumption. The current speech-to-text AI solutions are very far from perfect. On 100 AI-generated subtitles, I was getting ~30 errors that I needed to edit manually. Scale that to hours of footage and thousands of text, and you get yourself many hours of manual work.


Accessibility. Subtitles can’t be a permanent solution to expand your foreign audience, especially when it comes to podcasts. When was the last time you paid attention to video footage on a podcast? People listen to them while exercising, cooking, cleaning, and working. My podcast gets 40% more listeners on Spotify and Google Podcasts (which don’t have video) vs YouTube (which has video), meaning that people aren’t looking at the screen and subtitles when listening.


Retention. Even if a person bothers to actively read the subtitles, they still are getting less than people watching the original. Many intonations, jokes, or facial expressions are just not working when perceptive through text. Listeners are too busy reading to actually relax and enjoy the podcast. The Rask.ai platform solved all these problems for me. The amount of manual work I had to do had decreased to practically 0, since in 2 months of using the platform, I haven’t encountered any errors in translations. The retention improved a lot as well: YouTube has statistics that show you how many people watch your videos until the end, and this number significantly improved, since your listeners can actually LISTEN instead of reading. Another feature that I didn’t even know about when getting the subscription to Rask.ai is Shorts. Basically, it takes your hour-long footage and splits the most interesting moments into many short vertical videos. As you probably know, one of the best ways to promote your podcast in 2024 is to share these videos on TikTok, Shorts, and Reels, because these platforms have much better recommendation systems than YouTube, and putting the link to the full podcast in your bio. That’s how I personally discovered many of my favorite podcasts and began to host them myself. Before discovering that feature, I had a friend on a part-time salary doing that for me, because apart from the podcast, I also have a full-time job in Web3 that is taking a significant part of my day. Creating 15 short videos from one podcast took us about 5-6 hours of manual work since you have to fully watch the podcast, then edit all shorts separately, and add some graphics and subtitles. When I discovered a shorts feature on Rask.ai, the same amount of work took me from 5 to 10 minutes (again, depending on the length of the footage). This let me cut my editorial costs down to the sole price of my Rask.ai plan, without having to pay the salary to a whole person doing that. And you say AI won’t take our jobs :) There are a couple of smaller features that I haven’t really tested a lot until today, but I’m planning on exploring them in the future as well. For example, subtitles generation, and API for translations. The company also plans on integrating some features that I’m really excited about: AI video generation (Like MidJourney, but videos), and Zapier integration. Pros, Cons, and Who Can Use It? In conclusion, Rask.ai won’t make your video content more interesting or helpful and won’t make you a superstar overnight. However, with the right distribution strategy and exciting content, Rask.ai will significantly cut your editorial costs, saving you hundreds of dollars on people who translate and edit your content for you, as well as save you A LOT of time, letting you focus on the stuff that really matters: scripts, topics, guests, and other things that make your content better. Rask.ai is a great solution for podcast hosts like myself, social media or community managers producing content for their companies, or TikTok and Instagram influencers that focus on short video content. Pros: Clean and simple interface
Precise translations
Multiple features in one subscription Cons: Relatively high prices
Not all features are ready yet Hi! My name’s Amir. I’m a Web3 Community Manager and a podcast host. To improve the reach of my podcast, 2 months ago, I tried the Rask.ai platform, and have edited over 10 hours of footage since. In this review, I’ll go into the challenges Rask.ai can solve for you, its features, and the pros/cons of the platform. What Is Rask.ai, and Why Did I Buy It? What Is Rask.ai, and Why Did I Buy It? Essentially, Rask.ai is a platform that uses AI for video/audio localization and repurposing. By repurposing, I mean auto reframing to vertical format, and auto-editing long videos to capture the most interesting moments. Rask.ai poses itself as a platform that has all the AI features for content makers in one place, a one-stop-shop of some sort. In one subscription, you get all the AI tools you would need to make your video content better. The main purpose that I had in mind when checking out Rask.ai was to increase the reach of my podcast on TikTok and YouTube. The podcast originally is in Kazakh, but to expand the potential audience and potential advertisers, I was looking for ways to target the English-speaking market as well. Price, Interface, and Functionality Price, Interface, and Functionality The platform is very flexible when it comes to pricing: you have 3 options to choose from, which vary by the price, the amount of minutes, and other available features. Honestly, I think the only metric you should really look at is the amount of minutes, as it will influence your work the most. I’ve picked the second plan, Creator Pro. It includes 100 minutes of edited footage per month and costs me $120. They also offer a plan for $50 and 25 minutes, as well as a corporate plan - $600 and 500 minutes. The biggest plan the platform offers is enterprise, which starts at 2000 minutes with a custom price. Note that all prices are correct if you pay annually - plans with monthly payments will be ~15% more expensive. Last month, I had 2 podcasts instead of one, and I ran out of minutes on my regular plan. However, you can always buy additional minutes at a $3 rate. Setting up the account is very easy (as it should be in 2024): you can use your Google account to log in or create the account with your email. The interface is very clean and isn’t overloaded by different buttons and links. You have FAQ and customer support at the top right corner, accessible from any page, so if you get a bit confused, there’s always that. I can’t say that I’m a tech guru of any kind, but I got used to the website in a matter of minutes. I’ve never worked with a mobile version of the site as I always store all raw footage on my Mac, but if you’re editing a small TikTok/Reels, you can easily do it from your phone. What Does Rask.ai Do, and How Do I Use It? What Does Rask.ai Do, and How Do I Use It? The platform has 2 main features: localization and shorts. Let’s start with the first one, which became a selling point for me. You download a video from your device or paste a link from YouTube or Google Drive, choose the number of speakers, original language, and the language you want to translate the video to, and click “translate.” After that, the magic begins. In a matter of minutes (depending on the length of the video), you get the same video but translated to a different language. The voice-over keeps the voice’s depth, tone, and emotion, and even creates a lip sync, so the video footage is similar to the audio. How do I use that feature? Before discovering Rask.ai, in order to reach the English-speaking market, I was adding EN subtitles to all my podcasts. However, that approach was far from ideal as I was always encountering many barriers: Time-consumption. The current speech-to-text AI solutions are very far from perfect. On 100 AI-generated subtitles, I was getting ~30 errors that I needed to edit manually. Scale that to hours of footage and thousands of text, and you get yourself many hours of manual work. Accessibility. Subtitles can’t be a permanent solution to expand your foreign audience, especially when it comes to podcasts. When was the last time you paid attention to video footage on a podcast? People listen to them while exercising, cooking, cleaning, and working. My podcast gets 40% more listeners on Spotify and Google Podcasts (which don’t have video) vs YouTube (which has video), meaning that people aren’t looking at the screen and subtitles when listening. Retention. Even if a person bothers to actively read the subtitles, they still are getting less than people watching the original. Many intonations, jokes, or facial expressions are just not working when perceptive through text. Listeners are too busy reading to actually relax and enjoy the podcast. Time-consumption. The current speech-to-text AI solutions are very far from perfect. On 100 AI-generated subtitles, I was getting ~30 errors that I needed to edit manually. Scale that to hours of footage and thousands of text, and you get yourself many hours of manual work. Time-consumption. The current speech-to-text AI solutions are very far from perfect. On 100 AI-generated subtitles, I was getting ~30 errors that I needed to edit manually. Scale that to hours of footage and thousands of text, and you get yourself many hours of manual work. Accessibility. Subtitles can’t be a permanent solution to expand your foreign audience, especially when it comes to podcasts. When was the last time you paid attention to video footage on a podcast? People listen to them while exercising, cooking, cleaning, and working. My podcast gets 40% more listeners on Spotify and Google Podcasts (which don’t have video) vs YouTube (which has video), meaning that people aren’t looking at the screen and subtitles when listening. Accessibility. Subtitles can’t be a permanent solution to expand your foreign audience, especially when it comes to podcasts. When was the last time you paid attention to video footage on a podcast? People listen to them while exercising, cooking, cleaning, and working. My podcast gets 40% more listeners on Spotify and Google Podcasts (which don’t have video) vs YouTube (which has video), meaning that people aren’t looking at the screen and subtitles when listening. Retention. Even if a person bothers to actively read the subtitles, they still are getting less than people watching the original. Many intonations, jokes, or facial expressions are just not working when perceptive through text. Listeners are too busy reading to actually relax and enjoy the podcast. Retention. Even if a person bothers to actively read the subtitles, they still are getting less than people watching the original. Many intonations, jokes, or facial expressions are just not working when perceptive through text. Listeners are too busy reading to actually relax and enjoy the podcast. The Rask.ai platform solved all these problems for me. The amount of manual work I had to do had decreased to practically 0, since in 2 months of using the platform, I haven’t encountered any errors in translations. The retention improved a lot as well: YouTube has statistics that show you how many people watch your videos until the end, and this number significantly improved, since your listeners can actually LISTEN instead of reading. Another feature that I didn’t even know about when getting the subscription to Rask.ai is Shorts. Basically, it takes your hour-long footage and splits the most interesting moments into many short vertical videos. As you probably know, one of the best ways to promote your podcast in 2024 is to share these videos on TikTok, Shorts, and Reels, because these platforms have much better recommendation systems than YouTube, and putting the link to the full podcast in your bio. That’s how I personally discovered many of my favorite podcasts and began to host them myself. Before discovering that feature, I had a friend on a part-time salary doing that for me, because apart from the podcast, I also have a full-time job in Web3 that is taking a significant part of my day. Creating 15 short videos from one podcast took us about 5-6 hours of manual work since you have to fully watch the podcast, then edit all shorts separately, and add some graphics and subtitles. When I discovered a shorts feature on Rask.ai, the same amount of work took me from 5 to 10 minutes (again, depending on the length of the footage). This let me cut my editorial costs down to the sole price of my Rask.ai plan, without having to pay the salary to a whole person doing that. And you say AI won’t take our jobs :) There are a couple of smaller features that I haven’t really tested a lot until today, but I’m planning on exploring them in the future as well. For example, subtitles generation, and API for translations. The company also plans on integrating some features that I’m really excited about: AI video generation (Like MidJourney, but videos), and Zapier integration. Pros, Cons, and Who Can Use It? Pros, Cons, and Who Can Use It? In conclusion, Rask.ai won’t make your video content more interesting or helpful and won’t make you a superstar overnight. However, with the right distribution strategy and exciting content, Rask.ai will significantly cut your editorial costs, saving you hundreds of dollars on people who translate and edit your content for you, as well as save you A LOT of time, letting you focus on the stuff that really matters: scripts, topics, guests, and other things that make your content better. Rask.ai is a great solution for podcast hosts like myself, social media or community managers producing content for their companies, or TikTok and Instagram influencers that focus on short video content. Pros: Pros: Clean and simple interface Precise translations Multiple features in one subscription Clean and simple interface Precise translations Multiple features in one subscription Cons: Cons: Relatively high prices Not all features are ready yet Relatively high prices Not all features are ready yet

How to Localize Your Shorts in 3 Clicks

This paper is available on arxiv under CC 4.0 license. Authors: (1) Ali Ghanbari, Dept. of Computer Science, Iowa State University; (2) Deepak-George Thomas, Dept. of Computer Science, Iowa State University; (3) Muhammad Arbab Arshad, Dept. of Computer Science, Iowa State University; (4) Hridesh Rajan, Dept. of Computer Science, Iowa State University. Table of Links Abstract & Introduction Background Motivating Example Proposed Approach Supported DNN Bugs Evaluation Threats to Validity Related Work Conclusion, Acknowledgments & References VII. THREATS TO VALIDITY As with most empirical evaluations, we do not have a working definition of representative sample of DNN bugs, but we made efforts to ensure that the bugs we used in the evaluation is as representative as possible by making sure that our dataset has diverse examples of bugs from each subcategory of model bugs. ategory of model bugs. Many of the bugs obtained from StackOverflow did not come with accompanying training datasets. To address this issue, we utilized the dataset generation API provided by scikit-learn [64] to generate synthetic datasets for regression or classification tasks. We ensured that the errors described in each StackOverflow post would manifest when using the synthesized data points and that applying the fix suggested in the accepted response post would eliminate the bug. However, it is possible that this change to the training process may introduce new unknown bugs. To mitigate this risk, we have made our bug benchmark publicly available [36]. Another potential threat to the validity of our results is the possibility of bugs in the construction of deepmufl itself, which could lead to incorrect bug localization. To mitigate this, we make the source code of deepmufl publicly available for other researchers to review and validate the tool. Another threat to the validity of our results is the potential impact of external factors, such as the stochastic nature of the training process and the synthesized training/testing datasets, as well as system load, on our measurements. To address this, besides using deterministic seeds for dataset generation and splitting, we repeated our experiments with deepmufl three times. Similarly, we also ran other dynamic tools three times to ensure that their results were not affected by randomness during training. We did not observe any differences in effectiveness between the rounds for either deepmufl or the other studied techniques. Additionally, we repeated the time measurements for each round, and reported the average timing, to ensure that our time measurements were not affected by system load. Furthermore, judging whether or not any of the tools detect a bug requires manual analysis of textual description of the bugs and matching it to the tools; output messages which might be subject to bias. To mitigate this bias, we have made the output messages by the tools available for other researchers [36]. Lastly, deepmufl uses a threshold parameter to compare floating-point values (see §IV-C). In our experiments, we used the default value of 0.001 and ensured that smaller threshold values yield the same results.

Mutation-based Fault Localization of Deep Neural Networks: Threats to Validity

This paper is available on arxiv under CC 4.0 license. Authors: (1) Ali Ghanbari, Dept. of Computer Science, Iowa State University; (2) Deepak-George Thomas, Dept. of Computer Science, Iowa State University; (3) Muhammad Arbab Arshad, Dept. of Computer Science, Iowa State University; (4) Hridesh Rajan, Dept. of Computer Science, Iowa State University. Table of Links Abstract & Introduction Background Motivating Example Proposed Approach Supported DNN Bugs Evaluation Threats to Validity Related Work Conclusion, Acknowledgments & References VIII. RELATED WORK Neuralint [12] uses graph transformations [65] to abstract away unnecessary details in the model and check the bug patterns directly on the graph. While Neuralint is orders of magnitude faster than deepmufl, it proved to be less effective than deepmufl in our dataset. DeepLocalize [11] and DeepDiagnosis [8] intercept the training process looking for known bug patterns such as numerical errors. DeepDiagnosis pushes the envelope by implementing a decision tree that gives actionable fix suggestions based on the detected symptoms. A closely related technique, UMLAUT [34], works by applying heuristic static checks on, and injecting dynamic checks in, various parts of the DNN program. deepmufl outperforms DeepLocalize, DeepDiagnosis, and UMLAUT in terms of the number of bugs detected. DeepFD [66] is a recent learning-based fault localization technique which frames the fault localization as a learning problem. MODE [25] and DeepFault [26] implement whitebox DNN testing technique which utilizes suspiciousness values obtained via an implementation of spectrum-based fault localization to increase the hit spectrum of neurons and identify suspicious neurons whose weights have not been calibrated correctly and thus are considered responsible for inadequate DNN performance. MODE was not publicly available, but DeepFault was, but unfortunately it was hard-coded to the examples shipped with its replication package, so we could not make the tool work without making substantial modifications to it, not to mention that these techniques work best on ReLUbased networks and applying them on most of the bugs in our dataset would not make much sense. Other related works are as follows. PAFL [67] operates on RNN models by converting such models into probabilistic finite automata (PFAs) and localize faulty sequences of state transitions on PFAs. Sun et al. [68] propose DeepCover, which uses a variant of spectrum-based fault localization for DNN explainability.

Mutation-based Fault Localization of Deep Neural Networks: Related Work

This paper is available on arxiv under CC 4.0 license. Authors: (1) Ali Ghanbari, Dept. of Computer Science, Iowa State University; (2) Deepak-George Thomas, Dept. of Computer Science, Iowa State University; (3) Muhammad Arbab Arshad, Dept. of Computer Science, Iowa State University; (4) Hridesh Rajan, Dept. of Computer Science, Iowa State University. Table of Links Abstract & Introduction Background Motivating Example Proposed Approach Supported DNN Bugs Evaluation Threats to Validity Related Work Conclusion, Acknowledgments & References IX. CONCLUSION This paper revisits mutation-based fault localization in the context of DNN and presents a novel DNN fault localization technique, named deepmufl. The technique is based on the idea of mutating a pre-trained DNN model and calculating suspiciousness values according to Metallaxis and MUSE approaches, Ochiai and SBI formulas, and two types of impacts of mutations on the results of test data points. deepmufl is compared to state-of-the-art static and dynamic fault localization systems [11], [8], [34], [12] on a benchmark of 109 model bugs. In this benchmark, while deepmufl is slower than the other tools, it proved to be almost two times more effective than them in terms of the total number of bugs detected and it detects 21 bugs that none of the studied tools were able to detect. We further studied the impact of mutation selection on fault localization time. We observed that we can halve the time taken to perform fault localization by deepmufl, while losing only 7.55% of the previously detected bugs. ACKNOWLEDGMENTS The authors thank Anonymous ASE 2023 Reviewers for their valuable feedback. We also thank Mohammad Wardat for his instructions on querying StackOverflow. This material is based upon work supported by the National Science Foundation (NSF) under the grant #2127309 to the Computing Research Association for the CIFellows Project. This work is also partially supported by the NSF grants #2223812, #2120448, and #1934884. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF. REFERENCES [1] IEEE Standard Classification for Software Anomalies, 2010. [2] A. McPeak, “What’s the true cost of a software bug?” https://smartbear. com/blog/software-bug-cost/, 2017, accessed 08/10/23. [3] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015. [4] M. J. Islam, G. Nguyen, R. Pan, and H. Rajan, “A comprehensive study on deep learning bug characteristics,” in ESEC/FSE, 2019, pp. 510–520. [5] M. J. Islam, R. Pan, G. Nguyen, and H. Rajan, “Repairing deep neural networks: Fix patterns and challenges,” in ICSE, 2020, pp. 1135–1146. [6] Y. Zhang, Y. Chen, S.-C. Cheung, Y. Xiong, and L. Zhang, “An empirical study on tensorflow program bugs,” in ISSTA, 2018, pp. 129–140. [7] N. Humbatova, G. Jahangirova, G. Bavota, V. Riccio, A. Stocco, and P. Tonella, “Taxonomy of real faults in deep learning systems,” in ICSE, 2020, pp. 1110–1121. [8] M. Wardat, B. D. Cruz, W. Le, and H. Rajan, “Deepdiagnosis: automatically diagnosing faults and recommending actionable fixes in deep learning programs,” in ICSE. IEEE, 2022, pp. 561–572. [9] K. Pei, Y. Cao, J. Yang, and S. Jana, “Deepxplore: Automated whitebox testing of deep learning systems,” in SOSP, 2017, pp. 1–18. [10] J. Kim, R. Feldt, and S. Yoo, “Guiding deep learning system testing using surprise adequacy,” in ICSE, 2019, pp. 1039–1049. [11] M. Wardat, W. Le, and H. Rajan, “Deeplocalize: fault localization for deep neural networks,” in ICSE, 2021, pp. 251–262. [12] A. Nikanjam, H. B. Braiek, M. M. Morovati, and F. Khomh, “Automatic fault detection for deep learning programs using graph transformations,” TOSEM, vol. 31, no. 1, pp. 1–27, 2021. [13] M. Usman, D. Gopinath, Y. Sun, Y. Noller, and C. S. Pas˘ areanu, “Nn ˘ repair: Constraint-based repair of neural network classifiers,” in CAV, 2021, pp. 3–25. [14] X. Zhang, J. Zhai, S. Ma, and C. Shen, “Autotrainer: An automatic dnn training problem detection and repair system,” in ICSE, 2021, pp. 359–371. [15] W. E. Wong, R. Gao, Y. Li, R. Abreu, and F. Wotawa, “A survey on software fault localization,” TSE, vol. 42, no. 8, pp. 707–740, 2016. [16] M. Papadakis and Y. Le Traon, “Using mutants to locate” unknown” faults,” in ICST, 2012, pp. 691–700. [17] S. Moon, Y. Kim, M. Kim, and S. Yoo, “Ask the mutants: Mutating faulty programs for fault localization,” in ICST, 2014, pp. 153–162. [18] R. A. DeMillo, R. J. Lipton, and F. G. Sayward, “Hints on test data selection: Help for the practicing programmer,” IEEE Computer, vol. 11, pp. 34–41, 1978. [19] R. Abreu, P. Zoeteweij, and A. J. Van Gemund, “On the accuracy of spectrum-based fault localization,” in TAICPART-MUTATION, 2007, pp. 89–98. [20] V. Dallmeier, C. Lindig, and A. Zeller, “Lightweight Bug Localization with AMPLE,” in ISAADD, 2005, pp. 99–104. [21] J. A. Jones, M. J. Harrold, and J. Stasko, “Visualization of test information to assist fault localization,” in ICSE, 2002, pp. 467–477. [22] L. Naish, H. J. Lee, and K. Ramamohanarao, “A model for spectra-based software diagnosis,” TOSEM, vol. 20, no. 3, pp. 1–32, 2011. [23] S. Yoo, “Evolving human competitive spectra-based fault localisation techniques,” in SBSE, 2012, pp. 244–258. [24] X. Xie, F.-C. Kuo, T. Y. Chen, S. Yoo, and M. Harman, “Provably optimal and human-competitive results in sbse for spectrum based fault localisation,” in SBSE, 2013, pp. 224–238. [25] S. Ma, Y. Liu, W.-C. Lee, X. Zhang, and A. Grama, “Mode: automated neural network model debugging via state differential analysis and input selection,” in ESEC/FSE, 2018, pp. 175–186. [26] H. F. Eniser, S. Gerasimou, and A. Sen, “Deepfault: Fault localization for deep neural networks,” in FASE, 2019, pp. 171–191. [27] N. Humbatova, G. Jahangirova, and P. Tonella, “Deepcrime: mutation testing of deep learning systems based on real faults,” in ISSTA, 2021, pp. 67–78. [28] Q. Hu, L. Ma, X. Xie, B. Yu, Y. Liu, and J. Zhao, “Deepmutation++: A mutation testing framework for deep learning systems,” in ASE, 2019, pp. 1158–1161. [29] L. Ma, F. Zhang, J. Sun, M. Xue, B. Li, F. Juefei-Xu, C. Xie, L. Li, Y. Liu, J. Zhao et al., “Deepmutation: Mutation testing of deep learning systems,” in ISSRE, 2018, pp. 100–111. [30] M. Papadakis and Y. Le Traon, “Metallaxis-fl: mutation-based fault localization,” STVR, vol. 25, no. 5-7, pp. 605–628, 2015. [31] F. Chollet et al., “Keras,” https://keras.io, 2015. [32] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and X. Zheng, “Tensorflow: A system for large-scale machine learning,” in OSDI, 2016, p. 265–283. [33] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, highperformance deep learning library,” in NIPS, 2019, pp. 8024–8035. [34] E. Schoop, F. Huang, and B. Hartmann, “Umlaut: Debugging deep learning programs using program structure and model behavior,” in CHI, 2021, pp. 1–16. [35] W. E. Wong and A. P. Mathur, “Reducing the cost of mutation testing: An empirical study,” JSS, pp. 185–196, 1995. [36] A. Ghanbari, D.-G. Thomas, M. A. Arshad, and H. Rajan, “Mutationbased fault localization of deep neural networks,” https://github.com/ ali-ghanbari/deepmufl-ase-2023, 2023. 37] J. H. Andrews, L. C. Briand, and Y. Labiche, “Is mutation an appropriate tool for testing experiments?” in ICSE, 2005, pp. 402–411. [38] M. Papadakis, M. Kintis, J. Zhang, Y. Jia, Y. Le Traon, and M. Harman, “Mutation testing advances: an analysis and survey,” in Advances in Computers, 2019, vol. 112, pp. 275–378. [39] V. Debroy and W. E. Wong, “Using mutation to automatically suggest fixes for faulty programs,” in ICST, 2010, pp. 65–74. [40] A. Ghanbari, S. Benton, and L. Zhang, “Practical program repair via bytecode mutation,” in ISSTA, 2019, pp. 19–30. [41] G. Fraser and A. Arcuri, “Achieving scalable mutation-based generation of whole test suites,” ESE, pp. 783–812, 2015. [42] F. C. M. Souza, M. Papadakis, Y. Le Traon, and M. E. Delamaro, “Strong mutation-based test data generation using hill climbing,” in IWSBST, 2016, pp. 45–54. [43] D. Shin, S. Yoo, M. Papadakis, and D.-H. Bae, “Empirical evaluation of mutation-based test case prioritization techniques,” STVR, p. e1695, 2019. [44] J. P. Galeotti, C. A. Furia, E. May, G. Fraser, and A. Zeller, “Inferring loop invariants by mutation, dynamic analysis, and static checking,” TSE, pp. 1019–1037, 2015. [45] A. Groce, I. Ahmed, C. Jensen, and P. E. McKenney, “How verified is my code? falsification-driven verification (t),” in ASE, 2015, pp. 737– 748. [46] B. Liblit, M. Naik, A. X. Zheng, A. Aiken, and M. I. Jordan, “Scalable statistical bug isolation,” ACM SIGPLAN Notices, vol. 40, no. 6, pp. 15–26, 2005. [47] R. Abreu, P. Zoeteweij, and A. J. Van Gemund, “An evaluation of similarity coefficients for software fault localization,” in PRDC, 2006, pp. 39–46. [48] Wikipedia contributors, “Hierarchical data format — Wikipedia, the free encyclopedia,” 2022, accessed 08/10/23. [49] H. Coles, T. Laurent, C. Henard, M. Papadakis, and A. Ventresque, “Pit: a practical mutation testing tool for java,” in ISSTA, 2016, pp. 449–452. [50] P. Ammann and J. Offutt, Introduction to software testing. Cambridge University Press, 2016. [51] R. Just, F. Schweiggert, and G. M. Kapfhammer, “Major: An efficient and extensible tool for mutation analysis in a java compiler,” in ASE, 2011, pp. 612–615. [52] Y.-S. Ma, J. Offutt, and Y. R. Kwon, “Mujava: an automated class mutation system,” STVR, pp. 97–133, 2005. [53] D. Schuler and A. Zeller, “Javalanche: Efficient mutation testing for java,” in ESEC/FSE, 2009, pp. 297–298. [54] “Junit,” http://junit.org/, 2019, accessed 08/10/23. [55] “Testng documentation,” https://testng.org/doc/documentation-main. html, 2017, accessed 08/10/23. [56] M. M. Morovati, A. Nikanjam, F. Khomh, Z. Ming et al., “Bugs in machine learning-based systems: A faultload benchmark,” arXiv, 2022. [57] X. Li and L. Zhang, “Transforming programs and tests in tandem for fault localization,” Proceedings of the ACM on Programming Languages, vol. 1, no. OOPSLA, pp. 1–30, 2017. [58] C. Parnin and A. Orso, “Are automated debugging techniques actually helping programmers?” in ISSTA, 2011, pp. 199–209. [59] P. S. Kochhar, X. Xia, D. Lo, and S. Li, “Practitioners’ expectations on automated fault localization,” in ISSTA, 2016, pp. 165–176. [60] D. Shin and D.-H. Bae, “A theoretical framework for understanding mutation-based testing methods,” in ICST, 2016, pp. 299–308. [61] A. V. Pizzoleto, F. C. Ferrari, J. Offutt, L. Fernandes, and M. Ribeiro, “A systematic literature review of techniques and metrics to reduce the cost of mutation testing,” Journal of Systems and Software, vol. 157, p. 110388, 2019. [62] J. Zhang, “Scalability studies on selective mutation testing,” in ICSE, vol. 2, 2015, pp. 851–854. [63] A. J. Offutt, J. Pan, K. Tewary, and T. Zhang, “An experimental evaluation of data flow and mutation testing,” Software: Practice and Experience, vol. 26, no. 2, pp. 165–176, 1996. [64] scikit-learn Contributors, “scikit-learn: Machine learning in python,” 2020, accessed 08/10/23. [Online]. Available: https://scikit-learn.org/stable/ [65] R. Heckel, “Graph transformation in a nutshell,” ENTCS, vol. 148, no. 1, pp. 187–198, 2006. [66] J. Cao, M. Li, X. Chen, M. Wen, Y. Tian, B. Wu, and S.-C. Cheung, “Deepfd: Automated fault diagnosis and localization for deep learning programs,” in ICSE, 2022, p. 573–585. [67] Y. Ishimoto, M. Kondo, N. Ubayashi, and Y. Kamei, “Pafl: Probabilistic automaton-based fault localization for recurrent neural networks,” IST, vol. 155, p. 107117, 2023. [68] Y. Sun, H. Chockler, X. Huang, and D. Kroening, “Explaining image classifiers using statistical fault localization,” in ECCV, 2020, pp. 391–406.

Mutation-based Fault Localization of Deep Neural Networks: Conclusion, Acknowledgments & References

This paper is available on arxiv under CC 4.0 license. Authors: (1) Ali Ghanbari, Dept. of Computer Science, Iowa State University; (2) Deepak-George Thomas, Dept. of Computer Science, Iowa State University; (3) Muhammad Arbab Arshad, Dept. of Computer Science, Iowa State University; (4) Hridesh Rajan, Dept. of Computer Science, Iowa State University. Table of Links Abstract & Introduction Background Motivating Example Proposed Approach Supported DNN Bugs Evaluation Threats to Validity Related Work Conclusion, Acknowledgments & References IV. PROPOSED APPROACH Our technique deepmufl comprises four components: (1) mutation generator, (2) test case splitter, (3) mutation executor/tester, and (4) suspiciousness calculator. Figure 2 depicts these components as processes, numbered accordingly, taking inputs and producing outputs. Mutation generator (marked ⃝1 in Figure 2), applies 79 mutators on all the layers of the input buggy DNN, so as to generate a pool of mutants, i.e., variants of the original, buggy DNN model with small perturbations, e.g., replacing activation function of a layer. Test case splitter (marked ⃝2 in the figure), applies the original buggy DNN on a given set of test data points, i.e., test cases (or input values) paired with their expected output values, so as to partition the set into two subset, namely passing test cases and failing test cases. Passing test cases are referred to as those input values for which the expected output matches that of produced by the original model, whereas failing test cases are referred to as those input values for which the expected output does not match that of produced by the model. This component also stores the output of the original model on each of the test cases. Next, mutation executor (which is also called mutation tester, marked ⃝3 in the figure) applies the generated mutants on each of the passing and failing test cases and the results are compared to that of the original model recorded in the previous step. This amounts to a mutation execution matrix that is used to calculate suspiciousness values for each layer in the model (marked ⃝4 in the figure). The user may instruct deepmufl to use a specific fault localization formula, e.g., MUSE or Metallaxis with SBI or Ochiai, for calculating suspiciousness values. The layers are then ranked based on the calculated suspiciousness values for user inspection. The ranked list is accompanied with the information about the mutations conducted on each layer to facilitate debugging. A. Mutation Generator Mutation generator component receives the original, buggy DNN model and generates as many variants of the model, i.e., mutants, as possible, by systematically mutating every elements of the input model. This component implements 79 mutators. Mutators can be viewed as transformation operators that when applied to a given element, e.g., a neuron or a layer, in the model, returns a new variants of the model with that particular element mutated. Table 2 lists all the mutators implemented in deepmufl, the types of model elements on which they can operate, and the way each mutator affects the target element. These mutators are inspired by the ones implemented in the existing mutation analysis systems, e.g., [27], [28], [29], [49], [50], [51], [52], [53], to name a few. Ma et al. [29], and Hu et al. [28], define so-called model-level mutators that also operate on pre-trained models. Direct reuse of all of their mutators was not possible, as those mutators depend on random values which would introduce a source of nondeterminism in deepmufl: mutating the same model element with random values, e.g., Gaussian fuzzing, as in [29], would yield a different mutant each time, making deepmufl to produce different outputs on each run for the same model. In general, as far as MBFL is concerned, using variable values (whether it is deteministic or not), instead of the current hard-coded ones, for mutation of weights would not bring about any benefit, as the goal here is to break the model in some way and observe its impact on originally failing and passing test cases. We argue that not all model bugs could be emulated using mutators at the level of pre-trained models, e.g., missing batch normalization, but the mutators listed in Table 2 are sufficient for emulating a subset of such bugs, e.g., wrong activation function or missing/redundant layer. Please see §V for a more detailed discussion on supported bugs. Mutation generation in deepmufl is done directly on the trained model and there is no need for retraining the model. This makes deepmufl quite fast and perhaps more attractive from a practical point of view. However, this comes with a risk of not being traceable, i.e., a mutation on pre-trained .h5 model does not directly correspond to a line of source code for the user to inspect. In the Keras programs that we studied, this limitation was mitigated by the fact that the models with Sequential architecture were implemented using a wellunderstood structure and mapping layer numbers/identifiers in deepmufl’s reports to source code was trivial. In a future work, with the help of simple auto-generated annotations, e.g., for lexical scoping of the code snippet for model declaration, we will extend deepmufl to automatically map layer numbers/identifiers in its reports to actual lines of source code. Humbatova et al. [27] argue about the importance of mutators emulating real DNN faults. We acknowledge that mutators emulating real faults would help generating more informative reports that would also give hints on how to fix the program. However, unlike mutation analysis, the main objective of an MBFL technique is to assign suspiciousness values to the program elements which can, in theory, be done using any kind of mutator, whether or not it makes sense from the standpoint of a developer. It is worth noting that the alternative design decision of using DeepCrime [27] as a mutation generation engine for deepmufl would result in finding more bugs than the current version of deepmufl, e.g., bugs related to training hyper-parameters or training dataset, but such a design is expected to be impacted by the nondeterminacy inherent in training process and, given the fact that we do not employ any training data selection, would be significantly slower due to numerous re-training. Nevertheless, finding more bugs would be an incentive for exploring this alternative as a future work. B. Test Case Splitter Before we introduce this component, we need to clarify certain terminology. .** Definition 1 A data point in a testing dataset for a DNN model is defined to be a pair of the form (I, O), where I ∈ R m and O ∈ R n, with m and n being positive integers. In this paper I is called test case, test input, or input, while O is called expected output or ground-truth value. Given a test dataset, test case splitter component applies the original model on each of the test cases for the data points and checks if the model output matches to the expected output. If the two outputs match, then the corresponding test case will be marked as passing, otherwise it will be marked as failing. This component also records the output produced by the original model to be used during impact analysis, described below. C. Mutation Executor (Mutation Tester) We start describing this component with a definition. Definition 2. A mutation execution matrix E is a k×l matrix, where k is the number of generated mutants, while l is the number of test cases. Each element Ei,j in the matrix is a member of the set {✓, ✗, ❍}, wherein ✓ indicates that the i th mutant impacts j th test case, whereas ✗ indicates that the mutant does not affect the test case. ❍ denotes a nonviable mutant, i.e., a mutant that fails when loading or applying it on a test case. Such mutants might be generated, e.g., due to creating a shape error [32] after the mutation. Mutation executor component construct mutation execution matrix by applying each of the generated mutants (see §IV-A) on the failing and passing test cases to determine which of the test cases are impacted by which mutants. The impact of mutation on test cases is measured using two types of impacts, i.e., type 1 impact and type 2 impact, defined below. Definition 3. Given a DNN model M, its mutated version M′ , and a data point (I, O), we define the two types of impacts: • Type 1: Mutant M′ impacts the test case I if M(I) = O but M′ (I) ̸= O, or M(I) ̸= O but M′ (I) = O. In other words, type 1 impact tracks pass to fail and fail to pass test cases, a la MUSE [17]. ` • Type 2: Mutant M′ impacts the test case I if M(I) ̸= M′ (I). In this definition, M(I) or M′ (I) denotes the operation of applying model M or M′ on the test case I. It is worth noting that checking for equality of two values can be tricky for regression models, as those models approximate the expected values. To work around this problem, deepmufl compares values obtained from regression models using a user-defined delta threshold, i.e., the values are deemed equal if their absolute difference is no more than a threshold. By default, deepmufl uses a threshold of 0.001. This is the approach adopted by well-known testing frameworks for comparing floating-point values [54], [55]. Also, whether deepmufl uses type 1 or type 2 impact is a user preference and is determined alongside the threshold. D. Suspiciousness Value Calculator Armed with the above definitions, we can now give concrete definitions for the terms used in Eq. 1, 2, and 3, specialized to DNNs. • Given a model element, i.e., neuron or a layer, e, M(e) is defined to be the set of all mutants generated by mutating e. These sets are produced by mutation generator process. • Assuming that m is a mutant on the element e, Tf (m, e) (or Tp(m, e)) is defined as the set of failing (or passing) test cases that are impacted in type 1 or type 2 fashion by m. More concretely, Tp(m, e) = {t | Em,t = ✓ ∧ t is passing}, similarly Tf (m, e) = {t | Em,t = ✓ ∧ t is failing}. These two sets are defined using a quite similar notation; the readers are encouraged to review Definitions 2 and 3 to avoid confusion. • Tf (or Tp) is the set of originally failing (or passing) test cases. These sets are constructed by test case splitter. • F ⇝ P (or P ⇝ F), for a model element e, is defined as the set of originally failing (or passing) test cases that turned into passing (or failing) as a result of some mutation on e. More concretely, assuming the execution matrix E is constructed using type 1 impact, F ⇝ P is defined as {t | t is failing ∧ ∃m ∈ M(e) · Em,t = ✓}. Similarly, P ⇝ F is defined as {t | t is passing ∧ ∃m ∈ M(e)·Em,t = ✓}. In other words, these sets track all the failing/passing test cases that are type 1 impacted by some mutant of a given element. These two sets are defined using a quite similar notation; the readers are encouraged to review Definitions 2 and 3 to avoid confusion. Having specialized definitions for the terms used in the fault localization formulas described earlier, we are now able to calculate suspiciousness values for the elements in a DNN model. Guided by the user preferences, deepmufl calculates all the values for |Tf (m, e)|, etc., and plugs into the specified formula to calculate suspiciousness values for the elements. It is worth noting that if all the mutants generated for a given element are nonviable, MUSE formula (Eq. 3) and all the variants of Metallaxis (e.g., Eq. 1), by definition, will return 0 as the suspiciousness value for the element. Nonviable mutants do not contribute toward localizing the bug, therefore they are considered overhead to the fault localization process. Fortunately, based on our observations in our dataset of buggy DNNs, nonviable mutants are rare. Equivalent mutants are another source of overhead for deepmufl. Currently, we do not have any means of detecting equivalent mutants, but we argue that these mutants do not impact MBFL results, as they are equivalent to the original model and do not impact any passing or failing test cases.

Mutation-based Fault Localization of Deep Neural Networks: Proposed Approach

This paper is available on arxiv under CC 4.0 license. Authors: (1) Ali Ghanbari, Dept. of Computer Science, Iowa State University; (2) Deepak-George Thomas, Dept. of Computer Science, Iowa State University; (3) Muhammad Arbab Arshad, Dept. of Computer Science, Iowa State University; (4) Hridesh Rajan, Dept. of Computer Science, Iowa State University. Table of Links Abstract & Introduction Background Motivating Example Proposed Approach Supported DNN Bugs Evaluation Threats to Validity Related Work Conclusion, Acknowledgments & References VI. EVALUATION We evaluate deepmufl and compare it to state-of-the-art static and dynamic DNN fault localization techniques, by investigating the following research questions (RQs). • RQ1 (Effectiveness): How does deepmufl compare to state-of-the-art tools in terms of the number of bugs detected? How many bugs does deepmufl detect from each subcategory of model bugs in our dataset and how does that compare to state-of-the-art tools? What are the overlap of detected bugs among deepmufl and other fault localization techniques? • RQ2 (Efficiency): What is the impact of mutation selection on the effectiveness and efficiency of deepmufl? How does deepmufl compare to state-of-the-art tools in terms of end-to-end fault localization time? A. Dataset of DNN Bugs To evaluate deepmufl and compare it to state-of-the-art DNN fault localization techniques, we queried StackOverflow Q&A website for posts about Keras that had at least one accepted answer. Details about the SQL query used to obtain the initial list of posts is available online [36]. The query resulted in 8,412 posts that we manually sieved through to find the programs with model bugs. Specifically, we kept the bugs that satisfied the following conditions. • Implemented using Sequential API of Keras, • The bug in the program was a model bug supported by deepmufl as described in §V, and • The bug either had training datatset available in the post in some form (e.g., hard-coded, clearly described in the body of the post, or a link to the actual data was provided) or we could see the described error using synthetic data obtained from scikit-learn’s dataset generation API. This resulted in 102 bugs and we paired each bug with a fix obtained from the accepted answer to the question. We further added 7 bugs from DeepLocalize dataset [11] that are also coming from StackOverflow and we paired these bugs also with their fixes that are obtained from the most up-voted answer. Thus, we ended up with 109 bugs in total. To the best of our knowledge, this is the largest dataset of model bugs obtained from StackOverflow and it overlaps with the existing DNN bug datasets from previous research [12], [56]. Our bug dataset contains 85 classifiers (45 fully-connected DNNs, 29 CNNs, and 11 RNNs) and 24 regression models (19 fullyconnected DNNs, 3 CNNs, and 2 RNNs). And each category has at least one example of model bugs. Therefore, we believe that our dataset is greatly representative of model bugs, i.e., the bugs supported by deepmufl (and other tools that support this type of bugs), as we have examples of each sub-category of bug in various locations of the models for various regression and classification tasks. After loading the training dataset for the bugs, we fitted the buggy models three times and stored them in .h5 file format separately. The repetition was conducted to take randomness in training into account. Randomness in data generation was mitigated by using deterministic random seeds. For fault localization purposes, we used the test dataset, and if it was not available, we used the training dataset itself. When we had to use synthesized data points, we deterministically splitted the generated set of data into training and testing datasets. B. Baseline Approaches and Measures of Effectiveness In RQ1 and RQ2, we compare five different configurations of deepmufl to recent static and dynamic DNN fault localization tools. The five configurations of deepmufl are as follows. • [30]: In this setting, we use the Metallaxis formula to calculate suspiciousness values of model elements. Metallaxis, by default, uses SBI [46] to calculate suspiciousness values for individual mutants. A recent study [57] provides empirical evidence on the superiority of Ochiai [47] over SBI when used within Metallaxis formula. Thus, we considered the following four combinations: type 1 impact: (1) SBI formula (i.e., Eq. 1); (2) Ochiai formula (i.e., Eq. 2), and type 2 impact: (3) SBI formula (i.e., Eq. 1); (4) Ochiai formula (i.e., Eq. 2). Metallaxis • [17]: We used the default formula of MUSE to calculate the suspiciousness of model elements. For this, only type 1 impact is considered, as the heuristics behind MUSE are defined based on type 1 impact. MUSE Our technique follows a more traditional way of reporting root causes for the bugs [30], [17], [57], [19], [22], [21], [15], in that it reports a list of potential root causes ranked based on the likelihood of being responsible for the bug. This allows the users find the bugs faster and spend less time reading through the fault localization report, which in turn increases practicality of the technique [58]. We have used top-N, with N = 1, metric to measure the effectiveness of deepmufl in RQ1 and RQ2. Specifically, if the numbers of any of the buggy layers of the bug appeared in the first place in the output of deepmufl, we reported it as detected, otherwise we marked the bug as not-detected. We emphasize that top-1 metric gives a strong evidence on the effectiveness of deepmufl, as the developers usually only inspect top-ranked elements, e.g., over 70% of the developers only check top-5 ranked elements [59]. Our selection criteria for the studied fault localization techniques are: (1) availability; (2) reproducibility of the results reported in the original papers, so as to have a level of confidence on the correctness of the results reported here; and (3) support for model bugs in our dataset, so that we can make a meaningful comparison to deepmufl. Below we give a brief description of each of the selected tools, why we believe they support model bugs, and how we have interpreted their outputs in our experiments, i.e., when we regard a bug being detected by the tool. A static fault localization tool that uses 23 rules to detect faults and design inefficiencies in the model. Each rule is associated with a set of rules of thumb to fix the bug that are shown to the user in case the precondition for any of the rules are satisfied. The five rules described in Section 4.2.1 of the paper target model bugs. Neuralint produces outputs of the form [Layer L ==> MSG] ∗ , where L is the suspicious layer number, and MSG is a description of the detected issue and/or suggestion on how to fix the problem. A bug is deemed detected by this tool if it is located in the layer mentioned in the output message or the messages describe any of the root causes of the bug. 1) Neuralint: A dynamic fault localization technique that detects numerical errors during model training. One of three rules described in Section III.D of the paper checks model bugs related to wrong activation function. DeepLocalize produces a single message of the form Batch B Layer L : MSG, where B is the batch number wherein the symptom is detected and L and MSG are the same as we described for Neuralint. A bug is deemed detected if it is located in the layer mentioned in the output message or the message describes any of the root causes of the bug. 2) DeepLocalize: A tool similar to DeepLocalize, but with more bug pattern rules and a decision procedure to give actionable fix suggestions to the users based on the observations. All 8 rules in Table 2 of the paper monitor the symptoms of model bugs. Similar to DeepLocalize, DeepDiagnosis produces a single message of the form Batch B Layer L : MSG1 [OR MSG2], where B and L are the same as described in DeepLocalize and MSG1 and MSG2 are two alternative solutions that the tool might suggest to fix the detected problem. A bug is deemed detected if it is located in the layer mentioned in the output message or the message describes any of the root causes of the bug. 3) DeepDiagnosis: : A hybrid, i.e., a combination of static and dynamic, technique that works by applying heuristic static checks on, and injecting dynamic checks in, the program, parameters, model structure, and model behavior. Violated checks raise error flags which are propagated to a web-based interface that uses visualizations, tutorial explanations, and code snippets to help users find and fix detected errors in their code. All three rules described in Section 5.2 of the paper target model bugs. The tool generates outputs of the form [< MSG1 > · · · < MSGm >] ∗ , where m > 0 and MSGi is a description of the problem detected by the tool. A bug is deemed detected if any of the messages match the fix prescribed by the ground-truth. 4) UMLAUT C. Results To answer RQ1, we ran deepmufl (using its five configurations) and four other tools on the 109 bugs in our benchmark. We refer the reader to the repository [36] for the raw data about which bug is detected by which tool, and here we describe the summaries and provide insights. At top-1, deepmufl detects 42, 47, 26, 37, and 53 bugs using its Metallaxis SBI + Type 1, Metallaxis Ochiai + Type 1, Metallaxis SBI + Type 2, Metallaxis Ochiai + Type 2, and MUSE, respectively, configurations. Meanwhile Neuralint, DeepLocalize, DeepDiagnosis, and UMLAUT detect 21, 26, 30, and 25, respectively, bugs. Therefore, as far as the number of bugs detected by each technique is concerned, MUSE configuration of deepmufl is the most effective configuration of deepmufl, significantly outperforming studied techniques, and Metallaxis Ochiai + Type 2 is the least effective one, outperformed by DeepDiagnosis. An empirical study [57], which uses a specific dataset of traditional buggy programs, concludes that Metallaxis Ochiai + Type 2 is the most effective configuration for MBFL. Meanwhile, our results for DNNs corroborates the theoretical results by Shin and Bae [60], i.e., we provide empirical evidence that in the context of DNNs MUSE is the most effective MBFL approach. Table 3 reports more details and insights on the numbers discussed above. Specifically, it reports the number of bugs detected by each configuration of deepmufl an four other studied tools from each sub-category of model bugs present in our dataset of bugs. As we can see from the upper half of the table, MUSE is most effective in detecting bugs related to activation function (SC1), bugs related to model type/properties (SC2), and wrong/redundant/missing layer (SC4), while Metallaxis Ochiai + Type 1 configuration outperforms other configurations in detecting bugs related to layer properties (SC3). Similarly, from bottom half of the table, we can see that other tools are also quite effective in detecting bugs related to activation function, with DeepDiagnosis being the most effective one among others. We can also observe that UMLAUT has been the most effective tool in detecting bugs related to layer properties. As we can see, MUSE configuration of deepmufl is consistently more effective than other tools across all bug sub-categories. Table 4 provides further insights on the overlap of bugs detected by each variant of deepmufl and those detected by the other four tools. Each value in row r and column c of this table, where 2 ≤ r ≤ 5 and 2 ≤ c ≤ 6, denotes the percentage of bugs detected by the deepmufl variant corresponding to row r and tool corresponding to column c. The values inside the parenthesis are the actual number of bugs. For example, 8 out of 42, i.e., 19.05%, of the bugs detected by Metallaxis SBI + Type 1 configuration of deepmufl are also detected by DeepLocalize. The last column of the table reports same statistics, except for all four of the studied tools combined. As we can see from the table, 60.38% of the bugs detected by MUSE configuration of deepmufl are already detected by one of the four tools, yet it detects 21 (=53-32) bugs that are not detected by any other tools. This is because deepmufl approaches fault localization problem from a fundamentally different aspect giving it more flexibility. Specifically, instead of looking for conditions that trigger a set of hard-coded rules, indicating bug patterns, deepmufl breaks the model using a set of mutators to observe how different mutation impact the model behavior. Then by leveraging the heuristics underlying traditional MBFL techniques, it performs fault localization using the observed impacts on the model behavior. Listing 2 shows an example of a model bug that only deepmufl can detect. The problem with this regression model is that it does not output negative values, and given the fact that the dataset contains negative target values, the model achieves high MSE. Since this model does not result in any numerical errors during training, DeepLocalize does not issue any warning messages, and since MSE decreases and the model does not show any sign of erratic behavior DeepDiagnosis does not detect the bug. UMLAUT’s messages instruct adding softmax layer and checking validation accuracy which is clearly not related to the problem, because the bug is fixed by changing the activation function of the last layer to tanh and normalizing the output values. Lastly, Neuralint issues an error message regarding incorrect loss function which also seems to be a false positive. To answer RQ2, we ran deepmufl and the other four tools on a Dell workstation with Intel(R) Xeon(R) Gold 6138 CPU at 2.00 GHz, 330 GB RAM, 128 GB RAM disk, and Ubuntu 18.04.1 LTS and measured the time needed for model training as well as the MBFL process to complete. We repeated this process four times, and in each round of deepmufl’s execution, we randomly selected 100% (i.e., no selection), 75%, 50%, and 25% of the generated mutants for testing. Random mutation selection is a common method for reducing the overhead of mutation analysis [61], [35]. During random selection, we made sure that each layer receives at least one mutants, so that we do not mask any bug. The last row in Table 5 reports the average timing (of 3 runs) of MBFL in each round of mutation selection. The table also reports the impact of mutation selection on the number of bugs detected by each configuration of deepmufl. As we can see, in MUSE configuration of deepmufl, by using 50% of the mutants, one can halve the execution time and still detect 92.45% of the previously detected bugs. Therefore, mutation selection can be used as an effective way for curtailing MBFL time in DNNs. For a fair comparison of deepmufl to state-of-the-art fault localization tools in terms of efficiency, we need to take into account the fact that deepmufl requires a pre-trained model as its input. Thus, as far as the end-to-end fault localization time from an end-user’s perspective is concerned, we want to take into consideration the time needed to train the input model in addition the time needed to run deepmufl. With training time taken into account, deepmufl takes, on average, 1492.48, 1714.63, 1958.35, and 2192.4 seconds when we select 25%, 50%, 75%, and 100% of the generated mutants, respectively. We also emphasize that the time for DeepLocalize and DeepDiagnosis varied based on whether or not they found the bug. Given the fact that a user could terminate the fault localization process after a few epochs when they lose hope in finding bugs with these two tools, we report two average measurements for DeepLocalize and DeepDiagnosis: (1) average time irrespective of the fact that the tools succeed in finding the bug; (2) average time if the tools successfully finds the bug. Unlike these two tools, the time for Neuralint and UMLAUT does not change based on the fact that they detect a bug or not. DeepLocalize takes on average 1244.09 seconds and it takes on average 57.29 seconds when the tool successfully finds the bug. These numbers for DeepDiagnosis are 1510.71 and 11.05 seconds, respectively. Meanwhile, Neuralint and UMLAUT take on average 2.87 seconds and 1302.61 seconds to perform fault localization. D. Discussion It is important to note that while deepmufl outperforms state-of-the-art techniques in terms of the number of bugs detected in our dataset, it is not meant to replace them. Our dataset only covers a specific type of bugs, i.e., model bugs, while other studied techniques push the envelope by detecting bugs related to factors like learning rate and training data normalization, which are currently outside of deepmufl’s reach. We observed that combining all the techniques results in detecting 87 of the bugs in our dataset; exploring ways to combine various fault localization approaches by picking the right tool based on the characteristics of the bug is an interesting topic for future research. Moreover, depending on the applications and resource constraints, a user might prefer one tool over another. For example, although Neuralint might be limited by its static nature, e.g., it might not be able analyze models that use complex computed values and objects in their construction, it takes only few seconds for the tool to conduct fault localization. Thus, in some applications, e.g., online integration with IDEs, approaches like that of Neuralint might be the best choice A major source of overhead in an MBFL technique is related to the sheer number of mutants that the technique generates and tests [62], [61]. Sufficient mutator selection [63] is referred to the process of selecting a subset of mutators that achieve the same (or similar) effect, i.e., same or similar mutation score and same or similar number of detected bugs, but with smaller number of mutants generated and tested. For the mutators of Table 2, so far, we have not conducted any analysis on which mutators might be redundant, as a reliable mutator selection requires a larger dataset that we currently lack. We postpone this study as a future work. Combining fault localization tools can be conducted with the goal of improving efficiency. We see the opportunity in building faster, yet more effective, fault localization tools by predicting the likely right tool upfront for a given model or running tools one by one and moving on to the next tool if we have a level of confidence that the tool will not find the bug. We postpone this study for a future work. Lastly, we would like to emphasize that comparisons to the above-mentioned techniques in a dataset of bugs that deepmufl supports is fair, as the other tools are also designed to detect bugs in the category of model bugs. However, making these tools to perform better than this, would require augmenting their current rule-base with numerous new rules, yet adding new rules comes with the obligation of justifying the generality and rationale behind them, which might be a quite difficult undertaking. deepmufl, on the other hand, approaches the fault localization problem differently, allowing for more flexibility without the need for hard-coded rules.

Mutation-based Fault Localization of Deep Neural Networks: Evaluation

This paper is available on arxiv under CC 4.0 license. Authors: (1) Ali Ghanbari, Dept. of Computer Science, Iowa State University; (2) Deepak-George Thomas, Dept. of Computer Science, Iowa State University; (3) Muhammad Arbab Arshad, Dept. of Computer Science, Iowa State University; (4) Hridesh Rajan, Dept. of Computer Science, Iowa State University. Table of Links Abstract & Introduction Background Motivating Example Proposed Approach Supported DNN Bugs Evaluation Threats to Validity Related Work Conclusion, Acknowledgments & References V. SUPPORTED DNN BUGS Due to the complex nature of DNN bugs, and MBFL itself, we do not hope to give a formal account of what types of DNN bugs deepmufl is capable of localizing. Instead, we attempt to provide as accurate description of the supported bugs as possible and discuss the way such bugs manifest in DNN programs. The discussion given in this section leverages the characterization of DNN bugs provided by previous research [7], [4], [6]. As we mentioned earlier, current version of deepmufl operates on pre-trained Keras Sequential models. This means that much of the information, such as training hyperparameters and whether or not the input data is normalized, has already been stripped away from the input to deepmufl, and the current version of the technique is not capable of detecting any bug related to training process, e.g., training data and hyper-parameters. Moreover, a pre-trained model does not contain bugs related to tensor shapes (as otherwise, the training would fail with shape errors), and since deepmufl does not receive the source code of the buggy model as input, bugs related to GPU usage and API misuse are also out of the reach of the technique, by definition. This leaves us with the socalled model bugs [7] the extent to which deepmufl is capable of localizing is explicated below. The four model bug subcategories are represented with identifiers SC1, ..., SC4 in the rest of this paper for ease of reference. These bugs are related to the use of wrong activation function in a layer. We observed that deepmufl detects this type of bugs and it also gives actionable, direct fixes. • SC1: Activation function. These bugs include wrong weight initialization, wrong network architecture, wrong model for the task, etc. Through altering the weights and biases in layers, deepmufl detects weight/bias initialization bugs and pinpoint the location of the bug, but the bug report produced by the tool does not provide helpful information for fixing. • SC2: Model type or properties. These bugs include wrong filter/kernel/stride size, sub-optimal number of neurons in a layer, wrong input sample size, etc. deepmufl detects and pinpoints the bugs related to filter/kernel/stride size and sub-optimal number of neurons. We observed that, the former case sometimes produce non-viable mutants. In the cases where deepmufl produced viable mutants, effective MBFL takes place and it has been able to pinpoint the bug location and provide explanation on how to fix it. In the latter case, deepmufl was able to pinpoint the bug location, but the bug report does not give helpful information on how to fix the bugs in this sub-category. • SC3: Layer properties. These bugs include missing/extra one dense layer, missing dropout layer, missing normalization layer, etc. By mutating the layers adjacent to the missing layer, or deleting the redundant layer, deepmufl detects and pinpoints the location of the missing/culprit layer, and in most of the cases, it provides useful information on how to fix such bugs. • SC4: Missing/redundant/wrong layer. By manually examining the bug descriptions provided by the programmers in our dataset of bugs, and also referring to the previous work on DNN bugs and root cause characterization [4], these bugs might manifest as low test accuracy/MSE, constant validation accuracy/MSE/loss during training, NaN validation accuracy/MSE/loss during training, dead nodes, vanishing/exploding gradient, and saturated activation. At this point, we would like to emphasize that deepmufl is not intended to repair a model, so if a mutation happens to be the fix for the buggy model, the model has to be retrained from scratch so that correct weights and biases will be calculated.

Mutation-based Fault Localization of Deep Neural Networks: Supported DNN Bugs

This paper is available on arxiv under CC 4.0 license. Authors: (1) Ali Ghanbari, Dept. of Computer Science, Iowa State University; (2) Deepak-George Thomas, Dept. of Computer Science, Iowa State University; (3) Muhammad Arbab Arshad, Dept. of Computer Science, Iowa State University; (4) Hridesh Rajan, Dept. of Computer Science, Iowa State University. Table of Links Abstract & Introduction Background Motivating Example Proposed Approach Supported DNN Bugs Evaluation Threats to Validity Related Work Conclusion, Acknowledgments & References III. MOTIVATING EXAMPLE In this section, we first describe how deepmufl helps programmers detect and fix bugs by presenting a hypothetical use case scenario and then motivate the idea behind deepmufl by describing the details of how deepmufl works, under the hood, on the example developed in the use case story. Courtney is a recent college graduate working as a junior software engineer at an oil company, which frequently makes triangular structures, made of epoxy resin, of varying sizes to be used under the water. The company needs to predict with at least 60% confidence that a mold of a specific size will result in an epoxy triangle after it has been dried, and potentially shrunk, and it does not need to spend time on cutting and/or sanding the edges. Over time, through trial and error, the company has collected 1,000 data points of triangle edge lengths and whether or not a mold of that size resulted in a perfect triangle. Courtney’s first task is to write a program that given three positive real numbers a, b, and c, representing the edge lengths of the triangle mold, determines if the mold will result in epoxy edges that form a perfect triangle. As a first attempt, she writes the program shown in Listing 1. The program uses 994 out of 1,000 data points for training a model. After testing the model on the remaining 6 data points, she realizes that the model achieves no more than 33% accuracy. Fortunately, Courtney uses an IDE equipped with a modern DNN fault localization tool, named deepmufl, which is known to be effective at localizing bugs that manifest as stuck accuracy/loss. She uses deepmufl, with its default settings, i.e., Metallaxis with SBI, to find the faulty part of her program. The tool receives the fitted model in .h5 format [48] together with a set of testing data points T and returns a ranked list of model elements; layers, in this case. After Courtney provides deepmufl with the model saved in .h5 format and the 6 testing data points that she had, within a few seconds, the tool returns a list with two items, namely Layer 2 and Layer 1, corresponding to the lines 5 and 4, respectively, in Listing 1. Once she navigates to the details about Layer 2, she receives a ranked list with 5 elements, i.e., Mutant 12: replaced activation function ‘relu’ with ‘softmax’, ..., Mutant 10: divided weights by 2, Mutant 11: divided bias by 2. By seeing the description of Mutant 12, Courtney immediately recalls her machine learning class wherein they were advised that in classification tasks they should use softmax as the activation function of the last layer. She then changes the activation function of the last layer at Line 5 of Listing 1 from relu to softmax. By doing so, the model achieves an accuracy of 67% on the test dataset, and similarly on a cross-validation, exceeding the expectations of the company. We now describe how deepmufl worked, under the hood, to detect the bug via Metallaxis’ default formula. Figure 1 depicts the structure of the model constructed and fitted in Listing 1. Each edge is annotated with its corresponding weight and the nodes are annotated with their bias values. The nodes are using ReLU as the activation function. In this model, the output T is intended to be greater than the other output if a, b, and c form a triangle, and ∼T should be greater than or equal to the other output, otherwise. Table 1 shows an example of how deepmufl localizes the bug in the model depicted in Figure 1. In the first two columns, the table lists the two layers, and within each layer, the neurons. For each neuron three mutators are applied, i.e., halving weight values, halving bias value, and replacing the activation function. More mutators are implemented in deepmufl, but here, for the sake of simplicity, we only focus on 3 of them and also restrict ourselves to only one activation function replacement, i.e., ReLU vs. softmax. As we saw in Courtney’s example, she had a test dataset T with 6 data points which initially resulted in 33% accuracy. These six data points are denoted T1, ..., T6 in Table 1, where correctly classified ones are colored green, whereas misclassified data points are colored rose. deepmufl generates 12 mutants for the model of Figure 1, namely, M1, ..., M12. Each mutant is a variant of the original model. For example, M1 is the same as the model depicted in Figure 1, except the weights of the incoming edges to neuron N1 are halved, i.e., 0.51, -0.38, and -0.52 from left to right, while M9 is the same as the model depicted in Figure 1, except that the activation functions for N3 and N4 are softmax instead of relu. After generating the mutants, deepmufl applies each mutant on the inputs T1, ..., T6 and compares the results to that of the original model. For each data point T1, ..., T6, if the result obtained from each mutant M1, ..., M12 is different from that of the original model, we put a bullet point in the corresponding cell. For example, two bullets points in the row for M3 indicate that the mutant misclassifies the two data points that used to be correctly classified, while other data points, i.e., T1, ..., T4, are misclassified as before. Next, deepmufl uses SBI formula [46] to calculate suspiciousness values for each mutant m ∈ {M1, ..., M12}, individually. These values are reported under the one but last column in Table 1. Lastly, deepmufl takes the maximum of the suspiciousness values of the mutants corresponding to a layer and takes it as the suspiciousness value of that layer (c.f. Eq. 1 in §II). In this particular example, layer L1 gets a suspiciousness value of 0, while L2 gets a suspiciousness value of 1. Thus, deepmufl ranks L2 before L1 for user inspection and for each layer it sorts the mutants in descending order of their suspiciousness values, so that the user will understand what change impacted most the originally correctly classified data points. In this case, M12 and M9 wind up at the top of the list, and as we saw in Courtney’s story, the information associated with the mutations helped fixing the bug.

Stories, News & Mentions about LOCALEAI

Localeai’s Stories on HackerNoon