UpMetrics empowers community-impact organizations and philanthropic funding sources to be more data-driven and maximize their impact

UpMetrics

upmetrics

Product management has grown in popularity recently as businesses recognize the importance of developing products that meet customer needs and solve real-world problems. Product managers plan ahead of time and collaborate with cross-functional teams to oversee the entire product development process, from inception to building to launch and beyond. Product managers need an in-depth knowledge of the market, customers, business, and data in order to strategize effectively. As a product manager, you define the product vision, and roadmap, conduct research, launch the product, and continuously refine the product. You work with cross-functional teams, including engineering, , and sales, to create a unique, value-driven, and successful product. design There are different types of product managers, including Product Owners, Growth Product Managers, Technical Product Managers, and Platform Product Managers. Although their core responsibilities are the same, their roles differ in subtle ways. You will learn their distinct roles in this blog. The role of a product manager is complex, and your tasks and responsibilities overlap with other positions. On a daily basis, PMs handle a multitude of tasks. As a PM, you conduct daily meetings with the product team to address their concerns and discuss the progress of the ongoing projects. You talk to the users to understand their pain points and how the product is performing. You analyze data, prioritize features using various metrics, and create roadmaps to constantly refine the product. You strive to make the product better by staying aligned with the company's goals. This article will provide an in-depth insight into the roles and responsibilities of product managers. You will also understand what it takes to be a successful product manager. At the end of this article, you will find a few useful tips for aspiring product managers to help them succeed as product managers. What is a Product Manager? Product Managers oversee the product development process from its inception and ensure the growth and success of the product by collaborating with multiple stakeholders. PMs have a unique blend of technical skills, business savvy, and leadership ability to lead the way for the product team to craft the best product. In order to be successful, it is imperative that the Product Manager understands the product's purpose, target market, and value proposition. PMs articulate and communicate the vision of the product to the development team. In this way, they empower the product team and facilitate product growth. Product managers are often called the “CEO” of the product. However, as Marty Cagan has rightly mentioned in his book Inspired: PMs coordinate with cross-functional teams to ensure their efforts are aligned with the goals of the business and also cater to solving the real-life problems of customers. ‘even though they are called CEOs, they are not a boss to anyone.’ According to Marty Cagan, a product manager’s four key responsibilities include the following: Customer Market/Industry Data Business In other words, as a product manager, you should engage in extensive research to understand the marketing landscape, industry, customer requirements, and business viability to carefully strategize the product journey. The role of a product manager is complex, and many of their tasks and responsibilities overlap with other positions. Product managers have multiple roles and take care of various things like business cases, , communication with the business, decision makings, and so much more product roadmaps Types of Product Managers Product managers, as previously stated, are crucial to the success of any product. PMs are in high demand as more businesses recognize the value of product management. The responsibilities of a product manager would differ amongst companies.  Although all product managers and product teams do the same essential tasks, they slightly differ in terms of titles and job descriptions. Here, we will be discussing five types of product managers and their major responsibilities. Read on to understand how they are different from each other. Type Responsibilities Product Manager PMs oversee the entire product lifecycle from ideation to launch. PMs: define the product roadmap conduct market research launch the product continuously refine the product. They work with cross-functional teams such as engineering, design, and sales to bring the product to market. Product Owner The product owner is responsible for defining the product backlog and prioritizing the features for the development team. The product owner serves as the customer's voice and guarantees that the development team yields value-driven products. Growth Product Manager A growth product manager is responsible for owning up metrics like product revenue and user engagement. They identify opportunities for customer acquisition, retention, and upselling. Technical Product Manager They have a strong technical background. Technical product managers primarily work with the development team to curate products that fulfill user requirements. Platform Product Manager A platform product manager handles a platform that caters to different customers or products. They determine the roadmap for the platform, manage interactions with partners and clients, and guarantee the platform's scalability and security. Responsibilities of a Product Manager The stage of the firm has an impact on the roles and responsibilities of a PM. While mature companies have more defined PM roles, startups typically expect PMs to be in charge of "everything." Before accepting a PM position, it is recommended Product Managers study the company's philosophy, culture, and requirements to make sure they are a company fit. Day-to-Day tasks of a Product Manager with the Product Team The role of a PM demands a unique combination of technical knowledge, creativity, and business acumen. It is pretty obvious that each day is distinct for PMs. You are responsible for a vast array of tasks, and handling them all can be challenging. The following section will help you understand what do a product manager do on a day-to-day basis. Tasks Description Daily Standups PMs conduct regular meetings with the product team to analyze and understand the progress of ongoing projects. This helps them to address their concerns and to ensure the goals are met on time. Conduct Customer Interviews Conducting customer interviews helps them understand user requirements. This assists them in understanding the value that the customer receives from the product. Additionally, it aids in the refinement of products according to the needs of the users. Networking PMs frequently get ideas from their peers and constantly search for fresh and innovative approaches to product development. The improvement of the development process is facilitated by having effective communication with various stakeholders both inside and outside the firm. They motivate their team to go beyond their comfort zone by conceiving innovative ideas backed by compelling evidence. Roadmap Planning Defining roadmaps allows every stakeholder to be on the same page regarding product development. It ensures transparency in the product journey and keeps everyone informed and aligned toward achieving the larger goal. Curating a roadmap can be challenging. This is why software like is there to be a companion for PMs like you to create comprehensive roadmaps effortlessly. Zeda.io Data Analysis PMs have to analyze qualitative and quantitative data to measure the product's success and scalability. Data helps them prioritize feature requests and also to identify areas of improvement. In short, data analysis would help them to define success metrics. Framing PRD You have to develop PRD on a daily or weekly basis. A PRD has information on the product’s purpose, requirements, and user stories. PRD is important to keep the team focused on attaining the common goal. Cross functional Responsibilities of a Product Manager Product managers are responsible for creating products with customer and business value. Therefore, a PM is constantly in collaboration with customers, the product team, and key stakeholders of the company. As a PM, you should be able to find a balance between the needs of the customer and that of the company stakeholders. You are the facilitator of a cross-functional product group made up of experts from various divisions, including engineering, product marketing, design, customer service, finance, legal, marketing, program and project management, sales, and support. For the delivery of the entire product experience, close collaboration with each of these areas is essential. This section will help you understand how this cross-functionality works. Engineering PMs work closely with the engineering team to develop and improve the product development process. By converting business requirements into technical specifications, they ensure that the developer team meets the company's objectives. Finance They work with the financial team to track metrics and maximize business growth to fulfill your responsibilities. They track all the expenditures and revenue that are involved in the business. Also, often, they work with them to decide on the pricing of the product. Sales PMs collaborate with the sales team to ensure that the product meets the needs of the customer, is competitive in the market, and meets revenue targets. They work together on customer feedback, market insights, sales enablement, revenue targets, and marketing strategies. Customer Support They work together to collect feedback from customers and identify and resolve issues. They collaborate to ensure customer retention and satisfaction. Legal The legal team ensures that the product complies with relevant regulations, is protected by intellectual property laws, and that contracts and agreements are in place to support the product's development and ongoing management. Marketing Product managers devise strategies for positioning the product in the market. Also, PM works with marketing teams for the management of product releases. Project Management The project management team collaborates with PMs to ensure that their initiatives are delivered on time and within budget. Skills, Traits, and Qualifications of a Product Manager A product manager ought to have certain skills, traits, and qualities to drive the growth and success of the product. product management Let us look at a few skills that a PM needs: Strong Communication Skills As a PM, you must communicate effectively with numerous internal and external stakeholders. You should be able to influence others, empower your team, and navigate situations using your communication skills. Product Managers need to be able to articulate clearly the business case of a given initiative or feature, so the product team understands why they building it. Problem-Solving Skills It is important that you have problem-solving skills as a PM because you would have to solve the issues of your team. You should be able to tactically surpass any challenges that come your way. Product Managers should have the superpower to deal with problems and make informed decisions. Leadership Leadership skills help collaborate with cross-functional teams and make sure that everyone is focused on the same objectives and moving in the same direction. It aids Product Managers in keeping the Product team motivated to perform better and empowers them to take ownership of their work. Technical Knowledge Although having a technical background is optional for a PM, it can be useful in understanding the product and the development process through a technical lens. Also, technical know-how would help you communicate better with the development team. Market Knowledge You must thoroughly understand the market and target audience to ensure that the product meets their needs. With this knowledge, Product Managers can devise ground-breaking strategies to differentiate themselves in the market and create a product that customers value. Data Analysis Data analysis is a major skill that a product manager has to master. You have to sift through loads of data; hence, Product Managers must be able to independently analyze it effectively and pull the most out of it. Creativity You need to be creative and think outside the box to develop innovative solutions to problems. Creativity is essential to differentiate your brand, product, or service from your competitors. Adaptability You must be able to adjust your approach and adapt rapidly because the nature of your job is dynamic. The job of a PM can be challenging; you have to brace yourself up to face challenges to learn, evolve and thrive. How to Succeed as a Product Manager? The success of any product manager relates to the success of the product they are working for. Here are some factors that a product manager needs to keep in mind to be successful. Understand Company Goals You must align the product vision with the overarching business objectives in order to ensure the product's success. This helps a Product Manager stay on task and prevents from straying from the business goals and outlook. Build Strong Relationships As a PM, you must constantly interact with multiple stakeholders for improved productivity and efficiency. It is you who transforms your team's vision into reality and paves the way for product growth. Hence, Product Managers have to build strong relationships with stakeholders and cross-functional teams to ensure that everyone is aligned and working together. Be Customer-Focused Ultimately, the goal of any product is to create viable features that customers will find valuable. You need to be customer-focused and make decisions based on customer needs and customer feedback. This ensures customer satisfaction and thereby increases the customer retention rate. Strive for Perfection To work effectively, you must stay on top of changing trends and keep upskilling yourself. Persevere to create something above the ordinary for your customers, and shouldn’t settle for anything other than the best. Tips for Aspiring Product Managers Acquire Relevant Experience: In order to lay a solid career foundation, acquire experience in product management, marketing, or a related industry through internships and working on different projects. Gain Relevant Knowledge: You can attend that are available online to learn well around about product management. Moreover, it is also important to learn the language of computing and the language of business. several courses Establish a Strong Network: To keep abreast of the most recent trends and advancements, establish a solid network of people in the product management sector and go to industry events. Learn from Mentors: Seek guidance and mentorship of seasoned product managers and benefit from their knowledge and experiences. Develop Strong Communication Skills: To effectively communicate with stakeholders and cross-functional teams, develop strong communication skills. Emotional Intelligence- EQ enables you to put yourself in the shoes of your customers, understand their emotions and motivations, and create products that meet their needs. PMs benefit from high EQ when it comes to communicating effectively, building strong relationships, and resolving conflicts. Wrap Up Long story short, product managers play a critical role in developing, planning, and marketing a product. They oversee cross-functional teams, conduct market and user research, and are in charge of the product's success. Product managers need to be effective communicators, strategic thinkers, experts in their fields, and capable leaders. In order to be a successful PM, one has to have an in-depth knowledge of the market, business, customer, and data. Moreover, they should know the product inside out to guide the product team to conceive the best output that aligns with the business objectives. Managing a multitude of tasks at the same time can be chaotic for a product manager. However, is there for your rescue as it brings order to chaos. Zeda.io FAQs Should PMs know how to code? It is not essential for a product manager to know how to code. However, learning the coding language would help them collaborate better with the developer team. How to become a PM with an MBA? Even though an MBA would give you a perspective on how businesses work, it is optional to have an MBA to be a product manager.In order to shine as a PM, you need to have a deep understanding of the market, business, customer, and data. What are the four critical skills of a product manager? Problem-Solving, strategic thinking, communication, and leadership are the four critical skills necessary for a product manager. It's critical to acquire appropriate expertise, develop a strong network, seek out mentors, and maintain a customer-focused attitude if you want to become a product manager.

What Are the Roles & Responsibilities of Product Managers?

¿Cuáles son las funciones y responsabilidades de los gerentes de producto?

Quels sont les rôles et responsabilités des chefs de produit ?

उत्पाद प्रबंधकों की भूमिकाएं और जिम्मेदारियां क्या हैं?

プロダクト マネージャーの役割と責任とは?

Quais são as funções e responsabilidades dos gerentes de produto?

Vai trò & Trách nhiệm của Người quản lý sản phẩm là gì?

产品经理的角色和职责是什么？

Metrics are crucial to DevOps and Continuous Delivery as part of the process of continuous improvement. However, you must balance collecting and displaying data with being flooded with information. You need to decide what data to collect and what smaller set of data you pay attention to at any time. If your car had a dashboard that displayed every metric it collects through the engine management system, there'd be no space for a windscreen. Early cars only featured an on their dashboard, which measured the electric current between the battery and voltage regulator. This was important as it told you the charging system was working. There was no speedometer. Cars could only achieve a top speed of 35 miles per hour, and the suspension discouraged traveling at this speed. It wasn't necessary to measure velocity. ammeter In a modern car, no dashboard space is given to an ammeter (though a battery light will illuminate if there's a problem). But, you will find a speedometer in almost every car. The current dashboard design reflects the development of cars and the broader system they operate in. Engines are more powerful, suspension systems are better, roads are generally smoother, and more cars are on the road. Attitudes towards safety have also changed. Similarly, the metrics you collect and show will change over time as you respond to differences in your team and organization. As you create and evolve your measurement system, there are some common mistakes you should avoid. Ignoring the data The first problem with metrics is that you expend great effort collecting them, but they're not always used. This happens even in organizations that regularly review the numbers. Your data needs some process that generates insights. You can turn what you discover from the information into theories you use for experiments. The experiments should then provide new data to start the cycle again. The only good reason to collect DevOps metrics is to learn more about your work and find ways to improve it. If you collect data , the chances are it will be misused or not used at all. just in case Activity bias There are 4 levels of measurement, with activity often being the earliest and easiest to collect. You can usually track activity in near real-time. The result of the activity is not typically available as an output or outcome-based metric until some later date. Activity metrics are usually a built-in feature of your existing tools, so they're already available. The problem is not all activity represents progress. Some activities might even reduce output and result in worse outcomes. You can use activity metrics to predict future changes to outputs and outcomes. To do this, you must continually test the relationship between your activity metrics and your related output or outcome metrics. If you only measure activity, you get a lot of motion but no progress. Tracking too much at once The number of metrics you collect and display can increase, often rapidly. Before long, you've filled your dashboard with charts, and you don't know what's important and what isn't. You need to keep your tracked metrics lean, current, and relevant. When a chart is no longer helpful, you should remove it from your dashboard. You should also consider whether the metric still needs to be collected and retire it if you don't have a good reason to track it. If you already have a dashboard, open it, and for each chart ask, "What would I do differently if this number went up or down?". Revisit this question frequently, deleting any chart where you don't have an answer. Your metric set should focus on key long-term output and outcome metrics, with dashboards showing short-term metrics across all categories in your current improvement efforts. Jumping into tools Data visualization tools like Microsoft Power BI, Tableau, or Google Data Studio are among the most useful software products you'll have in your organization. Many business tools have a grid or text-based interface, but data tools come stacked with colorful animated charts. It's easy to be distracted by creating an engaging dashboard. If you don't begin with metric design, you end up with lots of pleasing dashboards that have no impact on your daily work. You need dashboards and charting tools to help you make sense of the information but design the metrics first. It's better to start in low-fidelity to collect meaningful metrics. It's okay to start with a simple spreadsheet or even a whiteboard. After you work out which measurements are helpful to your team and organization, start automating the collection and creating slick displays. If you invest too much time creating a stunning dashboard, you'll find it harder to delete charts when they're no longer required. Standardization Some organizations try to replicate the success of a high-performing team by making other teams follow the same process. This is rarely successful, as each team works on different problems and has different skill levels. Just as the process and practices need to be context-specific, so do the metrics. You should address metrics as part of your continuous improvement activities. For a team with long lead times, you measure batch size, queue times, and the time work spends in each state. These metrics are unsuitable for a team whose primary problem is quality. This requires the data, insight, theory, and experiment cycle where you look at the information you have about the problem you want to solve, form a theory about what might improve the issue, then run an experiment to test your idea. The metrics you collect also signal to the team what's important . You often see improvements simply because your metrics communicate that you care about some aspect of software delivery. at the current time Relying on eyeballs You should create a simple dashboard to display the metrics you're tracking for experiments. This should be displayed on an information radiator so everyone on the team can see the data. However, if you only react to data when someone looks at a dashboard, you'll crowd the dashboard with too much information and miss key events. Long-term metrics you keep to track progress over time will seem as important as the short-term metrics you use to improve your software delivery. The key to resolving the problem of retaining long-term metrics without cluttering your dashboard is to put in place a monitoring and alerting process for the data. Automated alerts should tell you if the metrics cross a threshold, and you could use anomaly detection to tell you if something interesting happened. With metrics tracked automatically, you can remove them from the dashboard to free up space. Rewarding performance If your team is working to increase its deployment rate, it can be tempting to incentivize them with a reward if they achieve daily deployments. This approach to motivation leads to poor outcomes. A team might let other critical work slip to achieve the goal - not to cheat the system but because you made daily deployments more important than anything else. In the landmark book, explains that attempting to manage people with incentives leads to long-term harm for your organization. Hundreds of studies have found that people do worse work when offered rewards. Alfie Kohn Using metrics to create a competitive atmosphere, either for individual performance, comparison of different teams, or to gamify the workplace (where you introduce game elements as a form of "fun" competition), all lead to trouble. Competition conflicts with what you really need in your organization: collaboration. If you work on the assumption that people want to do good work, you'll find you don't need to use rewards or punishments to get them to improve their performance. Conclusion The 5 DORA metrics and the SPACE framework provide pre-built, balanced ways to measure software delivery performance. (There used to be 4 DORA metrics, but an additional metric was added.) reliability A good set of metrics will mix leading indicators to predict performance with lagging indicators that check the accuracy of predictions. Measurements should span activity, output, system output, and outcome categories. Whatever you measure, you need to constantly refine your metrics to ensure they remain useful to your team and organization. Ideally, the metrics you collect are relevant to a specific experiment you're running to test a theory. If you use metrics well, you amplify performance and learning as you seek to become one of the elite performers in software delivery. Also published here.

7 Common Misconceptions about DevOps Metrics

The choice of working with Java, which opens up a wide range of possibilities for programming, is well-founded. This is a typeset, high-level programming language created by  Sun Microsystems, later redeemed Oracle, with fairly flexible syntax. For more than 20 years, Java has maintained its popularity, taking one of the leading positions in the market of world technologies. One of its advantages is automated memory management in Java, especially when compared to the C and C++ languages, which don’t have this function. It’s performed using the built-in garbage collector technology «Garbage Collector» or GC. To explain this important feature, it’s worth noting that the system accumulates in its memory objects that occupy memory but are not used. That tool significantly helps in the elimination of such a problem as a . memory leak Of course, Java is not the only platform that has such an important and useful function as a garbage collector. But in addition to its other features, such as HotSpot optimizer or excellent backward compatibility, comparison – Java certainly stands out among other platforms. So what’s the problem if Java has such a useful function of distribution and clearing of memory? Let’s see. What are memory leak and their causes To better understand the essence of a direct example, you should start with the Java memory structure. 1. Memory structure and its cleanup Java application data can be stored in spatial blocks such as «Stack» or «Heap.» Stack memory is a repository of store references to «Heap» elements and types of primitive value; Heap contains dynamic objects referenced by variables in the stack. As a result, under default settings, a heap can occupy much more space in memory than a stack. However, these settings can be changed manually by setting the heap size below the stack value after increasing it. Stack has the principle (enters last, leaves first). Whenever a new method is called with reference to an element or a primitive value, a block of memory is released at the top of the stack: LIFO The heap, unlike the stack, is not cleaned by itself. So, the system is in dire need of a garbage collector function. Without its presence, we can manage only the size of the heap: A reference to an object from a heap that may contain a stack variable becomes a suitable target over time for the garbage collector, depending on its type. And, when red-colored items appear in the heap memory, they can be assembled by the collector. Example of garbage items in memory: 2. Memory leak Java memory leak is a kind of error when GC leaves elements in the system that are not used. This may occur because of the inability to remove some garbage items that may be referenced in the stack. A typical example of the leak: This leak has a negative impact on the capacity to utilize the resources of the system and its overall productivity. If this problem is ignored, the system can completely exhaust data storage space, and end with an irreversible error, «Java Out Of Memory Error.» It’s better to use tools of memory management to optimize it. The most relevant of them: OpenView Operations (OVO) by HP; JMETER by Apache; IBM Tivoli by IBM; JProbe Profiler by TechRepublic. 3. Causes of memory leaks in Java Memory leaks in Java can occur due to unforeseen errors in the code, which keeps references to unwanted objects in the cloud. These links block GC functions. As a result, it’s not possible to clean the repository, which is not usefully used by these elements. Main causes of Java memory leaks: Unlimited caching; Overflow of files in one session; Excess of replacement operating system pages; Bugs in the system of users’ data; Inserting elements in a collection without deleting them; Non-reproducible ways of listening. Memory leak in Java and its types Various types of leaks are possible. Their differences are based on how they arise and what caused them. Most common types of leaks: 1. Excessive use of static variables The life cycle of static fields in Java usually corresponds to the application session time, without taking into account garbage collection with «ClassLoader» capabilities. When analyzing the heap of memory during command execution, we can observe an increase in memory between control points 1 and 2: But stopping method «populateList()» at the point 3, in VisualVM – heap repository remains untreated. But if you don’t take into account «static» in line 2, it will change the memory values: In this case, after manipulation with the method «populateList()», the heap memory is cleaned by the collector, as all references to objects are deactivated: 2. Availability of internal classes with reference to external To initialize static classes, you always need examples from external classes. Each non-static default class contains a hidden reference to the class in which it’s saved. If you use an internal class object – it will not collect by GC, even if you close an external class object. For example, take a class that contains a non-static value and has a reference to several volumetric elements. In this case, the creation of an internal class object has such statistical indicators: Banal change of the value of the internal or anonymous class to a static, the memory values change radically. This can occur as a result of the contents of the internal class of references to objects external, thus blocking the main function of GC. 3. Unclosed resources in the app Whenever we create a new connection or open a thread, the JVM always allocates space in memory for new threads or connections. Such connections may have session objects with a comprehensive database. If we leave such resources not closed, we risk allowing the storage to become blocked. This creates the risk that the garbage collector will not be able to detect and remove these objects. And if you ignore the maintenance of stable and correct closing of resources, they will fill up the memory, up to the appearance of an error «Java Out Of Memory Error.» 4. Using finalize() methods In the scheme, when we have a class with a redefined finalize() method, its object may not collect GC in time but will queue for deletion. In addition, incorrectly overridden code in finalize() and its speed mismatch to garbage collector can cause an error «OutOfMemoryError.» For example, take a class with its redefined method finalize() and for which time is needed. If you have a large number of elements collected by GC, such a score of the heap is formed: If you simply remove a redefined finalize(), you can get this value: 5. Using interned strings In the Java system version 6, it was necessary to be careful about the use of volumetric strings. Version 7 moved the change of the string pool that moved to HeapSpace from PermGen. If you call a method «intern()» to read a large string – it is saved to a string pool in constant memory (PermGen). This method is stored in PermGen until the end of the session, which causes the application to run out of memory. An example of permanent memory is when reading a string of a file without its international: 6. ThreadLocals engagement ThreadLocal – a tool that creates stream security by closing the values of its variables. At the same time, all threads have a hidden link to the duplicate ThreadLocal variable, and save their copies instead of using the resource in all threads. This feature, besides its usefulness – has errors. They can affect leaks when misused. ThreadLocal variables must be collected by GC after deleting the thread that contains them. However, some system servers may not be able to use this feature right. This can occur because the server does not create a new thread for all requests but applies the entire thread pool. Pools in servers reuse threads, making them inaccessible to the garbage collector and taking up memory. 7. Implementing incorrect equals() and hashCode() In creating new classes, we can often encounter the override error of equals() and hashCode() methods. Such methods are commonly used by HashSet and HashMap, and if they contain override errors, this can cause excessive memory consumption. For example, you can use the ORM Hibernate, which takes the equals() and hashCode() methods to process items and store them in the cache. In the event that these methods are not overridden, Hibernate will not analyze the items, and the cache will fill their copies, causing a memory leak. Symptoms and Java memory leak detection 1. Memory leak symptoms There are several suspicious points that may indicate a leak: persistent and unforeseen system failures; unstable application functionality support; an error «Java.lang.OutOfMemoryError» occurred during a long session; connection object removal by the system; significant reduction in overall system performance. 2. How to detect Java memory leak In order to detect a leak, several tools and techniques are required, as well as a combination of them. There is a list of the trusted methods: : tools that track files and items that take up space in the repository. They are able to detect leaks and analyze the correct distribution of the elements used in the system. Also, estimate the time taken to process away. 2.1. Memory profilers The most common tools for profiling Java memory: JProfiler by EJ-Technologies; Java VisualVM by Oracle Corporation; YourKit by Yourkit GmbH. This is a tool for creating instant and timely snapshots of the heap in the Java memory store. Such images are needed to control the number of objects used and their weight in memory. Also, the tool tracks the number of elements created by the system and what may affect the leak. 2.2. Engagement heap dump: This tool is able to demonstrate changes to heap configuration in the repository and GC. It provides the most accurate features and performance of the application. And optimizes the collector’s performance by identifying suitable elements in the heap, its alternative methods, and JVM parameters. 2.3. Activating the verbose garbage collection log. For example, activate a verbose collection in-app using JVM startup methods: «-XX: +UseSerialGC -Xms1024m -Xmx1024m -verbose:gc» The «-verbose:gc» argument activates the recording of collection information. The log, by default, is saved to stdout and produces rows for all GC. Also specify sequential GC, using the «-XX: +UseSerialGC» argument, and set the heap sizes. This can help to detect a leak in a timely manner and configure application health metrics. Examples of GC log activation can be found on the GitHub service. How to fix memory leaks. Error correction methods In order to address this problem, it is necessary first to consider the reason why it has arisen. Therefore, it is necessary to identify the type of leak and solve the problem based on its variety. For each type, there is a different scheme how to fix memory errors in Java: 1. How to resolve a memory error? 1.1. When using static variables: Minimize the use of static fields in the system. In this case, instead of urgent loading of objects – you can use the «lazy» one. 1.2. If there are internal classes, with reference to external: You can convert an internal class to a static class if it does not need external class elements. 1.3. If the resources of the application are not closed: «finally» should be activated in a timely manner to complete the use of resources. 1.4. When using finalize() methods: Should reduce to zero any work with finalists. 1.5. When using interned strings Try to upgrade the Java app to the latest version. This can work by moving the pool of strings to the free place of the heap, after the 6th version. And to avoid the error «Out Of Memory Error», in work with bulk strings you can extend the «PermGen» size. 1.6. When using ThreadLocals Stable cleaning of ThreadLocal variables when they are not needed. ThreadLocal that has the remove() property – removes variable values for all current threads. It must be closed in the «finally» block to ensure that it is deactivated. 1.7. When implementing incorrect equals() and hashCode() When creating new elements, it is always optimal to override the equals() and hashCode() path. 2. Other ways to eliminate memory leaks You can also apply other methods to combat memory leaks when there is no clear understanding of the reason for their occurrence. 2.1. Using benchmarking: For a detailed analysis of code performance in Java, you can use its testing with benchmarking. In this way, it’s easy to assess the productivity of different ways of performing tasks by comparing their effectiveness. This gives preference to best practices, which will avoid unnecessary memory consumption. 2.2. Reference object app: The option of applying special reference objects in Java instead of direct references to elements in the application. The use of such links contained in the package «java.lang.ref» allows the simple removal of superfluous objects by the garbage collector. 2.3. Code review: Banal, but no less effective in some cases method – checking the code. Sometimes it helps one time to eliminate memory leaks. 2.4. Enabling profiling: By using the above method, you can discover storage areas that can be usefully used to store application resources. 2.5. Verbose garbage collection: Another way mentioned earlier. Including this mode, we can closely monitor the work of GC. Compliance with such rules of storage space and its rational use can help eliminate situations that create Java surges back up programming and Java surges back up language. This is the basis for the normal operation of the system as a whole. 3. Error removing tools: Quest Foglight for Java (for heap monitoring); MAT (for heap dump analysis). As the expert on working with Java reports, on the portal – a quote: Habr “ After finding and fixing the leak, it will not be superfluous to go through all the steps again, analyze the memory, and report to the Memory Analyzer to make sure the fix helped.“ Conclusion So, analyzing the question – what is a memory leak in Java, it becomes clear that it affects the system as a disease that strikes it and worsens its overall condition. Without treatment, it can lead to irreparable consequences. And since this is a very serious problem, though not visible at first sight – it needs to be identified and addressed in a timely manner. The problem is not easily remedied if it is detected long after the first signs of its existence have emerged. There is no «magic pill» that will fix the leak once, even when involving the work of a highly qualified specialist in Java programming. But if working with the app consistently involved the use of proven methods, profiling, tracking, Java memory management, and code verification – the appearance of such problems can be reduced to zero. And have You ever encountered memory leaks in Java? Share your experience of identifying the type of problem and Your method of solving it in the comments! Also published . here

How to Detect and Avoid Memory Leak in Java

What Really Influences A Product Managers’ Job Satisfaction [Deep Dive]

Lauren has maybe half of the skills listed within an attractive job posting. But she’s not going to let that stop her from talking her way into a 6-figure annual salary. Believe it or not, that little story is absolutely true. Lauren’s gift of gab bought her a position she was very underqualified for. It’s time to end these costly, totally unnecessary shenanigans. is blending AI with blockchain tech to bring some much-needed honesty and transparency to any company’s hiring process. TalentSnap So, what is TalentSnap and what problems does the project want to solve? I’ll tell you in a bit. But first, go ahead and read the next section. It won’t be as painful as filling out paperwork during onboarding for a new job but it’s still kinda important… Disclaimer : This is not investment or financial advice. I’m not a financial expert by any stretch of the imagination. Most information within this article is speculative and merely my own personal opinion. Always conduct your own research before contributing to any startup projects. Always remember that what you do with your money is your decision to make. And if that decision proves too difficult for you alone, seek guidance from a financial professional. I may or may not receive a small allocation of tokens for creating this content. That said, I will do my best to remain unbiased and fair. I try to avoid all FOMO and FUD and never wish to impart those emotions on my fellow crypto-heads. Questionable Practices If Lauren had been upfront about her actual skills, she never would’ve made it to the 2nd round of interviews. Maybe she figured she’d just learn the role as she went along. And maybe, in Laurens eyes, at least, that’s just fine. But I guarantee her employer views the situation differently. If you’re hiring for a $100K+ job, you’re looking for top talent, not a glorified intern. Let’s face it, if you want a new gig, you may be willing to bend the truth a bit. But if you spin those tales a little too long you can get yourself in trouble. really “Wait a minute. Are you saying that Lauren isn’t really a certified Level 18 Excel Sorceress? I have her resume right here. She lists her Excel skills as ‘otherworldly’ and ‘astonishing.’ But you’re telling me otherwise? Let’s go have a little chat with her…” Uh oh, Lauren, your days at the company may be numbered… Everyone’s entitled to a little break after doing some work. After all that reading, let’s take one now. Here’s a superquick video that’ll get you up to speed with the TalentSnap project’s aspirations: Sorry, Lauren, you’ll have to apply offchain It’s way too easy for deception to creep into the hiring process. Similar to dating, people being interviewed tend to project an image of themselves that’s not exactly reality. And once an employee has inked a deal, it may be very difficult and costly to get rid of them. This is especially true in certain European countries where the laws are heavier on the employee’s side of the scale. People can get all up in your business — like a wedgie — and become very difficult to dislodge. As a hiring manager, you want to make sure you’re taking a risk on the right candidate. Sadly, stories like Lauren’s certainly aren’t isolated. I have another one for you. And, once again, it’s true. To protect the identity of the guys who have absolutely no idea they’re getting thrown under a digital bus, I’ll change their names just a little… Mad Fraudz Once upon a time, I worked for a consumer electronics company that eventually went bankrupt due to mismanagement. The organization’s name? Well, their IP was recently purchased on the cheap and they’re now back in business under new ownership. My lips are sealed. But I will drop some hints for you. Their name rhymes with sad hats. They earned themselves a derogatory entry on urbandictionary.com. Nearly everything they created was named after a nonsensical acronym. And, they were the inspiration for from CollegeHumor. this video You see, the root of the major hiring snafu I’m about to explain was the boss, himself a total fraud. The King of Narcissism himself — Ronny — claimed to be an ‘advertising man’ with experience at Coca-Cola. But 99% of his subordinates believed his experience there was actually in the mailroom rather than the boardroom. Not sure who’s bright idea it was to give this head of the department a say in the hiring process for as well as , but there you go. Fraud begets fraud. severely tech-challenged Creative IT web development We already had a Jr. developer — Al — working for us but since we were expanding our multi-brand web presence, we began looking for a Sr. dev. Along came Bessie Bueno. The buzzwords must have been flying during the interview because Bessie was practically worthless. I sat within view of Bessie’s monitor and a typical day was 5% work and 95% YouTube videos. And when Bessie got stuck on a project — unable to conquer a task that should be a snap for a Sr. developer — who do you think Bessie would turn to? That’s right, Al, the Jr. developer. I really thought Al was going to quit the day he discovered Bessie’s actual job title. I could see the rage of injustice and disbelief in his eyes. Someone who’s making a solid 25% more than you shouldn’t need to visit your desk asking questions about things they should’ve known before ever getting hired! Now, this ridiculous and totally inappropriate work scenario could’ve been prevented if Bessie had to possession of job-specific skills. Had Ronny been able to easily check, he may have gone with a different candidate. prove Verifiable skills are at the heart of the TalentSnap project. And the team is combining AI with blockchain to make the verification process quick and easy. TalentSnap’s MVP is currently up and running Ensuring a Good Fit Hiring metrics go far beyond relevant skills and experience. Every workspace has a distinct culture and if you won’t be comfortable somewhere you’d best keep the job hunt going. Your personality type should ideally match your role as well as the environment you work within. If you don’t play nicely with others, you’ll have a hard time fitting into any organization. Which makes personal interviews so crucial. They’re not always in-person but you must spend time talking to someone — on Skype or a similar service, at a minimum — getting a feel for their personality before giving them a keycard to the building. But before you ask a candidate to visit your office — after chatting with them on the phone to verify they’re not a complete freakshow— there’s a heap of data for you to wade through. hopefully And that’s where TalentSnap comes in. Before you reach out to a job seeker, you’ll have a wealth of role-relevant info at your fingertips. The team has created an app — now live with a full dashboard and Chrome extension — to trim the fat and smarten your hiring efforts. View a candidate’s details & LinkedIn profile, email them, add them to your favorites, or leave yourself a note FitScore Now we’re getting to the interesting part. Plenty of hiring sites have various filters like education, skills, and years of experience. But few if any of them take personality type into consideration. And none of those sites can claim AI or machine learning as a skill on its resume. The TalentSnap service analyzes dozens of hiring metrics to create a unique FitScore — ranging from 1 to 100 — used for linking employers to job hunters. And as candidates and employers create data within the TalentSnap platform, the service will only get better with its matchmaking. You see, AI needs information in order to evolve. And with 100s of millions of datapoints generated by the global job market every day, TalentSnap’s algorithms have no shortage of food for thought. You can test the right now if you’d like. Just make sure you come back and finish what you started here. FitScore MVP Hiring Automation The TalentSnap project will evolve over time. The goal is to create a platform that becomes a global benchmark for the hiring process. Elevating the FitScore measurement to industry-standard status will require the team to branch-out their Research & Development efforts. Here are a few subjects the project plans to delve into… Federated Learning Don’t worry, you won’t find any evil characters here. Star Wars Federated learning is a setting within the realm of machine learning. The aim is to train a centralized model using training data spread among a large, distributed userbase. In TalentSnap’s case, they’ll use this method to smarten the platform’s AI. As users generate data on the network, it’s sent to a smart contract. And every time a contract makes computations, it gets a little better at interpreting incoming data. Which reinforces my point above: Every FitScore generated by the platform will be a little smarter than the last. Homomorphic Encryption This technique is how the platform will process encrypted data without decrypting it first. Homomorphic encryption will allow everyone on the network to safely share their personal data. Users of the TalentSnap service won’t need to subject their data to the prying eyes of info-harvesting companies aiming to turn a profit from what’s not rightfully theirs. It’s important to note that homomorphic encryption and federated learning aren’t scheduled for implementation until 2020. Proof of Identity Are you who you say you are? The blockchain knows. In yet another effort to secure the privacy of users’ identities, TalentSnap plans to integrate existing blockchain tech into their upcoming platform. Civic, for example, is a blockchain project that allows users to create a digital ID that they can use all throughout the cryptosphere. BTW, if you’re active with ICOs and tired of KYC, give Civic a try. The service is convenient and easy to use. Proof of Identity will assure employers that applicants are legitimate candidates with real profiles on the platform. TSC Utility Token TalentSnap’s native token — TSC — is an ERC20. The team has stated that should the Ethereum network fail to solve its current scalability issues, they may jump ship to another blockchain or build one of their own. The TSC token’s main purpose is as a payment mechanism within the platform. TSC tokens are deducted from a user’s balance every time a FitScore is generated. Good news for candidates — there’s no deduction unless the FitScore match is 90 and above. In addition, candidates and employers can be compensated with TSC tokens should they choose to share their data with third-party companies connected to the TalentSnap platform. Final thoughts… Automating the hiring process is a long road. Take a look at the TalentSnap project’s roadmap and you’ll see a milestone that’s much farther away than what most projects display. 2035 — the team’s projected target for hiring — may seem like it’s light-years away. And way back in 2000, waiting for 2020 seemed like an eternity, too. Yet it’s right around the corner. autonomous I think a long-term approach shows the team is grounded in reality. I prefer projects to take their time and do things right as opposed to rushing to hit self-imposed milestones. Although the AI industry is growing rapidly, there’s still of progress that needs to occur before we start seeing the technology integrated into our everyday lives. a lot Keep the project’s plan for slow and steady growth in mind if you plan to contribute. The project’s tech is 100% reliant upon data. They’ll need terabyte after terabyte before AI becomes smart enough to improve entire industries. As TalentSnap begins to accumulate massive amounts of data to feed the platform’s info-hungry algorithms, their service will improve. And tokenizing the data we already share — for free — with social hiring platforms is making the most of what blockchain tech has to offer. Simplification of the hiring process also brings cost- and time-savings for all parties. As you know, finding a job or a strong candidate can be a long and tedious game of waiting. Companies can cut right through the smoke and mirrors and only deal with people who truly seem like a good fit. And candidates will become empowered to apply for positions that closely match their skill set. Once the platform is fully operational and FitScores become highly reliable indicators of a match between employers and candidates, companies will realize an uptick in the quality of new hires. When that day arrives — hopefully within the next couple of years — companies will be able to keep the likes of Lauren and Ronny and Bessie out of their organizations. And then, everyone except the fraudsters will win the hiring game. DYOR It’s always a good idea to form your own opinion of any project you may want to participate in. Here’s where you can dive a little deeper into TalentSnap: TalentSnap Website TalentSnap Whitepaper Get Social Bitcointalk Facebook LinkedIn Reddit Telegram Twitter YouTube By the way… If you like what you see here and need help getting your crypto message to the masses, I’d love to hear about your project. Feel free to reach out to me here: blockchainauthor at Gmail

TalentSnap ICO Review: How Blockchain Tech Safely Automates the Hiring Process

One common question I get asked is: “ Should I learn to code if I want to be a tech entrepreneur?” Having been a software engineer who has built and scale products from nothing, I’ve found it immensely useful to know how to code as I transitioned to being a tech entrepreneur myself for the following reasons: Having conducted 100+ interviews of technical talent from interns to CTOs, I am capable of conducting a phone screen, and technical interview to gauge people’s level accurately. First, it helps me to vet talent. and identify when people are wasting time. Second, I can call BS on complicated solutions, I can also empathize when things take longer or it’s hard to find a solution. Third, while I can call BS, like taking the time to test the an application, paying down tech debt, or setting up an onboarding process for additional hires. Fourth, people find it helpful that they don’t need to explain why doing something is valuable once people know that I am capable of building. Fifth, funnily enough there is a level of respect I get Is it ever too late to learn how to code? If you didn’t make the choice to go to a 4-year college and major in engineering, it’s not too late! And with the growing number of coding bootcamps and online courses, you can develop a working proficiency within 3–6 months. Note that it doesn’t mean you’ll instantly be hired by the likes of Google or Facebook, but you can start to put together a working prototype. One of my previous students did just that! If you’re thinking about learning how to code, here are some other things you need to think about before you make a decision: What is the tradeoff for learning to code? It can take you anywhere from 3 months and few hundred dollars to $25,000+ to learn to code. Once that’s done you can start to build your prototype, but could you have instead invested those three months into finding someone to build it for you? And are there other skills you could be honing like in sales and marketing? After all, the competition in SaaS products is rising, and the real challenge going forward is attracting and retaining customers to build a sustainable business or a unicorn, should you choose to. Hence, tech entrepreneurs are being put to the test when it comes to customer acquisition more so than the complexity of building a product. Learning to code isn’t enough. Building a prototype is a necessary but insufficient condition. In the early days of your startup, you also need to understand how to manage a product, which includes: understanding customer needs and being able to translate them into product features and benefits being able to make business decisions for building one feature versus another setting up metrics and analytics to measuring the usage of features being able to communicate priorities to other teammates managing the needs of other people on your team such as designers and engineers putting a process in place and having it evolve as your team and your product scale up and down If your goal is to become a tech entrepreneur, then you will be the first product manager, because you are the one with the domain expertise. You understand the customer, the market, the competition, and what needs to get built and why. While some software engineers are capable of managing product, it’s a lot to ask, and many prefer to stay focused and build. Even if you hire someone to be a product manager, you need to be able to communicate the business reasons behind why someone should or shouldn’t be built, until they can develop an understanding of the problem space for themselves and an intuition around making decisions. What you maybe looking for is literacy. Having a high-level understanding of how code works, how systems like servers, databases, and mobile devices interact, jargon like APIs, new technology frameworks and architecture like Docker, and the process behind creating a product is a worthwhile goal to have. Your depth of understanding depends on the type of product you are building. If your goal is to build infrastructure products like , then you better having working knowledge, and team up with a CTO who knows the ins and outs of the technology! Twilio But if your goal is to develop a more business or consumer oriented product, then having a high level understanding is fine. Of course, as an outsider, it’s hard to gauge the technical depth of a product. So at the end of the day, it would be prudent to enlist a technical advisor early on to help you talk through some of these issues as they arise. If you have never built a software product before, then I highly recommend taking the time to attend hackathons, , or even better joining someone else’s startup. Even as a software engineer, I found it immensely helpful to be on the ground floor of a startup. working on a side project to learn the process Sure you can read books like , , , and of course, my personal favorite ;) But nothing beats hands on learning! Sprint Lean Startup Mental Models How To Transform Your Ideas Into Software Products Taking 1–2 years or more can be a valuable experience, and if you’re still not convinced, check out this series I put together on . Why You Should Consider Being An Early Stage Startup Employee This post might have brought up, even more, questions like: how do you find and vet a technical advisor, how can you contribute to an early stage startup if you don’t know how to code, and more. Don’t worry I’ll tackle those in future posts! Finally, knowing to code isn’t a must. , and . Some people opt out of it altogether knowing to code doesn’t guarantee success in tech Now I want to know if you’re a tech entrepreneur, did you decide to go for it and learn to code or not? How did you make the decision? Let me know in the comments below! Enjoyed this post and want more? Help others enjoy it too by hitting the ♡ below! And check out additional videos I’ve created related to this topic: How Non-Technical Founders Can Bring A Product To Market How To Change Careers Later In Life And Transition Into A Technical

Should I learn to code if I want to be a tech entrepreneur?

This brings us to the end of this trilogy series focusing on performance evaluation metrics for conversation assistants. Part 1 - Metrics: Swallow the Red Pill
Part 2 - Metrics Reloaded: The Oracle
Part 3 - The Metrics Revolution: Scaling What we have discussed so far? I would highly encourage reading the first 2 articles of this series to get a more detailed understanding. User Perceived Metrics and User Reported Metrics form the foundation for a highly reliable conversational assistant.
Reliability and Latency are two different aspects to evaluate end user performance for conversational agents.
Never fully trust a metric - always question negative and positive movement in a metric.
There are 2 metric collection pipelines

Ground truth metric measure performance end user metrics in a manual and semi-automated setting. This has high confidence but lower coverage.
Production metric measure performance end user metrics from real users in production. This has low confidence but higher coverage.


Metric collection from production should not affect degrading performance.

Logs should be harmless and simple.
Metrics derivation and logs should be orthogonal. Scaling to multiple surfaces This article will be focussing on how do we scale the metric instrumentation framework to accommodate new user cases across different form factors - phone, tablet, headsets, car etc. Support different product features It is important to note that different form factors may have different product features. However, for conversational assistants, the high level mental model should still be the same. How long does it take for assistant to start up?
How long does it take for the AI agent to respond?
How long does it take for the AI agent to complete a task end to end? As a result, there is a need to enable metric collection from different log types along with customizations needed for additional form factor specific features. Seamless Extension The metric collection framework should work out of the box for new form factors without needing any specific change on metric collection framework. This is essential to ensure that the metrics required for determining success for the conversational AI agent are available from early stage and can be used to plan the future roadmap for next milestones. This also removes additional engineering bandwidth and ensures that the metrics are measured in an objective manner. The solution I will be discussing a simple solution that can work for most use cases - transform the device specific logs to an uniform representation. I am also the first author patent holder in this area - Standardizing analysis metrics across multiple devices. I will not be discussing low level details of the solution but the high level details. Implementations relate to generating standardized metrics from device specific metrics that are generated during an interaction between a user and an automated assistant. The solution consists of the following steps: Identify the metrics that are agnostic of the form factor - these are core metrics for the conversational AI agent in question.
Identify the logging signals needed for instrumenting these metrics.
Adding mapping configuration for the corresponding logging signals on each form factor.
Transform the form factor specific logging signals to an uniform space which is agnostic of the form factor.
The metric instrumentation framework will only be based on the uniform logging signals. Key Benefits Metric instrumentation framework and form factor based logging signals are decoupled from each other.
It makes it easy to backfill or correct issues with metrics easily without needing changes on the specific form factor.
Provides fast scaling to newer surfaces or upgraded tech stacks without affect the core metrics for the conversational agent. Word of caution It is important to note that “ground truth” metrics should continue to be based on device specific logs. This can help uncover issues with the metric instrumentation framework. “Ground truth” metrics should always reflect the metrics with high confidence. It is crucial to remember to never fully trust metrics - “swallow the red pill”. Conclusion As AI applications integrate more into our daily lives, it is crucial to ensure that these AI agents are reliable for the end user. AI agents should be useful and usable. As a result, end user performance evaluation infrastructure provides a foundational backbone for the development of conversational agents. I hope this trilogy series provides a good starting point to understand and instrument these metrics at high level. This brings us to the end of this trilogy series focusing on performance evaluation metrics for conversation assistants. Part 1 - Metrics: Swallow the Red Pill Part 2 - Metrics Reloaded: The Oracle Part 3 - The Metrics Revolution: Scaling Part 1 - Metrics: Swallow the Red Pill Metrics: Swallow the Red Pill Part 2 - Metrics Reloaded: The Oracle Metrics Reloaded: The Oracle Part 3 - The Metrics Revolution: Scaling The Metrics Revolution: Scaling What we have discussed so far? I would highly encourage reading the first 2 articles of this series to get a more detailed understanding. User Perceived Metrics and User Reported Metrics form the foundation for a highly reliable conversational assistant. Reliability and Latency are two different aspects to evaluate end user performance for conversational agents. Never fully trust a metric - always question negative and positive movement in a metric. There are 2 metric collection pipelines

Ground truth metric measure performance end user metrics in a manual and semi-automated setting. This has high confidence but lower coverage.
Production metric measure performance end user metrics from real users in production. This has low confidence but higher coverage. Metric collection from production should not affect degrading performance.

Logs should be harmless and simple.
Metrics derivation and logs should be orthogonal. User Perceived Metrics and User Reported Metrics form the foundation for a highly reliable conversational assistant. User Perceived Metrics User Perceived Metrics User Reported Metrics User Reported Metrics Reliability and Latency are two different aspects to evaluate end user performance for conversational agents. Reliability and Latency Reliability and Latency Never fully trust a metric - always question negative and positive movement in a metric. Never fully trust a metric Never fully trust a metric There are 2 metric collection pipelines Ground truth metric measure performance end user metrics in a manual and semi-automated setting. This has high confidence but lower coverage.
Production metric measure performance end user metrics from real users in production. This has low confidence but higher coverage. Ground truth metric measure performance end user metrics in a manual and semi-automated setting. This has high confidence but lower coverage. Production metric measure performance end user metrics from real users in production. This has low confidence but higher coverage. Ground truth metric measure performance end user metrics in a manual and semi-automated setting. This has high confidence but lower coverage. Ground truth metric This has high confidence but lower coverage. Production metric measure performance end user metrics from real users in production. This has low confidence but higher coverage. Production metric This has low confidence but higher coverage. Metric collection from production should not affect degrading performance. Logs should be harmless and simple.
Metrics derivation and logs should be orthogonal. Metric collection from production Logs should be harmless and simple. Metrics derivation and logs should be orthogonal. Logs should be harmless and simple. Metrics derivation and logs should be orthogonal. Scaling to multiple surfaces This article will be focussing on how do we scale the metric instrumentation framework to accommodate new user cases across different form factors - phone, tablet, headsets, car etc. Support different product features It is important to note that different form factors may have different product features. However, for conversational assistants, the high level mental model should still be the same. How long does it take for assistant to start up? How long does it take for the AI agent to respond? How long does it take for the AI agent to complete a task end to end? How long does it take for assistant to start up? How long does it take for the AI agent to respond? How long does it take for the AI agent to complete a task end to end? As a result, there is a need to enable metric collection from different log types along with customizations needed for additional form factor specific features. Seamless Extension The metric collection framework should work out of the box for new form factors without needing any specific change on metric collection framework. This is essential to ensure that the metrics required for determining success for the conversational AI agent are available from early stage and can be used to plan the future roadmap for next milestones. This also removes additional engineering bandwidth and ensures that the metrics are measured in an objective manner. The solution I will be discussing a simple solution that can work for most use cases - transform the device specific logs to an uniform representation. I am also the first author patent holder in this area - Standardizing analysis metrics across multiple devices . I will not be discussing low level details of the solution but the high level details. Standardizing analysis metrics across multiple devices Implementations relate to generating standardized metrics from device specific metrics that are generated during an interaction between a user and an automated assistant. Implementations relate to generating standardized metrics from device specific metrics that are generated during an interaction between a user and an automated assistant. The solution consists of the following steps: Identify the metrics that are agnostic of the form factor - these are core metrics for the conversational AI agent in question. Identify the logging signals needed for instrumenting these metrics. Adding mapping configuration for the corresponding logging signals on each form factor. Transform the form factor specific logging signals to an uniform space which is agnostic of the form factor. The metric instrumentation framework will only be based on the uniform logging signals. Identify the metrics that are agnostic of the form factor - these are core metrics for the conversational AI agent in question. Identify the logging signals needed for instrumenting these metrics. Adding mapping configuration for the corresponding logging signals on each form factor. Transform the form factor specific logging signals to an uniform space which is agnostic of the form factor. The metric instrumentation framework will only be based on the uniform logging signals. Key Benefits Metric instrumentation framework and form factor based logging signals are decoupled from each other. It makes it easy to backfill or correct issues with metrics easily without needing changes on the specific form factor. Provides fast scaling to newer surfaces or upgraded tech stacks without affect the core metrics for the conversational agent. Metric instrumentation framework and form factor based logging signals are decoupled from each other. It makes it easy to backfill or correct issues with metrics easily without needing changes on the specific form factor. Provides fast scaling to newer surfaces or upgraded tech stacks without affect the core metrics for the conversational agent. Word of caution It is important to note that “ground truth” metrics should continue to be based on device specific logs. This can help uncover issues with the metric instrumentation framework. “ground truth” “Ground truth” metrics should always reflect the metrics with high confidence. It is crucial to remember to never fully trust metrics - “swallow the red pill”. Conclusion As AI applications integrate more into our daily lives, it is crucial to ensure that these AI agents are reliable for the end user. AI agents should be useful and usable. As a result, end user performance evaluation infrastructure provides a foundational backbone for the development of conversational agents. I hope this trilogy series provides a good starting point to understand and instrument these metrics at high level.

The Metrics Revolution: Scaling

This is the second part of the trilogy focusing on performance evaluation infrastructure for conversational AI agents. Part 1 - Metrics: Swallow the Red Pill
Part 2 - Metrics Reloaded: The Oracle
Part 3 - The Metrics Revolution: Scaling Quick Recap In case you have not read the first part of this trilogy, we focused on some high level questions of what and why about metrics needed to ensure end user performance for an AI agent. Here is a summary: User Perceived Metrics and User Reported Metrics form the foundation for a highly reliable conversational assistant.


Reliability and Latency are two different aspects to evaluate end user performance for conversational agents.


Never fully trust a metric - always question negative and positive movement in a metric.


It is crucial to identify these metrics from early development stage to objectively measure the performance of an AI agent. In this article, we will focus on how to obtain these metrics in a manner that is easy to use, accurate and actionable. The Oracle We hope there was a “oracle” which could provide highly accurate end user performance metrics. Guess what? There is! We as humans are extremely good at doing this. However this comes as a cost of efficiency. The amount of data points are less but highly confident. We will discuss some ways to mitigate this gap. Human Testing: This can be the QA team or developers or PMs or executives testing the product. It is important to realize the value for this data - is truly the “golden dataset” for metrics. This also shows that there is no replacement for testing your product manually.


Automated Lab Testing: There can be test benches setup where a different software analyzes automated testing via audio and visual processing. This helps us increase coverage for the “golden dataset” metrics. There should be infrastructure in place to analyze these data points and referred to as the “ground truth”. The Next Best Thing While ground truth provides highly confident set of metrics, it cannot replicate all the conditions that might be occur for a real user in production. Note that this might have lower confidence but is needed to uncover unanticipated issues. Hence, there is a need to invest in ensuring we have logging signals in place for production metrics. There are few things to be cautious of for production logging. Privacy and security implications must be kept in mind This is pretty much self-explanatory but all production logs should be aggregated and anonymized to protect against any privacy and security issues. Log only what is needed On the second point, it is important to note that success metrics need to be decided beforehand and then the corresponding logs need to be identified from that. It is easy to just add a lot of logging and then determine the metrics we can derive. This actually ends up polluting the metrics making it less objective and only ends up measuring the metrics we can based on available logs. The other drawback for logging a lot is that it can end up affecting user behavior, causing performance degradation. Keep Logging simple It is important to note that while metrics should drive what logs are needed, logs should not try to derive metrics. In other words, logging should just capture what event happened and when. The derivation should happen on the metric processing side and not during the user interaction. This helps to keep logging and metric derivation orthogonal - single responsibility :) Keeping logging simple also helps with reducing unnecessary complexity in production code which can result in unexpected errors and lead to performance degradation. Putting everything together The two types of metric collection processes we discussed here are complementary to each other. Production metrics have high coverage but lower confidence.


Ground truth metrics have low coverage but high confidence. As a result, both of these should be used together to create a highly reliable and fast conversational agent. For any new changes (user facing or not), both the metric systems should be used to verify if there is any degradation in end user performance. The metric collection processes should aim to improve each other: Production metrics should be validated against “ground truth” metrics. A lot of deviation suggests issues with production logging or metric instrumentation pipeline. Issues captured by production metrics that have not been flagged by “ground truth” metrics reveal gaps in human or semi-automated metric pipeline. These can be used to add new manual or semi-automated tests to improve the coverage for “ground truth” metrics. High Level Takeaways Ground truth and production metrics complement each others for ensuring performance.

These two metric pipeline should be used to improve each other.



Metric collection from production should not affect degrading performance.

Metrics should dictate what logs needed to be added.
Logs should be harmless and simple.
Metrics derivation and logs should be orthogonal. What’s Next? As our conversation AI agent scales across different devices - phones, tablets, cars, earphones - it is becomes vital that the performance evaluation infrastructure can also scale seamlessly. We discuss more details on scaling performance metrics infrastructure in the final part of the trilogy - stay tuned! This is the second part of the trilogy focusing on performance evaluation infrastructure for conversational AI agents. Part 1 - Metrics: Swallow the Red Pill Part 2 - Metrics Reloaded: The Oracle Part 3 - The Metrics Revolution: Scaling Part 1 - Metrics: Swallow the Red Pill Metrics: Swallow the Red Pill Part 2 - Metrics Reloaded: The Oracle Metrics Reloaded: The Oracle Part 3 - The Metrics Revolution: Scaling The Metrics Revolution: Scaling Quick Recap In case you have not read the first part of this trilogy, we focused on some high level questions of what and why about metrics needed to ensure end user performance for an AI agent. what and why what and why Here is a summary: User Perceived Metrics and User Reported Metrics form the foundation for a highly reliable conversational assistant. Reliability and Latency are two different aspects to evaluate end user performance for conversational agents. Never fully trust a metric - always question negative and positive movement in a metric. It is crucial to identify these metrics from early development stage to objectively measure the performance of an AI agent. User Perceived Metrics and User Reported Metrics form the foundation for a highly reliable conversational assistant. User Perceived Metrics and User Reported Metrics form the foundation for a highly reliable conversational assistant. User Perceived Metrics User Perceived Metrics User Reported Metrics User Reported Metrics Reliability and Latency are two different aspects to evaluate end user performance for conversational agents. Reliability and Latency are two different aspects to evaluate end user performance for conversational agents. Reliability and Latency Reliability and Latency Never fully trust a metric - always question negative and positive movement in a metric. Never fully trust a metric - always question negative and positive movement in a metric. Never fully trust a metric Never fully trust a metric It is crucial to identify these metrics from early development stage to objectively measure the performance of an AI agent. It is crucial to identify these metrics from early development stage to objectively measure the performance of an AI agent. In this article, we will focus on how to obtain these metrics in a manner that is easy to use, accurate and actionable. how to obtain these metrics in a manner that is easy to use, accurate and actionable. how to obtain these metrics in a manner that is easy to use, accurate and actionable. The Oracle We hope there was a “oracle” which could provide highly accurate end user performance metrics. Guess what? There is! We as humans are extremely good at doing this. However this comes as a cost of efficiency . The amount of data points are less but highly confident . We will discuss some ways to mitigate this gap. a cost of efficiency data points are less but highly confident data points are less but highly confident Human Testing: This can be the QA team or developers or PMs or executives testing the product. It is important to realize the value for this data - is truly the “golden dataset” for metrics. This also shows that there is no replacement for testing your product manually. Automated Lab Testing: There can be test benches setup where a different software analyzes automated testing via audio and visual processing. This helps us increase coverage for the “golden dataset” metrics. Human Testing: This can be the QA team or developers or PMs or executives testing the product. It is important to realize the value for this data - is truly the “golden dataset” for metrics. This also shows that there is no replacement for testing your product manually. Human Testing: This can be the QA team or developers or PMs or executives testing the product. It is important to realize the value for this data - is truly the “golden dataset” for metrics. This also shows that there is no replacement for testing your product manually. Human Testing: Automated Lab Testing: There can be test benches setup where a different software analyzes automated testing via audio and visual processing. This helps us increase coverage for the “golden dataset” metrics. Automated Lab Testing: There can be test benches setup where a different software analyzes automated testing via audio and visual processing. This helps us increase coverage for the “golden dataset” metrics. Automated Lab Testing: There should be infrastructure in place to analyze these data points and referred to as the “ground truth”. There should be infrastructure in place to analyze these data points and referred to as the “ground truth”. There should be infrastructure in place to analyze these data points and referred to as the “ground truth”. “ground truth”. The Next Best Thing While ground truth provides highly confident set of metrics, it cannot replicate all the conditions that might be occur for a real user in production. Note that this might have lower confidence but is needed to uncover unanticipated issues. Hence, there is a need to invest in ensuring we have logging signals in place for production metrics. There are few things to be cautious of for production logging. Privacy and security implications must be kept in mind This is pretty much self-explanatory but all production logs should be aggregated and anonymized to protect against any privacy and security issues. Log only what is needed On the second point, it is important to note that success metrics need to be decided beforehand and then the corresponding logs need to be identified from that . It is easy to just add a lot of logging and then determine the metrics we can derive. This actually ends up polluting the metrics making it less objective and only ends up measuring the metrics we can based on available logs. metrics need to be decided beforehand and then the corresponding logs need to be identified from that The other drawback for logging a lot is that it can end up affecting user behavior, causing performance degradation. Keep Logging simple It is important to note that while metrics should drive what logs are needed, logs should not try to derive metrics. In other words, logging should just capture what event happened and when. The derivation should happen on the metric processing side and not during the user interaction. This helps to keep logging and metric derivation orthogonal - single responsibility :) logs should not try to derive metrics. This helps to keep logging and metric derivation orthogonal - single responsibility Keeping logging simple also helps with reducing unnecessary complexity in production code which can result in unexpected errors and lead to performance degradation. Putting everything together The two types of metric collection processes we discussed here are complementary to each other. Production metrics have high coverage but lower confidence. Ground truth metrics have low coverage but high confidence. Production metrics have high coverage but lower confidence. Production metrics have high coverage but lower confidence. Ground truth metrics have low coverage but high confidence. Ground truth metrics have low coverage but high confidence. As a result, both of these should be used together to create a highly reliable and fast conversational agent. For any new changes (user facing or not), both the metric systems should be used to verify if there is any degradation in end user performance. The metric collection processes should aim to improve each other: Production metrics should be validated against “ground truth” metrics. A lot of deviation suggests issues with production logging or metric instrumentation pipeline. Production metrics should be validated against “ground truth” metrics. A lot of deviation suggests issues with production logging or metric instrumentation pipeline. “ground truth” “ground truth” Issues captured by production metrics that have not been flagged by “ground truth” metrics reveal gaps in human or semi-automated metric pipeline. These can be used to add new manual or semi-automated tests to improve the coverage for “ground truth” metrics. Issues captured by production metrics that have not been flagged by “ground truth” metrics reveal gaps in human or semi-automated metric pipeline. These can be used to add new manual or semi-automated tests to improve the coverage for “ground truth” metrics. “ground truth” “ground truth” High Level Takeaways Ground truth and production metrics complement each others for ensuring performance.

These two metric pipeline should be used to improve each other. Metric collection from production should not affect degrading performance.

Metrics should dictate what logs needed to be added.
Logs should be harmless and simple.
Metrics derivation and logs should be orthogonal. Ground truth and production metrics complement each others for ensuring performance. These two metric pipeline should be used to improve each other. Ground truth and production metrics complement each others for ensuring performance. These two metric pipeline should be used to improve each other. These two metric pipeline should be used to improve each other. Metric collection from production should not affect degrading performance. Metrics should dictate what logs needed to be added.
Logs should be harmless and simple.
Metrics derivation and logs should be orthogonal. Metric collection from production should not affect degrading performance. Metrics should dictate what logs needed to be added. Logs should be harmless and simple. Metrics derivation and logs should be orthogonal. Metrics should dictate what logs needed to be added. Logs should be harmless and simple. Metrics derivation and logs should be orthogonal. What’s Next? As our conversation AI agent scales across different devices - phones, tablets, cars, earphones - it is becomes vital that the performance evaluation infrastructure can also scale seamlessly. We discuss more details on scaling performance metrics infrastructure in the final part of the trilogy - stay tuned!

Metrics Reloaded: The Oracle

This is the first part of a trilogy, where I will discuss some high level ideas for measuring end user performance for conversational voice assistant. In this article, I will be discussing how to think about metrics ideologically for conversational voice assistant and some pitfalls from a engineer’s point of view. The aim is keep the discussion simplified and fairly high level for everyone working in this domain. Part 1 - Metrics: Swallow the Red Pill
Part 2 - Metrics Reloaded: The Oracle
Part 3 - The Metrics Revolution: Scaling Quick Intro I am a senior engineer and tech lead for the Google Assistant. Here is my linkedin profile in case you want to know more about me. I also hold 2 patents in performance evaluation infrastructure for voice assistants: Standardizing analysis metrics across multiple devices
Device Agnostic Framework to Measure Reliability During User Interactions (patent pending) And yes I am a big fan of the “Matrix” trilogy :) Which metrics matter? Metrics should be classified into top 2 categories User Reported Metrics
User Perceived Metrics Note the word - “user” - is present in both which should be expected. The golden rule for any conversational product or any product in general is “user” should always be given the top priority. Before going into the definition for the above, a user facing metric usually consists of 2 aspects Reliability - Whether the user was able to complete a specific action?
Latency - How long did it take for the user to complete a specific action? Note that high latency can also mean low reliability. No user will wait for 20 seconds for a successful action completion. User Perceived Metrics These metrics which capture reliability and latency of actions that can be directly observed by the user. e.g. How long does it take for the voice assistant to start playing music when the user asks it to?
How many times does the voice assistant actually start playing music when the user asks it to? These metrics are similar to system level metrics and hence have high coverage but comparatively lower confidence (by virtue of being derived). User Reported Metrics These metrics are directly reported by the users either through feedback channels, internal testing, surveys etc. These metrics are explicitly coming from the user and hence have low coverage but comparatively higher confidence. The Red Pill A rule of thumb I follow for metrics is swallow the red pill which means never fully trust a metric - always question negative and positive movement in a metric. The reason I say this is because metrics cannot be measured explicitly most of the time and need to be derived many times from different signals. Example 1: “How many times does the voice assistant actually start playing music when the user asks it to?” We might not be able to capture the true “latency” that the user experiences in this case with a 100% accuracy. But we can observe different components such a long did a specific backend(s) take to respond and use specific heuristics to come up with a metric value which approximates the user perceived latency to the best of our knowledge. This is where system level component metrics come into the picture. Example 2: ”How many times does the voice assistant actually start playing music when the user asks it to” This is difficult to determine based on User Perceived Metrics. The simple rationale is if the voice assistant intelligence is able to detect that user wants to play music - then it would started playing music. So, this metric in reality will always reflect 100% reliability which is incorrect. This is where User Reported Metrics come into the picture. However, since user reported metrics can be detached from the system level components, it is important to connect those with system level metrics or User Perceived Metrics to make it actionable. Conclusion Both User Perceived Metrics and User Reported Metrics complement each other and form the foundation for a highly reliable conversational assistant. It might not be trivial to enable these metrics at a later stage and may require significant bandwidth. As a result, metric evaluation should be given high priority when designing conversational assistant products. This is the only way to objectively evaluate the user experience and may influence some of the design decisions. As a result, it might be worthwhile to understand the metric requirements in early stages of product development. There is a need to invest in infrastructure to ensure these metrics easy to use, accurate and actionable - we will be discussing more details on how to do that in the second part of this trilogy. So stay tuned! This is the first part of a trilogy, where I will discuss some high level ideas for measuring end user performance for conversational voice assistant. In this article, I will be discussing how to think about metrics ideologically for conversational voice assistant and some pitfalls from a engineer’s point of view. The aim is keep the discussion simplified and fairly high level for everyone working in this domain. Part 1 - Metrics: Swallow the Red Pill Part 2 - Metrics Reloaded: The Oracle Part 3 - The Metrics Revolution: Scaling Part 1 - Metrics: Swallow the Red Pill Metrics: Swallow the Red Pill Part 2 - Metrics Reloaded: The Oracle Metrics Reloaded: The Oracle Part 3 - The Metrics Revolution: Scaling The Metrics Revolution: Scaling Quick Intro I am a senior engineer and tech lead for the Google Assistant. Here is my linkedin profile in case you want to know more about me. linkedin profile I also hold 2 patents in performance evaluation infrastructure for voice assistants: Standardizing analysis metrics across multiple devices Device Agnostic Framework to Measure Reliability During User Interactions (patent pending) Standardizing analysis metrics across multiple devices Standardizing analysis metrics across multiple devices Device Agnostic Framework to Measure Reliability During User Interactions (patent pending) And yes I am a big fan of the “Matrix” trilogy :) Which metrics matter? Metrics should be classified into top 2 categories User Reported Metrics User Perceived Metrics User Reported Metrics User Perceived Metrics Note the word - “user” - is present in both which should be expected. The golden rule for any conversational product or any product in general is “user” should always be given the top priority . “user” “user” “user” should always be given the top priority Before going into the definition for the above, a user facing metric usually consists of 2 aspects Reliability - Whether the user was able to complete a specific action? Latency - How long did it take for the user to complete a specific action? Reliability - Whether the user was able to complete a specific action? Reliability - Reliability - Whether the user was able to complete a specific action? Latency - How long did it take for the user to complete a specific action? Latency - Latency - How long did it take for the user to complete a specific action? Note that high latency can also mean low reliability . No user will wait for 20 seconds for a successful action completion. Note that high latency can also mean low reliability Note that high latency can also mean low reliability User Perceived Metrics These metrics which capture reliability and latency of actions that can be directly observed by the user. e.g. How long does it take for the voice assistant to start playing music when the user asks it to? How many times does the voice assistant actually start playing music when the user asks it to? How long does it take for the voice assistant to start playing music when the user asks it to? How many times does the voice assistant actually start playing music when the user asks it to? These metrics are similar to system level metrics and hence have high coverage but comparatively lower confidence (by virtue of being derived). high coverage but comparatively lower confidence User Reported Metrics These metrics are directly reported by the users either through feedback channels, internal testing, surveys etc. These metrics are explicitly coming from the user and hence have low coverage but comparatively higher confidence. low coverage but comparatively higher confidence. The Red Pill A rule of thumb I follow for metrics is swallow the red pill which means never fully trust a metric - always question negative and positive movement in a metric. A rule of thumb I follow for metrics is swallow the red pill which means never fully trust a metric - always question negative and positive movement in a metric. The reason I say this is because metrics cannot be measured explicitly most of the time and need to be derived many times from different signals. Example 1: “How many times does the voice assistant actually start playing music when the user asks it to?” We might not be able to capture the true “latency” that the user experiences in this case with a 100% accuracy. But we can observe different components such a long did a specific backend(s) take to respond and use specific heuristics to come up with a metric value which approximates the user perceived latency to the best of our knowledge. This is where system level component metrics come into the picture. This is where system level component metrics come into the picture. This is where system level component metrics come into the picture. Example 2: ”How many times does the voice assistant actually start playing music when the user asks it to” This is difficult to determine based on User Perceived Metrics . The simple rationale is if the voice assistant intelligence is able to detect that user wants to play music - then it would started playing music. So, this metric in reality will always reflect 100% reliability which is incorrect. This is where User Reported Metrics come into the picture. However, since user reported metrics can be detached from the system level components, it is important to connect those with system level metrics or User Perceived Metrics to make it actionable. User Perceived Metrics User Perceived Metrics User Reported Metrics User Reported Metrics User Perceived Metrics User Perceived Metrics Conclusion Both User Perceived Metrics and User Reported Metrics complement each other and form the foundation for a highly reliable conversational assistant. It might not be trivial to enable these metrics at a later stage and may require significant bandwidth. As a result, metric evaluation should be given high priority when designing conversational assistant products. This is the only way to objectively evaluate the user experience and may influence some of the design decisions. As a result, it might be worthwhile to understand the metric requirements in early stages of product development. User Perceived Metrics User Perceived Metrics User Reported Metrics User Reported Metrics metric evaluation metric evaluation There is a need to invest in infrastructure to ensure these metrics easy to use, accurate and actionable - we will be discussing more details on how to do that in the second part of this trilogy. So stay tuned!

Metrics: Swallow the Red Pill

Authors:
(1) Aya Kherrour, Department of Information Engineering and Computer Science University of Trento;
(2) Marco Robol, Department of Information Engineering and Computer Science University of Trento;
(3) Marco Roveri, Department of Information Engineering and Computer Science University of Trento;
(4) Paolo Giorgini, Department of Information Engineering and Computer Science University of Trento. 3 Experimental environment and performance metrices Heuristic search algorithms play an important role in fields such as robotic pathfinding, as they determine the optimal path given a starting position and a goal position. Grid-based environments are commonly used for representing real-world environment scenarios, where these algorithms can be implemented, such as in autonomous navigation and robotics [6]. In this study, we comprehensively analyze well known heuristic search algorithms, namely D*, D* Lite, LPA*, LRTA*, RTAA*, and ARA* in different grid environments. We use the Euclidean distance heuristic to guide the search of the algorithms and assess the impact of a few grid characteristics, such as the obstacle density and the grid size, on the performance of the algorithms. In our study, we use grid-based environments due to their simplicity, and control ease, in addition to being commonly used in path-planning tasks in the research community. The grids are composed of white cells, representing traversable states, whereas the black ones represent non-traversable obstacles. The agent in our simulation can move in eight directions, with a cost equal to 1 for horizontal and vertical moves, and √ 2 for diagonal movements. We used two types of grid environments: randomly generated grid environments and personalized grid environments. Randomly generated grid-based environments: The grids are characterized by three parameters: grid size (NxN), obstacle density, and the distance between the start and the goal states. To investigate the impact of each parameter on the performance of the search algorithms, we varied each parameter independently while keeping the other parameters constant. The variations included varying the grid size, varying start to goal distance, and varying the obstacle densities. For each grid in a variation, we generated ten random instances of the same grid parameters (e.g., a grid with 0.25 of obstacle density, size 300x300, and 140 of start to goal distance, have ten instances, which were generated randomly). Personalised grid environment: Designed for simulating more specific scenarios. These grids have fixed size (71x31 units) and a fixed position for both the start and goal states, and they were divided into two parts based on their obstacle configuration: • Horizontal wall configuration: For these experiments, we add horizontal walls of half grid width every 10 units of grid length. We added the walls in two orientations: once from left to right, and once from right to left in the newly generated grid. • Horizontal wall length configuration: Here, we add all possible walls that can be placed within the grid length, and each time we generate a new grid we increase all wall lengths by 2 units. We adopted these two distinct environments to provide a thorough analysis, seeking to reveal nuanced insights into the performance of the search algorithm in the presence of two different hindering scenarios. In the first one, obstacles are scattered randomly within the grid, whereas in the second one, the wall-like structures, appear as a mass of connected obstacles. To evaluate the performance of the different search algorithms used in the experiments, we have selected the following metrics: • Path cost: The metric measures the path length or the number of executed actions from the start to the goal state. It indicates the quality of the solution. • Memory consumption: It measures the required amount of memory for the algorithm to find a solution. It is relevant to check the scalability of the algorithm, and it is measured in (KB). • Solving Time: Represent the total time an algorithm takes to find a solution in (ms), measured in milliseconds (ms). We carried out our experiments on 3.30GHz 27 Intel i9 cores, equipped with 250Gb of RAM and running Ubuntu Linux 22.04. Using the following settings: All algorithms were using the Euclidean distance as the heuristic function. For LRTA* and RTAA*, we set the number of expended nodes to 250 for each iteration. The ARA* algorithm was run with a weight of 2.5 for the heuristic. We ran each algorithm 100 times on each grid to account for randomness and to ensure the reliability of the results. This paper is available on arxiv under CC 4.0 license. Authors: (1) Aya Kherrour, Department of Information Engineering and Computer Science University of Trento; (2) Marco Robol, Department of Information Engineering and Computer Science University of Trento; (3) Marco Roveri, Department of Information Engineering and Computer Science University of Trento; (4) Paolo Giorgini, Department of Information Engineering and Computer Science University of Trento. Authors: (1) Aya Kherrour, Department of Information Engineering and Computer Science University of Trento; (2) Marco Robol, Department of Information Engineering and Computer Science University of Trento; (3) Marco Roveri, Department of Information Engineering and Computer Science University of Trento; (4) Paolo Giorgini, Department of Information Engineering and Computer Science University of Trento. 3 Experimental environment and performance metrices Heuristic search algorithms play an important role in fields such as robotic pathfinding, as they determine the optimal path given a starting position and a goal position. Grid-based environments are commonly used for representing real-world environment scenarios, where these algorithms can be implemented, such as in autonomous navigation and robotics [6]. In this study, we comprehensively analyze well known heuristic search algorithms, namely D*, D* Lite, LPA*, LRTA*, RTAA*, and ARA* in different grid environments. We use the Euclidean distance heuristic to guide the search of the algorithms and assess the impact of a few grid characteristics, such as the obstacle density and the grid size, on the performance of the algorithms. In our study, we use grid-based environments due to their simplicity, and control ease, in addition to being commonly used in path-planning tasks in the research community. The grids are composed of white cells, representing traversable states, whereas the black ones represent non-traversable obstacles. The agent in our simulation can move in eight directions, with a cost equal to 1 for horizontal and vertical moves, and √ 2 for diagonal movements. We used two types of grid environments: randomly generated grid environments and personalized grid environments. Randomly generated grid-based environments: The grids are characterized by three parameters: Randomly generated grid-based environments: grid size (NxN), obstacle density, and the distance between the start and the goal states. To investigate the impact of each parameter on the performance of the search algorithms, we varied each parameter independently while keeping the other parameters constant. The variations included varying the grid size, varying start to goal distance, and varying the obstacle densities. For each grid in a variation, we generated ten random instances of the same grid parameters (e.g., a grid with 0.25 of obstacle density, size 300x300, and 140 of start to goal distance, have ten instances, which were generated randomly). Personalised grid environment: Designed for simulating more specific scenarios. These grids have fixed size (71x31 units) and a fixed position for both the start and goal states, and they were divided into two parts based on their obstacle configuration: Personalised grid environment: • Horizontal wall configuration: For these experiments, we add horizontal walls of half grid width every 10 units of grid length. We added the walls in two orientations: once from left to right, and once from right to left in the newly generated grid. Horizontal wall configuration: • Horizontal wall length configuration: Here, we add all possible walls that can be placed within Horizontal wall length configuration: the grid length, and each time we generate a new grid we increase all wall lengths by 2 units. We adopted these two distinct environments to provide a thorough analysis, seeking to reveal nuanced insights into the performance of the search algorithm in the presence of two different hindering scenarios. In the first one, obstacles are scattered randomly within the grid, whereas in the second one, the wall-like structures, appear as a mass of connected obstacles. To evaluate the performance of the different search algorithms used in the experiments, we have selected the following metrics: • Path cost: The metric measures the path length or the number of executed actions from the start to the goal state. It indicates the quality of the solution. Path cost: • Memory consumption: It measures the required amount of memory for the algorithm to find a solution. It is relevant to check the scalability of the algorithm, and it is measured in (KB). Memory consumption: • Solving Time: Represent the total time an algorithm takes to find a solution in (ms), measured in milliseconds (ms). Solving Time: We carried out our experiments on 3.30GHz 27 Intel i9 cores, equipped with 250Gb of RAM and running Ubuntu Linux 22.04. Using the following settings: All algorithms were using the Euclidean distance as the heuristic function. For LRTA* and RTAA*, we set the number of expended nodes to 250 for each iteration. The ARA* algorithm was run with a weight of 2.5 for the heuristic. We ran each algorithm 100 times on each grid to account for randomness and to ensure the reliability of the results. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Performance Metrics of Heuristic Search Algorithms in Grid Environments

গ্রিড এনভায়রনমেন্টে হিউরিস্টিক সার্চ অ্যালগরিদমের পারফরম্যান্স মেট্রিক্স

Leistungsmetriken heuristischer Suchalgorithmen in Grid-Umgebungen

Métricas de rendimiento de algoritmos de búsqueda heurística en entornos de red

Mesures de performance des algorithmes de recherche heuristiques dans les environnements de grille

ग्रिड वातावरण में ह्यूरिस्टिक खोज एल्गोरिदम के प्रदर्शन मेट्रिक्स

グリッド環境におけるヒューリスティック探索アルゴリズムのパフォーマンス指標

그리드 환경에서 휴리스틱 검색 알고리즘의 성능 지표

Métricas de desempenho de algoritmos de pesquisa heurística em ambientes de grade

Метрики производительности алгоритмов эвристического поиска в грид-средах

Izgara Ortamlarında Sezgisel Arama Algoritmalarının Performans Ölçümleri

Số liệu hiệu suất của thuật toán tìm kiếm heuristic trong môi trường lưới

网格环境中启发式搜索算法的性能指标

Authors:
(1) Aya Kherrour, Department of Information Engineering and Computer Science University of Trento;
(2) Marco Robol, Department of Information Engineering and Computer Science University of Trento;
(3) Marco Roveri, Department of Information Engineering and Computer Science University of Trento;
(4) Paolo Giorgini, Department of Information Engineering and Computer Science University of Trento. The paper presents a comprehensive performance evaluation of some heuristic search algorithms in the context of autonomous systems and robotics. The objective of the study is to evaluate and compare the performance of different search algorithms in different problem settings on the pathfinding domain. Experiments give us insight into the behavior of the evaluated heuristic search algorithms, over the variation of different parameters: domain size, obstacle density, and distance between the start and the goal states. Results are then used to design a selection algorithm that, on the basis of problem characteristics, suggests the best search algorithm to use. 1 Introduction Autonomous agents and robotics have increasingly been used in various domains, such as, industrial applications [16, 18, 8], surveillance [17], and exploration [4]. These systems are designed to autonomously make decisions and execute actions based on their dynamic and unpredictable environments. Under such conditions, systems are required to be as reactive as possible to changes in the environment. Therefore, ensuring good performance is a significant challenge. In the case of planning, selecting the most effective search algorithm becomes crucial to enhance the overall performance of the system. Real-time heuristic search (RTS) (e.g.,[14, 9]) is a state-of-the-art approach for planning while executing, that helps minimize agent reaction time. These algorithms enable agents to make decisions by interleaving planning with execution while considering the evolving environment, which is an essential property in applications such as robotics, and video game agents. Despite the numerous methods proposed in this field [14, 13, 12, 3, 2], a comprehensive understanding of these algorithms remains elusive. Existing studies [1, 13, 10] have primarily focused on testing the performance of the algorithm based on a single parameter (e.g., look-ahead, sensor range). However, the influence of the problem domain characteristics on algorithm performance remains an understudied aspect. Our objective is to investigate the characteristics of the problem that may impact algorithm performances. To achieve this, we first review existing state of art of search algorithms and their applications. Then we design our experiments to evaluate some of the algorithm performances, also defining relevant metrics to use in the evaluation. Later on, experiment results are analyzed to provide a comprehensive understanding of how problem characteristics influence the performance of the different search algorithms. Finally, from the insight gained from our study, we introduce a selection algorithm that helps us to select the appropriate search algorithm. This paper is structured as follows. We begin by reviewing state-of-the-art search algorithms and previous performance evaluation studies. Next, we describe the approach we adopted to evaluate the search algorithms that are subject to our study, define the problem domain and its characteristics, and the performance metrics. In section four, we present our experimental results. Section five introduces our proposed selection algorithm with an execution example. Finally, we derive our conclusion, the limitations of our study, and possible directions for future work. This paper is available on arxiv under CC 4.0 license. Authors: (1) Aya Kherrour, Department of Information Engineering and Computer Science University of Trento; (2) Marco Robol, Department of Information Engineering and Computer Science University of Trento; (3) Marco Roveri, Department of Information Engineering and Computer Science University of Trento; (4) Paolo Giorgini, Department of Information Engineering and Computer Science University of Trento. Authors: (1) Aya Kherrour, Department of Information Engineering and Computer Science University of Trento; (2) Marco Robol, Department of Information Engineering and Computer Science University of Trento; (3) Marco Roveri, Department of Information Engineering and Computer Science University of Trento; (4) Paolo Giorgini, Department of Information Engineering and Computer Science University of Trento. The paper presents a comprehensive performance evaluation of some heuristic search algorithms in the context of autonomous systems and robotics. The objective of the study is to evaluate and compare the performance of different search algorithms in different problem settings on the pathfinding domain. Experiments give us insight into the behavior of the evaluated heuristic search algorithms, over the variation of different parameters: domain size, obstacle density, and distance between the start and the goal states. Results are then used to design a selection algorithm that, on the basis of problem characteristics, suggests the best search algorithm to use. 1 Introduction Autonomous agents and robotics have increasingly been used in various domains, such as, industrial applications [16, 18, 8], surveillance [17], and exploration [4]. These systems are designed to autonomously make decisions and execute actions based on their dynamic and unpredictable environments. Under such conditions, systems are required to be as reactive as possible to changes in the environment. Therefore, ensuring good performance is a significant challenge. In the case of planning, selecting the most effective search algorithm becomes crucial to enhance the overall performance of the system. Real-time heuristic search (RTS) (e.g.,[14, 9]) is a state-of-the-art approach for planning while executing, that helps minimize agent reaction time. These algorithms enable agents to make decisions by interleaving planning with execution while considering the evolving environment, which is an essential property in applications such as robotics, and video game agents. Despite the numerous methods proposed in this field [14, 13, 12, 3, 2], a comprehensive understanding of these algorithms remains elusive. Existing studies [1, 13, 10] have primarily focused on testing the performance of the algorithm based on a single parameter (e.g., look-ahead, sensor range). However, the influence of the problem domain characteristics on algorithm performance remains an understudied aspect. Our objective is to investigate the characteristics of the problem that may impact algorithm performances. To achieve this, we first review existing state of art of search algorithms and their applications. Then we design our experiments to evaluate some of the algorithm performances, also defining relevant metrics to use in the evaluation. Later on, experiment results are analyzed to provide a comprehensive understanding of how problem characteristics influence the performance of the different search algorithms. Finally, from the insight gained from our study, we introduce a selection algorithm that helps us to select the appropriate search algorithm. This paper is structured as follows. We begin by reviewing state-of-the-art search algorithms and previous performance evaluation studies. Next, we describe the approach we adopted to evaluate the search algorithms that are subject to our study, define the problem domain and its characteristics, and the performance metrics. In section four, we present our experimental results. Section five introduces our proposed selection algorithm with an execution example. Finally, we derive our conclusion, the limitations of our study, and possible directions for future work. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Evaluating Heuristic Search Algorithms in Pathfinding: A Comprehensive Study on Performance Metrics 

If you are using Databricks serving endpoint, and you wish to export metrics to Datadog, you can face with some challenges in Datadog documentation. This article will help you to solve them, at least it will help you to make a PoC. Here are list of links which you can use: https://docs.databricks.com/en/machine-learning/model-serving/metrics-export-serving-endpoint.html https://docs.datadoghq.com/agent/basic_agent_usage/ubuntu/?tab=agentv6v7 Databricks serving endpoint is a service which allows you to deploy your model and serve it. It is a part of Databricks MLflow. You can find more information about it here: https://docs.databricks.com/applications/mlflow/model-serving.html Once you created an endpoint, out of the box you can see some metrics in Databricks UI. Also list of metrics can be received via REST API: curl --request GET "https://${DATABRICKS_HOST}/api/2.0/serving-endpoints/{endpoint_name}/metrics" --header "Authorization: Bearer ${DATABRICKS_TOKEN}" But if you want to export them to Datadog, you need to install and configure Datadog agent. Pre-requisites Created Databricks serving endpoint Bearer PAT token behalf of Databricks user/service principal, which has `CAN_QUERY` permission on the endpoint Datadog API key Enabled OpenMetrics integration in your Datadog account, link: https://app.datadoghq.com/integrations Install Datadog agent Datadog offers several ways to install agent, they are listed here: , however looks like some of them are not working, or at least I was not able to make them work. That is why I decided to use the most simple way — install agent on Docker. https://app.datadoghq.com/account/settings/agent/latest?platform=overview Before agent installation, you need to create a file `datadog.yaml`, which should be located ` `, with the following content: /etc/datadog-agent/datadog.yaml api_key: "__DD_API_KEY__"
site: "datadoghq.com"
hostname: "__YOUR_HOST_NAME__" Then you need to create `metric-config.yaml` file, which should be located ` `, with the following content: /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml.default instances:
 - openmetrics_endpoint: https://__DATABRICKS_HOST__/api/2.0/serving-endpoints/__ENDPOINT_NAME__/metrics

   metrics:
     - cpu_usage_percentage:
         name: cpu_usage_percentage
         type: gauge
     - mem_usage_percentage:
        name: mem_usage_percentage
        type: gauge
     - provisioned_concurrent_requests_total:
        name: provisioned_concurrent_requests_total
        type: gauge
     - request_4xx_count_total:
        name: request_4xx_count_total
        type: gauge
     - request_5xx_count_total:
        name: request_5xx_count_total
        type: gauge
     - request_count_total:
        name: request_count_total
        type: gauge
     - request_latency_ms:
        name: request_latency_ms
        type: histogram

   tag_by_endpoint: false

   send_distribution_buckets: true

   headers:
     Authorization: Bearer __DATABRICKS_TOKEN__
     Content-Type: application/openmetrics-text The full possible options for ` ` can be found in template file which is located in a container under ` ` conf.yaml.default /opt/datadog-agent/agent/conf.d/openmetrics.d/ I’ve created the next , because I have some difficulties with that one which is provided in Datadog documentation: Dockerfile FROM ubuntu:20.04

# Set environment variables
ENV DD_API_KEY=[SPECIFY_DD_API_KEY]
ENV DD_SITE=datadoghq.com
ENV DD_HOSTNAME=datadog-agent-ubuntu
ENV DATABRICKS_HOST=[SPECIFY_DATABRICKS_HOST]
ENV DATABRICKS_TOKEN=[SPECIFY_DATABRICKS_TOKEN]
ENV ENDPOINT_NAME=[SPECIFY_ENDPOINT_NAME]

# Install curl
RUN apt-get update && apt-get install -y curl

# Copy the datadog.yaml file into the Docker image
COPY datadog.yaml /etc/datadog-agent/datadog.yaml
RUN sed -i -e "s@__DD_API_KEY__@$DD_API_KEY@g" /etc/datadog-agent/datadog.yaml

# Install the Datadog agent
RUN bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)"

# Copy the metrics-config.yaml file into the Docker image
COPY metrics-config.yaml /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml.default

RUN sed -i -e "s@__DATABRICKS_HOST__@$DATABRICKS_HOST@g; s@__ENDPOINT_NAME__@$ENDPOINT_NAME@g; s@__DATABRICKS_TOKEN__@$DATABRICKS_TOKEN@g; " /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml.default

RUN service datadog-agent restart

# Run a process that doesn't exit
CMD ["tail", "-f", "/dev/null"] Usage Run serving endpoint Define environment variables in `Dockerfile’ Build docker image Run docker container # Step 1: Build the Docker image
# Navigate to the directory containing the Dockerfile
cd /path/to/directory/with/dockerfile

# Build the Docker image
# Replace 'my-image' with the name you want to give to your Docker image
docker build -t datadoc-agent-image .

# Step 2: Run a container from the built image
# Replace 'my-container' with the name you want to give to your Docker container
docker run --name datadog-agent -d datadoc-agent-image

# Step 3: Connect to a container and restart agent
docker exec -it datadog-agent  bash
service datadog-agent restart Some usefully commands for DataDog agent: datadog-agent status
datadog-agent health

service datadog-agent start
service datadog-agent restart Also appears . here

How to Export Metrics from Databricks Serving Endpoint to Datadog

The Noonification: A Deep Dive Into Semantic Segmentation Evaluation Metrics (1/28/2024)

How are you, hacker?
🪐 has got you covered with fresh content from our trending stories of the day! Set email preference . Want to know what's trending right now?: The Techbeat by HackerNoon here A Deep Dive Into Semantic Segmentation Evaluation Metrics By [ 6 Min read ]
Semantic segmentation is an area of computer vision that specialises in dividing an image into regions based on pixel characteristics. @ivanpopov Read More. Open-Source: The Next Step in AI Revolution By [ 6 Min read ]
Explore the open-source revolution in AI development, distinguishing between genuine innovation and 'open-washing' attempts by Big Tech. @minio Read More. Welcoming the Future With Exciting Plans for ZKFair By [ 5 Min read ]
We sincerely share the journey of ZKFair thus far and promptly update the community on our future plans. @lumoz Read More. Meeting Customer Needs With User-Centric Product Development By [ 6 Min read ]
In a nutshell, user-centric product development implies attentiveness to the client's needs. This approach puts the customer at the heart of the product @tonythevoit Read More. Software QA: Solving Issues with Combinatorial Test Design By [ 5 Min read ]
Combinatorial test design, k-way testing, pairwise testing, software testing challenges, critical errors, test case design techniques, software QA @shad0wpuppet Read More. Python-Powered Performance Testing for QA Testers: A Beginner's Guide to Cloud API Load Testing By [ 8 Min read ]
Learn beginner-friendly Python scripts for conducting load testing on cloud app APIs, suitable for QA testers. Explore asynchronous and multiprocessing approach @shad0wpuppet Read More. Intro to Digital Fingerprints: Understanding, Manipulating, and Defending Against Online Tracking By [ 10 Min read ]
Digital fingerprinting, identifying users by hardware params. Learn about parameters, manipulation, fingerprint spoofing, online privacy, bot detection systems @shad0wpuppet Read More. Defending Your Web App: A Guide to Rate Limiting and Brute Force Attack Prevention By [ 4 Min read ]
Web app security - testing web applications: Rate Limits and X-Forwarded-For header, Brute Force attacks, and Restoring original visitor IPs @shad0wpuppet Read More. An Analysis of Combinatorial Test Design Techniques By [ 7 Min read ]
Combinatorial test design techniques: AllPairs Algorithm, Orthogonal Arrays, State-Transition Tables and Diagrams, Decision Table Testing @shad0wpuppet Read More. Bahamut Foundation Announces Launch of 10 Million $FTN Bahamut Grants Program for Development By [ 2 Min read ]
Bahamut Foundation unveils its $10 million $FTN coin fund. @btcwire Read More. How to Build Products People Love in 2024 and Beyond By [ 10 Min read ]
Product Management Trends in 2024: unpredictable user segmentation, subscription fatigue, look-alike products, cyber attacks etc. @dariabeliakova Read More. Personal Insights: From Tech Expert to Team Leader, Navigating Managerial and Product Challenges By [ 4 Min read ]
A tech expert evolving into a leader, managing managerial, product, and technical challenges - strategic decisions drive success in software development @shad0wpuppet Read More. How I Grew My Side Project to 20k Users Then Sold It for Six Figures By [ 8 Min read ]
Learn how this hobby project turned into a growing SaaS product and then a six-figure exit. Dive into the highs, lows, and valuable insights along the way. @liam637 Read More. Helium Mining in 2024: Still Worthwhile with Helium Farm By [ 2 Min read ]
With its innovative approach and lucrative mining plans, Helium Farm continues to make Helium mining not only accessible but highly profitable. @CryptoAdventure Read More. Coding Interview Prep: Algorithmic Puzzles and Cube Surface Mastery By [ 7 Min read ]
Learn to calculate a cube grid's 3D surface area in my guide, essential for FAANG interviews and software developer problem-solving. @bab3nk0v Read More. AI Stack for Text-to-Video Generation By [ 4 Min read ]
This story briefly covers how the AI video stack is evolving and how one can leverage text-to-video models to generate videos at a massive scale. @ratikeshmisra Read More. Carbon Aware Computing: Next Green Breakthrough or New Greenwashing? By [ 33 Min read ]
Big Tech is marketing its use of carbon-aware computing as way to cut emissions. without mentioning its limits or its dangers. A deep dive, and a way forward. @ismaelvelasco Read More. How to Use NextJS, TypeScript, Tailwind, and OneEntry CMS to Build an eCommerce Store 🛒👨‍💻 By [ 2 Min read ]
We will build a practical eCommerce project to see how NextJS, TypeScript, Tailwind CSS, and OneEntry CMS can be used to simplify content management. @madzadev Read More. How I Made $480,000 Through Web Development and Returned to SEO By [@Oleg Malkov](https://hackernoon.com/u/Oleg Malkov) [ 14 Min read ]
I’ve been in the SEO game since 2006, creating websites and earning from  affiliate programs. Read More. Interactively Critiquing Large Language Models by Converting Feedback into Principles By [ 6 Min read ]
Explore the groundbreaking ConstitutionMaker, an interactive tool shaping the future of chatbot customization. 🧑‍💻 What happened in your world this week? It's been said that , , . Feeling stuck? We got you covered ⬇️⬇️⬇️ We hope you enjoy this worth of free reading material. Feel free to forward this email to a nerdy friend who'll love you for it.
See you on Planet Internet! With love,
The HackerNoon Team ✌️ @feedbackloop Read More. writing can help consolidate technical knowledge establish credibility and contribute to emerging community standards ANSWER THESE GREATEST INTERVIEW QUESTIONS OF ALL TIME

The TechBeat: A Deep Dive Into Semantic Segmentation Evaluation Metrics (1/27/2024)

How are you, hacker?
🪐 has got you covered with fresh content from our trending stories of the day! Set email preference . Want to know what's trending right now?: The Techbeat by HackerNoon here A Deep Dive Into Semantic Segmentation Evaluation Metrics By [ 6 Min read ]
Semantic segmentation is an area of computer vision that specialises in dividing an image into regions based on pixel characteristics. @ivanpopov Read More. How I Made $480,000 Through Web Development and Returned to SEO By [@Oleg Malkov](https://hackernoon.com/u/Oleg Malkov) [ 14 Min read ]
I’ve been in the SEO game since 2006, creating websites and earning from  affiliate programs. Read More. AI Stack for Text-to-Video Generation By [ 4 Min read ]
This story briefly covers how the AI video stack is evolving and how one can leverage text-to-video models to generate videos at a massive scale. @ratikeshmisra Read More. Carbon Aware Computing: Next Green Breakthrough or New Greenwashing? By [ 33 Min read ]
Big Tech is marketing its use of carbon-aware computing as way to cut emissions. without mentioning its limits or its dangers. A deep dive, and a way forward. @ismaelvelasco Read More. How to Use an Uncensored AI Model and Train It With Your Data By [ 3 Min read ]
Learn how to run Mixtral locally and have your own AI-powered terminal, remove its censorship, and train it with the data you want. @jeferson Read More. PrivateGPT for Book Summarization: Testing and Ranking Configuration Variables By [ 15 Min read ]
Summarizing books with local llm. My methods for testing and optimizing the use of PrivateGPT for book summaries. @cognitivetech Read More. If You're a Facebook User, Thousands of Companies Are Watching You By [ 6 Min read ]
Using a panel of 709 volunteers who shared archives of their Facebook data, Consumer Reports found that a total of 186,892 companies sent data about them @TheMarkup Read More. What Could Have Stopped the 23andMe Hack? By [ 8 Min read ]
Here’s how the 23andMe hack happened and how different login-access control solutions could have stopped it. @hillpot Read More. 418 Stories To Learn About Fintech By [ 73 Min read ]
Learn everything you need to know about Fintech via these 418 free HackerNoon stories. @learn Read More. How I Grew My Side Project to 20k Users Then Sold It for Six Figures By [ 8 Min read ]
Learn how this hobby project turned into a growing SaaS product and then a six-figure exit. Dive into the highs, lows, and valuable insights along the way. @liam637 Read More. Coding Interview Prep: Algorithmic Puzzles and Cube Surface Mastery By [ 7 Min read ]
Learn to calculate a cube grid's 3D surface area in my guide, essential for FAANG interviews and software developer problem-solving. @bab3nk0v Read More. And the Web3 Award Goes To... By [ 8 Min read ]
The awards represent the innovative spirit and transformative potential of Web3. @nakedcollector Read More. Crosschain Interoperability: Soon, We Won’t Say “Web3” By [ 5 Min read ]
The Role of Blockchain Interoperability in Gaming and Beyond with Portal's Universal Ecosystem. @phillcomm Read More. The Modern AI Stack to Increase Developer Productivity By [ 5 Min read ]
The integration of these three pillars - Knowledge, Code Generation, and Code Review - forms a robust foundation in the AI-driven development process. @gitcommitshow Read More. KYC and Regulation: Solving the Problem of Financial Crime in Crypto By [ 6 Min read ]
Many specialists believe that the introduction of strict KYC procedures and crypto regulation would solve the financial crime problem. Is it correct though? @dshishov Read More. Ship Smaller Units of Software and Limit the Sizes of Your Local Host Branches By [ 3 Min read ]
As I product manager, early on with most software developers I work with, I  end up saying: ship smaller units of software and limit the size of your branch. @david Read More. BYOK (BringYourOwnKey) in Generative AI is a Double-edged Sword By [ 5 Min read ]
Do you know what Bring Your Own Key (BYOK) in generative AI is? Read on and learn more about this new AI concept. @emmanuelaj Read More. How to Build a Stress-Free IT Career By [ 10 Min read ]
How to be smart about choosing your job and build a stress-free career in tech. @gmakarov Read More. Lightbug 🔥🐝- The First Mojo HTTP Framework By [ 5 Min read ]
Intro to Lightbug 🔥🐝, a lightweight HTTP framework in pure Mojo that allows to build web services with simplicity of Python and performance of C++ @a2svior Read More. How to Revolutionize Your Startup Success in 2024 with AI Co-pilot Tools By [ 11 Min read ]
Revolutionize startup success in 2024 using PitchBob, an AI co-pilot tool for founders, enabling rapid task execution and personalized entrepreneurial guidance. 🧑‍💻 What happened in your world this week? It's been said that , , . Feeling stuck? We got you covered ⬇️⬇️⬇️ We hope you enjoy this worth of free reading material. Feel free to forward this email to a nerdy friend who'll love you for it.
See you on Planet Internet! With love,
The HackerNoon Team ✌️ @alexlash Read More. writing can help consolidate technical knowledge establish credibility and contribute to emerging community standards ANSWER THESE GREATEST INTERVIEW QUESTIONS OF ALL TIME

The TechBeat: A Deep Dive Into Semantic Segmentation Evaluation Metrics (1/23/2024)

Semantic segmentation is an area of computer vision that specialises in dividing an image into regions based on pixel characteristics to identify objects or boundaries, simplifying image analysis. The method has many different use cases, from detecting handwritten words and numbers in scanned documents to Google and Apple employing it for foreground-background separation in portrait mode. Platforms like Pinterest and Amazon leverage semantic segmentation for image uploading, enabling the identification of similar-looking products by segmenting the area of interest from the rest. Semantic segmentation is a cornerstone for the autonomous vehicle industry, as it helps identify lanes, cars, and pedestrians. This technology is crucial for self-driving cars to comprehend their surroundings through camera video inputs, enabling informed decisions about future actions. Moreover, semantic segmentation finds application in the field of in vitro fertilisation (IVF), particularly during embryo classification. It is used to precisely delineate embryonic structures, contributing to the accurate assessment of embryo development and selection in assisted reproductive technologies. In this article, I will explore the intricate field of semantic segmentation and explain how to evaluate semantic segmentation efficiency with examples from the embryology field. Core Concepts of Semantic Segmentation Semantic segmentation builds upon image classification models, refining the process by labelling each pixel to a predefined class. Pixels of the same class form a segmentation mask, offering precise object classification and boundary delineation. In this process, an input image undergoes neural network processing, resulting in a colourised feature map where each pixel colour signifies a distinct class label. This spatial understanding allows computers to differentiate objects, separate foreground from background, and automate tasks in robotics. Understanding Semantic Segmentation The semantic segmentation process unfolds in distinct steps: Initiated with an input image, the semantic segmentation model processes it through a complex neural network architecture. Input Image Processing The neural network dissects the image, extracting intricate features and patterns, and understands the visual content in granular detail. Neural Network Analysis Unlike traditional image classification, semantic segmentation operates at the pixel level. Each pixel in the image is individually classified, contributing to the creation of a detailed segmentation mask. Pixel-level Classification The output of the model is a colourised feature map of the image. Each pixel's colour corresponds to a specific class label (usually an integer), distinguishing various objects within the scene. Colorised Feature Map Pixels sharing the same class label are grouped, forming coherent segments in the segmentation mask. It helps to perform the precise identification and localisation of objects. Grouping Pixels The segmented image offers a spatial understanding and makes it possible for computers to discern objects, separate foreground elements from the background, and grasp the contextual relationships between different entities. Spatial Understanding Semantic segmentation, with its pixel-level analysis, provides a foundation for advanced visual comprehension, dramatically changing how machines interpret and interact with visual data. Evaluation Metrics The predominant class of metrics for semantic segmentation falls under the category of overlap metrics. These metrics measure how well the model's prediction matches the actual details in a given image. This provides important information about how accurate the model is. Overlap Metrics The most widely employed overlap metrics include the Dice Similarity Coefficient (DSC) and Intersection over Union (IoU), also known as the Jaccard Index. Both metrics provide a numerical representation of the overlap between ground truth and predicted masks, with values ranging from 0 (no overlap) to 1 (full overlap). Though these metrics provide considerable insights, they have limitations as well. Notably, they cannot discern shape differences, a critical factor in scenarios like blastocyst segmentation in the field of embryology, where accurate prediction of non-trivial shapes is essential. Boundary-based metrics can help to identify changes in shapes and complement overlap metrics. However, they also have limitations. Although they can detect shape differences, they often overlook the structural peculiarities of masks, such as holes within predictions. This is particularly significant in the context of blastocyst segmentation, where components may contain holes. Volume-Based Metrics Another dimension to consider in segmentation is volume accuracy. Metrics based on volume are crucial for checking how accurate semantic segmentation models are. They help to measure the size-related details of the predictions. Two key metrics in this category are Volume Similarity (VS) and Relative Volume Difference (RVD). measures the similarity between ground truth and prediction masks by considering their respective volumes. This metric offers a quantitative assessment of how closely the predicted volume aligns with the actual volume. It helps researchers make a more thorough evaluation of the segmentation accuracy. Volume Similarity (VS) quantifies the disparity in volume between the ground truth and prediction masks in relation to the volume of the ground truth mask. This metric provides a normalised measure of the volume difference, offering insights into the proportion of discrepancy relative to the true volume. Relative Volume Difference (RVD) Volume-based metrics are an effective way to evaluate the performance of semantic models, especially in situations where precise volume prediction is crucial. One major drawback of using such metrics is that they do not consider the location of the predicted mask. As a result, relying solely on them may result in predictions of the correct size but in the wrong place. To address this issue, overlap or boundary metrics should be integrated into the evaluation process. When evaluating models, it is crucial to select relevant metrics. Although boundary-based, and volume-based metrics can be useful in certain scenarios, they don't consider all of the intricacies of mask structure. The Dice Similarity Coefficient (DSC) is the most versatile and commonly used metric for semantic segmentation evaluation and is recommended as a go-to solution. This choice is in line with the trends observed in state-of-the-art models that focus on blastocyst semantic segmentation. [1] [2] [3] Challenges After summarising my experience in analysing semantic segmentation evaluation metrics, it is clear that there are several challenges to accurately assessing the performance of segmentation models. These challenges are mainly related to shape variability, hole recognition, and nuanced volume accuracy measurement. In the following insights, I will outline these key difficulties encountered: Existing metrics often struggle to accurately evaluate shape variability, particularly for objects with intricate or non-trivial structures. Capturing diverse shapes, which is utterly important in medical imaging applications or object recognition, becomes a challenging task. Shape Variability Assessment. Boundary-based metrics commonly used in evaluation may not effectively recognise holes within segmented objects. This limitation impacts the metric's ability to comprehensively assess the structure of objects, especially when internal voids are present. Hole Recognition Limitations. While volume-based metrics provide insights into volumetric aspects, accurately predicting the volume of each segmented component remains challenging. Relying solely on volume metrics may lead to incomplete evaluations, particularly in scenarios where precise volume prediction is essential. Volume Accuracy Measurement. Evaluating spatial relationships and contextual understanding within the segmented areas is a complex task. Existing metrics may struggle to capture the intricate interplay of objects and their surroundings, limiting the holistic assessment of segmentation quality. Difficulty in Addressing Spatial Relationships. Metrics may face difficulty distinguishing between similar structures, leading to potential misclassification or underestimation of segmentation accuracy. This becomes pronounced in scenarios where objects share visual similarities but have distinct semantic meanings. Inability to Differentiate Similar Structures. Evaluation metrics may lack adaptability to specific domains or applications. Thus, to ensure relevant and accurate assessments, it is crucial to fine-tune metrics for domain-specific nuances. Limited Adaptability to Domain-Specific Requirements. The outlined challenges signify a dynamic scientific landscape that calls for continual refinement in assessment methodologies. Semantic segmentation is a crucial technology that has wide-ranging applications in areas such as e-commerce, medical imaging, and autonomous vehicles. Its potential to improve people's lives is significant. However, to use it efficiently, we need to be able to measure its accuracy, and evaluating the accuracy remains challenging. Metrics such as the Dice Similarity Coefficient and Intersection over Union are not perfect, as they struggle with shape variability and hole recognition. Therefore, a deeper understanding and improvement of these evaluation metrics is essential. By combining overlap, boundary-based, and volume-based metrics, we can enhance the accuracy of semantic segmentation. This advancement will bring notable benefits to various industries.

A Deep Dive Into Semantic Segmentation Evaluation Metrics

These tools do more than just display data; they help in quickly reading, analyzing, and making informed decisions, which is critically important for both developers and businesses. A well-designed dashboard can become a key element in forecasting trends and anticipating outcomes, impacting the efficiency and motivation of the team. Hello! My name is Lisa, and I have worked as a product designer in large IT companies and small startups. Among my experience was the creation of internal products that help monitor metrics and the state of service operations. Thanks to this, I have accumulated significant experience in dashboard development, and today I want to share this experience with you. Today, I want to inspire you to create useful and effective dashboards by sharing my approach and emphasizing the importance of UI design in this process. The sense of dashboard I want to start with three fundamental principles that should guide the creation of dashboards. These are precisely what you, as a product manager or product designer, should advocate for: One of the less obvious but critically important tasks in dashboard creation is the ability . Visualization should not only display numbers but also convey context, show cause-and-effect relationships, and identify trends. This requires a deep understanding of how data correlates with real business processes and user behavior. Dashboard developers should aim not only to present data but also to explain it, creating an intuitive and engaging narrative structure. to tell a story with data Often in the dashboard development process, you may encounter the issue of “expectation overload” — when users want to see too many features and data. It’s important to be able to manage these expectations, explaining limitations and focusing on the most important aspects that truly add value. This might require communication and persuasion skills, as well as the ability to and the impact of various features on the final product. objectively assess priorities Technical and organizational obstacles often become a serious challenge. For instance, integrating data from various sources can be a complex task due to format incompatibilities or API limitations. In such cases, it’s crucial to to find solutions that ensure smooth integration and reliable system performance. It’s also important to consider organizational aspects, such as training the team to use the new dashboard and implementing it into work processes. closely collaborate with developers Goals Every successful dashboard begins with a clear understanding of its goals and audience. At my current job, before starting the development of a dashboard, we conduct . This could be, for example, monitoring key performance indicators for executives or providing technical data for the development team. an analysis of the objectives it needs to achieve and the audience it is intended for To illustrate, in one of our projects, the goal was to create a dashboard for monitoring server operations. The audience consisted of technical specialists who required detailed, real-time information about the system’s status. The goals and audience defined the structure and design of the dashboard, making it maximally informative and user-friendly for the end users. Moreover, an effective dashboard helps reduce risks for the company by providing For instance, thanks to the instant display of server downtime data, the team can quickly respond to issues, minimizing downtime and potential losses. quick access to critically important information. Equally important is the impact of the dashboard on the team’s motivation and awareness. When the team sees a dashboard tailored specifically for them with up-to-date metrics every day, it enhances their engagement and provides a clear understanding of the current state of the project. This creates a sense of control and confidence in the decisions being made, which in turn contributes to an overall increase in work efficiency. Users Before beginning the design of a dashboard, it’s critically important to understand . In my practice, this starts with a deep dive into the world of the end-users — developers, project managers, data analysts, and so on. I conduct interviews, analyze workflows, and gather feedback to understand their daily tasks and challenges. who the primary users will be and what tasks they will be addressing For example, might need real-time system performance monitoring, while may require an overview of task completion timelines and resources. By identifying these key usage scenarios, I can ensure that the dashboard will not only be functional but also maximally useful for specific users. developers project managers Next, I link the functionality of the dashboard to the overall It’s not just about presenting data; it’s about how these data can help achieve specific objectives. For instance, if a business goal is to increase efficiency, then the dashboard should highlight metrics that track progress and identify bottlenecks. business goals of the company. In one of the projects, the goal was to improve customer service quality. For this, we developed a dashboard that displayed key indicators in real-time related to the processing of customer inquiries, allowing the team to respond promptly to issues and improve service quality. Thus, understanding both user needs and business goals not only displays data but actively contributes to achieving the organization’s ultimate goals. Strategy When we start defining the key functions and priorities of a dashboard, it’s important to find a balance between what users want and what is truly needed to achieve business goals. In my experience, I begin by creating a list of features that could be useful and then prioritize them based on their impact on users and business objectives. For instance, for a dashboard designed for monitoring application performance, key features might include displaying system response time, code errors, and server load levels. This is not just about having “nice-to-have” features; it’s about creating a “must-have” list that is guided by strategic goals and the real needs of the user. In developing a dashboard, I always think about how it will fit into the broader context of the product. 👾How will it interact with other features? 📎 How will it help the user better understand and use the product as a whole? For example, if a dashboard is designed for a CRM system, it should not only provide data but also simplify interactions with customers and improve processes for working with them. In this way, we make the dashboard . a valuable addition, not just an isolated tool Design. Prototyping. Iterations. I start with basic sketches that allow for a quick assessment of ideas and structure. These sketches are more than just drawings; they help me and my team visualize the dashboard concept and initiate a dialogue about how it should function. Once the concept is approved, I move on to creating more detailed prototypes. I develop prototypes that not only resemble the final product but also . This is a critically important step as it allows for testing with real users and gathering feedback that will influence subsequent design iterations. give an insight into user interaction When designing the dashboard interface, my main goal is to create an intuitive and effective user experience. Here are a few key principles I always keep in mind: 🎧 Clarity and Noise Reduction It’s important that users can quickly understand what is displayed on the dashboard. I aim to avoid information overload, focusing on key metrics and data necessary for the specific audience. 🧭 Consistency and Predictability I use consistent design elements and layouts so that users can intuitively understand how to interact with the dashboard. This includes uniformity in colors, fonts, and layouts. 💠 Interactivity and Feedback The dashboard should be informative as well as interactive. I include elements such as filters and dropdown menus that allow users to customize data display according to their preferences. 🍟 Accessibility and User-Friendliness I believe that design should be accessible to all users. This includes considering people with different abilities and preferences, as well as ensuring that the dashboard is user-friendly on various devices. Data Visualization and Information Architecture One of the key aspects of creating effective dashboards is the proper selection of data . Each data type requires its own approach to presentation to ensure the information is understandable and easily digestible. visualization methods For example, line charts are ideal for time series as they clearly illustrate the dynamics of changes. Bar or pie charts work well for comparing values across categories. Heat maps can be useful in visualizing complex datasets, allowing for quick identification of patterns and anomalies. It’s important to remember that visualization should be not only informative but also . I use colors and shapes to make data more accessible and comprehensible, while avoiding overloading with visual information. appealing Additionally, I focus on and ensuring a consistent flow of information, which aids users in easily navigating and finding the data they need. For instance, in a dashboard designed for sales analysis, I might group data related to customer demographics, sales figures, and product performance in distinct but interconnected sections. This way, a sales manager can quickly locate specific insights, such as the best-selling products in a particular demographic segment. grouping related data Another example is a financial dashboard where I categorize data into revenue, expenses, and profit margins. By doing so, financial analysts can effortlessly switch between different financial aspects without losing track of the overall financial health of the company. These thoughtful groupings and layouts not only improve the usability of the dashboard but also enhance the decision-making process by presenting a coherent narrative of the data. Testing and Iteration To identify usability issues and understand how well the dashboard meets users’ needs and expectations, we showcase our developments to real users. Based on the feedback collected, the process of iterations begins. The key is to be the design in response to the data received. Iterations can affect both minor details of the interface and larger aspects of structure and functionality. open to changes and ready to adapt In one of our recent projects on creating a dashboard for server performance monitoring, user testing became a pivotal moment. After the initial development of the dashboard, we conducted a series of tests with our development team, who were the primary users of the product. During these sessions, I observed how they interacted with the dashboard, . A key issue was that the developers had trouble interpreting the performance time series. The charts were overloaded with data, making it difficult to quickly analyze the situation. noting any difficulties or misunderstandings Based on this initial testing, we began the process of iterative improvements. Our first step was to simplify the charts by highlighting the most critical metrics and reducing visual clutter. We also added the ability to filter data by time intervals, allowing developers to focus more easily on specific periods. After implementing these changes, we conducted , this time with a broader group of users, including project managers. The feedback was significantly more positive: the charts became clearer, and the data filtering feature was particularly appreciated for its usefulness. a second round of testing Product Integration The next key stage is the integration of the dashboard into the overall product. In my project of creating a dashboard for server monitoring, one of the main tasks was to ensure its . This required close collaboration with the development team to ensure that the dashboard accurately collects and displays data in real-time. seamless integration with the existing system We utilized APIs to connect the dashboard to various data sources, ensuring high performance and reliability. Work was also done to optimize data loading so that the dashboard could respond quickly to user queries, even when processing large volumes of information. It’s crucial that the dashboard not only meets current needs but can also . To this end, we developed a modular architecture that allows for the easy addition of new features and the integration of additional data sources. Ensuring continuous support and updates for the dashboard was also an important part of the process, guaranteeing its stability and the relevance of its data. adapt to future changes and expansions of the system Creating dashboards is an integral part of product design. They not only provide valuable information for decision-making but also significantly enhance user convenience and efficiency. My experience has shown that a well-designed dashboard can have a substantial impact on business, improving processes and aiding in the achievement of goals. Tips and Tricks for Product Designers Start by figuring out what your users and . It’s like being a detective, but instead of solving crimes, you’re solving design puzzles. business really need Regularly . Be ready to iterate based on the feedback test the dashboard with real users Aim for in data visualization. Avoid information overload and focus on the key metrics. simplicity Develop dashboards with and potential data or process changes in mind. future scaling Postscript Alternative Ways to Create Dashboards: No-Code and Low-Code Solutions In the modern world of product design, there is an increasing number of . These no-code and low-code platforms offer convenient means for quick and efficient dashboard development, making them accessible not just for professional developers but also for business analysts, marketers, and other specialists. tools that allow for dashboard creation without deep programming knowledge Platforms: is  a powerful platform for creating interactive dashboards, particularly valuable for monitoring and analyzing time-series data. Ideal for real-time system monitoring, network analysis, and IoT applications, Grafana supports multiple data sources, including Prometheus, MySQL, PostgreSQL, and many others. This tool is highly flexible in visualization customization, allowing for detailed and adjustable dashboards. However, it may require deeper technical knowledge for setup and use compared to other no-code/low-code solutions. Grafana is one of the most popular platforms for data visualization. It offers robust tools for creating a variety of dashboards with the ability to connect to different data sources. Tableau makes it easy to integrate data from various sources and create interactive dashboards. It is ideal for integration with other Microsoft products, such as Excel and Azure. Microsoft Power BI is a free solution from Google that allows for creating customizable dashboards. Particularly well-suited for working with data from Google Analytics, Google Ads, and other Google services. Google Data Studio : This tool provides flexible options for dashboard creation and business analytics. It enables not just data visualization but also in-depth analysis. Looker Using no-code and low-code platforms has its advantages, including . However, one should consider their limitations in terms of customization and integration with some complex systems. Therefore, it’s important to assess the needs of the project and choose a tool that best meets these requirements. development speed, ease of use, and flexibility in design alteration

Creating the Perfect Dashboard: Visualizing Your Metrics

নিখুঁত ড্যাশবোর্ড তৈরি করা: আপনার মেট্রিক্স ভিজ্যুয়ালাইজ করা

Erstellen des perfekten Dashboards: Visualisierung Ihrer Metriken

Creando el panel perfecto: visualizando tus métricas

Créer le tableau de bord parfait : visualiser vos métriques

परफेक्ट डैशबोर्ड बनाना: अपने मेट्रिक्स को विज़ुअलाइज़ करना

完璧なダッシュボードの作成: 指標の視覚化

완벽한 대시보드 만들기: 측정항목 시각화

Criando o painel perfeito: visualizando suas métricas

Создание идеальной информационной панели: визуализация показателей

Mükemmel Kontrol Panelini Oluşturma: Metriklerinizi Görselleştirme

Tạo Trang tổng quan hoàn hảo: Trực quan hóa các chỉ số của bạn

创建完美的仪表板：可视化您的指标

Learn about the key performance metrics that mobile teams should track in their mobile apps so they can deliver better user experiences. In order to build the best experiences, engineers need the best data available. Mobile makes this challenge more complex, with variables that include device types, different operating systems, and connectivities, to name just a few. With a growing number of tools that all offer varying levels of visibility into your mobile app's health, performance, and stability, it can be difficult to know exactly which metrics you should be tracking. In this post, we’ll outline the most important performance metrics you need to track and how they help you get to the root cause of issues more quickly so you can build better mobile experiences. 1. Startup time Mobile users typically check their apps on the go and are accustomed to instantaneous results. Therefore, they won’t sit around and wait for the app to load. For example, if the Uber app takes too long to load, the user will probably switch to the Lyft app instead. In addition, users who had a poor experience in the Uber app and a decent experience in the Lyft app are much more likely to become a loyal Lyft user. Now, not only have you lost revenue from that user’s session, but your cost per acquisition and churn increased, not to mention the LTV that particular customer could have brought the company. Therefore, is essential to its success. ensuring your app’s startup time meets user expectations However, if the mobile team only has access to the , they probably miss key changes and respond reactively rather than proactively. average startup time For example, let’s say a company is launching its app into a new market, and the overall feedback is negative. The average startup time shows that the app is a few milliseconds longer than it was before the launch, but it doesn’t indicate that it became dramatically slower during the launch. So there must be another problem, right? Unfortunately, because the percentage of users in the new market is only a fraction of the overall user base, it makes sense that even an abysmal startup time in the new market would only minimally impact the total user base’s startup time. Instead, the team needs to be able to segment the data to better understand how startup time impacts the business. For example, how are high-value users suffering from slow startups? Are users in a new market experiencing worse performance? Are certain devices experiencing slower startups? This data puts your company back in control of the situation and omits the guesswork in time-sensitive scenarios that impact your revenue. To learn more, check out our eBook on . how to improve mobile app startup time 2. Crash rate are a surefire way to anger customers and for a good reason! It’s essentially the equivalent of a customer walking into a brick-and-mortar store where the staff kicks them out the door halfway through their shopping. Crashes This is a major issue for the company’s brand for two reasons: It as customers feel their time is disrespected. damages the reputation of your brand It hurts your revenue as customers can’t complete their immediate transaction, and you lose the lifetime value of that customer if they decide to switch to a competitor. Here are just a few examples from various industries where a crash directly translates to lost revenue: If an e-commerce app crashes during checkout, the customer won’t be able to make their purchase and likely won’t be back. E-commerce Apps: : If a POS system crashes during a live event, none of those live customers will be able to make a purchase or enter the venue. POS Systems : If a smart device, such as a toothbrush, crashes during the setup process, it’s very likely the customer will return the product. Smart Device Apps However, while tracking the average crash rate is a good start, it’s . still not enough to understand how crashes impact your business For example, if the current crash rate is just 0.5%, you might not see the need to drill deeper. However, what if all of the crashes that occur happen at the checkout screen? That tiny percentage could be cheating the business out of significant revenue. So, in addition to looking at the major metric numbers, it’s also important to have data that shows patterns in the crash rate. Specifically, how are various high-value areas in your app performing? Which devices tend to experience the most crashes? How is the app performing for high-value users? Are there any regions that experience particularly poor crash rates? And if so, should the app be fixed or removed from those regions? By segmenting crash details, your team can better prioritize fixing issues. 3. ANR rate are typically described as freezes or glitches. Application Not Responding (ANR) errors Essentially, if the main thread is blocked, the applications cannot run effectively. Therefore, the user cannot proceed, which can have a major impact on your business. For example, a retailer we worked with had an issue with ANRs. This issue was causing startup times to increase by almost 60%, translating to an estimated revenue loss of $6.5 million per year. , the engineering team was able to quickly address the issue and reclaim that lost revenue. With the right data Beyond revenue, ANRs can also in the Google Play Store and make it less visible to new customers. negatively impact an app’s rankings To effectively track ANRs, you can look at the stack trace and see how users responded to the issue. From there, the mobile team can prioritize what to fix first based on how many people leave at various thresholds, where high-value users are most affected, and which device types and screens are most affected by ANRs. To learn more, check out . this post on investigating Android crashes like ANRs 4. Region metrics Tracking region metrics is also very important as users in different regions have different devices and different connectivities. This can have a major impact on the business for several reasons. First, regions differ in their importance to a business’s bottom line. For example, perhaps only a portion of your users are in Singapore, but they might make up a large portion of your total annual revenue. Therefore, checking region metrics on a granular level will spotlight opportunities to improve the app, particularly for high-value users. Tracking segmented region metrics is also essential when launching into a new region. For example, suppose you expand into Australia, yet that geographic area only makes up 5% of all users on launch. In that case, it won’t impact median/average metrics enough to allow the team to effectively track performance. It’s also important to take into account cultural differences, and with a tool that provides region-specific metrics, the team can test specific aspects of the app just for that region. For example, an e-commerce checkout screen that converts well in the United States may not convert as well in Dubai. In addition, granular region metrics make it easy to roll out new features slowly. For example, the team can roll out a new feature in a smaller region and see how it performs. If it performs well, roll it out to more and more regions with higher-value users. This data can guide your team in answering critical questions like: Do we need to build a new app for this region? How is this new launch performing compared to launches in other regions? Which regions are the most problematic? And should we stop serving them altogether? ​​How are various critical areas of the app performing in one region versus another? 5. Session duration Another key metric to keep an eye on is session duration, as it signals how long users are using your app. If the average session duration changes dramatically in a week, it’s a pretty good hint that there is an issue with the app. For example, if you have a gaming app and notice the average session duration dropped from 15 minutes to 5 minutes, there’s a good chance that users had a poor experience. With this clue, you can ask questions like: Did we ship a bad release? Are users less engaged with the app? Is there a corresponding change in another metric that can help explain the change in session duration? Tracking session duration also allows the team to investigate patterns among the affected individual sessions and discover the root cause of issues. By analyzing session duration in parallel with other metrics, mobile engineers can draw a clearer picture of which issues are most intrusive to users. For example, an e-commerce store might have an OOM issue on the main scrolling feed of products that correlates strongly with a shorter session duration, whereas slow network calls on another screen have little to no correlation with the session duration. Therefore, tracking session duration is a great way to prioritize which issues should be fixed first as it directly reveals reduced user engagement. 6. Churn/retention rate It costs a lot more to acquire a new customer than it does to keep a current customer happy, and a leading cause of churn in mobile app users is a poor user experience. Downloading a new app only takes a few seconds, so if an app’s experience interrupts the user in a way that takes more than a few seconds of their time, don’t expect them to stick around. Therefore, track not only the overall churn and retention rate but also the churn and retention rates of segments of the user base (by devices, connectivities, regions, etc.). This will illuminate various opportunities to improve and prioritize retention. For example, you may find that while the churn rate is very high in one particular region, that region has few high-value users. Therefore, you may decide to invest engineering resources somewhere else where they will translate to larger business outcomes. Tracking churn and retention is also a great way to understand how users respond to new releases and various experiments. If there is a correlation between high churn and a new feature update, that’s a strong indicator that the team needs to roll back that feature update. Detailed data will allow the team to track the churn of various segments (such as specific regions or devices) to which the update is rolled out. Without segmented data, it will be difficult to see the impact that various feature updates have on small testing groups. Therefore, only once the feature is rolled out to a large group of users (and presumably caused many monthly active users to leave) will it become apparent that the feature rollout was premature. 7. User termination rate While some user terminations are nothing more than a user decluttering their phone, many terminations occur because the app has frozen, and the user has no choice but to swipe up and terminate the session. Of course, a user who is forced to terminate their session will likely be disgruntled and choose a competitor’s app instead. To prevent this, monitoring the average user termination rate is an excellent indicator of potential issues that could lead to churn, including: Failed ad loads are causing freezes Slow or broken user flows Failing purchases Server outages causing login errors Excessive media loading that causes slowdowns In addition to providing an overview of the average user termination rates, mobile teams need to know the source of every poor user experience. Therefore, one of the key features we built into Embrace is the ability to see . which screens cause frustrated users to abandon your app Therefore, rather than wasting valuable time and losing sales while engineers guess where the terminations are occurring, the mobile team is immediately directed to the issue so that they can fix the problem as efficiently as possible. 8. Timing of key user actions There are likely a few user actions within your app that absolutely must work 100% of the time. For example, if an office offers keyless entry, that feature must work every time. Otherwise, people may not be able to enter the office without calling additional support. Therefore, select a few key user actions and add them to the list of performance metrics the mobile team is tracking. In many cases, customer reviews don’t say where they experienced an issue in the app, so tracking specific user actions is a great way to catch issues that otherwise aren’t immediately apparent. This also helps you answer questions like: How many users experience issues in these critical areas of the app? Is there a direct correlation between issues in specific areas of the app and churn? How long do users try before giving up and abandoning the app? For example, one of our customers noticed that about 1% of all purchase attempts resulted in purchase failures. However, both of the associated network calls were resolving successfully, so there were no obvious errors to inspect. Therefore, they began tracking the exact moment a customer would make a purchase and found that the two network calls were happening out of order in 1% of purchase attempts, which resulted in a failed purchase. Though customers had been complaining about this issue, the mobile team couldn’t pinpoint the root cause without knowing the timing, outcome, and order of all events within the affected sessions. High-fidelity user experience data was vital in helping them reclaim 1% of their total sales. For a company doing $10 million in annual sales, that’s $100,000 that would otherwise be lost each year! 9. Memory usage It’s important to track your app’s memory consumption to identify any potential memory leaks. The efficiency of your app’s memory usage directly relates to how responsive your user experience is. This is why monitoring and optimizing memory usage plays a crucial role in delivering a seamless user experience. You can avoid issues with your app’s memory usage by: Optimizing your codebase and pinpointing areas where memory is unintentionally retained causes an increase in memory usage. Tracking your app’s memory consumption on a regular basis, noting any changes after new releases. Carefully monitoring memory-intensive operations like loading large images or processing large files and creating for abnormal spikes or sustained high memory usage. alerts A fine-tuned strategy regarding your app’s memory consumption creates responsive, stable, and effective apps that your users will love. 10. Connectivity Mobile apps operate in diverse environments with varying , including 3G, 4G, 5G, Wi-Fi, and sometimes limited or unstable connectivity. network conditions This is why it’s important to recognize the challenges posed by these varying conditions to create great user experiences. Indicators of poor network performance include: Latency and slow response times. Bandwidth usage. Inefficient API calls. Device-specific offline capabilities. For example, is an agricultural app that allows farmers and agronomists to document their findings while they’re in the field with their peers and colleagues. Farm Dog The app would crash often when network response times were exceptionally slow and when devices couldn’t determine whether they were connected. They noticed that Google Maps response times between 18-22 seconds when they should be only a few seconds. Without the proper tooling, they would need to use convoluted workarounds to solve issues using a proxy to simulate a bad network connection. However, with higher-fidelity data, they can see the exact conditions users have in the field, including: Network calls across device types, app versions, Wi-Fi, and cellular. Insight into 4xx and 5xx error trends across common domains to identify problematic pathways. The broken endpoints that prevent your users from starting up the app, loading key content, or completing crucial transactions. The duration of every network call from the client side reveals hidden points of latency. Armed with this information, the Farm Dog team simulated the issues their users faced, easily identified the problematic conditions, and fixed them. Set your mobile team up for success If you care about mobile experiences, then you need the data that enables your team to see all of the metrics mentioned above. Embrace helps you build better mobile experiences by giving you just that data, making your engineers more efficient and less bogged down by tedious work. Learn more about and download report to learn about key app frustrations according to users. Embrace The State of Mobile Experience Author: Colin Contreary Also published . here

Stories, News & Mentions about UPMETRICS

Upmetrics’s Stories on HackerNoon