Labelbox

labelbox

A Deep Dive into Benchmarks, Consensus, and Review Originally published on blog.labelbox.com Training data quality is critical for a machine learning model’s performance. Quality is measured by both the consistency and the accuracy of labeled data. The industry standard methods for calculating training data quality are benchmarks (aka gold standard), consensus, and review. As a data scientist in AI, an essential part of your job is figuring out what combination of these quality assurance procedures is right for your project. In this article you will learn: The definitions of quality, consistency and accuracy and why they matter The industry standard methods to quantify quality The most cutting-edge tools to automate quality assurance processes Consistency vs. Accuracy Quality is comprised of consistency and accuracy. It is not just how correct a label is, but also how often it is correct. This section describes the industry standard methods for measuring consistency and accuracy. Consistency is the degree to which labeler annotations agree with one another. Consistency prevents random noise by ensuring that labels are correct or incorrect in a consistent manner. Consistency is measured through a algorithm. Without the automation of cutting-edge AI tools, this process is manual, time intensive, and a security liability. Since labels can be consistently right or consistently wrong, high consistency alone is not enough to fully explain quality. consensus Accuracy measures how close a label is to the ‘Ground Truth’. Ground truth data is a subset of the training data labeled by the knowledge expert or data scientist to test annotator accuracy. Accuracy is measured through . Benchmarks enables data scientists to monitor the overall quality of their data and then investigate and troubleshoot any potential dips in quality by providing insight into the accuracy of the labelers work. benchmarks is another method to ensure accuracy. Once the labels have been completed, a trusted expert reviews label accuracy. The review is usually conducted by visually spot checking labels, but some projects review all labels. Review is often used to identify low-accuracy and inconsistencies in the labeling process while benchmarks is often used to get a sense of labeler performance. Review Benchmarks tends to be the cheapest quality assurance option since it involves the least amount of overlapping work. However it only captures a subset of the training data set. Consensus and Review may be more expensive or cheaper than each other depending on the consensus settings and the level of review (both can test 0–100% of the dataset and have multiple labeler overlap). Quality Workflows Ideally, quality assurance is an automated process that operates continuously throughout your training data development and improvement processes. With the Labelbox consensus and benchmark features, you can automate consistency and accuracy tests. The tests allow you to customize the percentage of your data to test and the number of labelers that will annotate the test data. For many data scientists, discovering the best combination of quality assurance tests for a project is often an emergent process in which clarity is gained through experimentation. Benchmarks Benchmarks workflow Create a new Benchmark by starring an existing label (Automatic) Labelers get benchmarked at random intervals Keep track of your project’s overall quality and dive into any deviation, either by labeler (3a) or by benchmark (3b). Tip: Many find it helpful to follow along with the examples by trying it out yourself with the free version of Labelbox . Benchmarks Example Below is a sample jellyfish project to exemplify the benchmarks workflow summarized above. This project uses a rectangular bounding box, but there are polygon and point options for more fine-tuned shapes. Once you have started a project and labeled the ground truth data, you can tag the labels with the benchmark star. As the labelers are randomly benchmarked, you can monitor project quality with the overall quality graph. To troubleshoot drops in quality, you can explore performance by labeler or benchmark. Systemic poor labeler performance is often indicative of poor instructions, while poor performance on certain pieces of data is often indicative of edge cases. Data scientists use these values to help them improve labeler on-boarding and education processes. By clicking on the benchmark with 65% accuracy, you can see the crowded jellyfish image and compare it to the benchmark. While they are very similar, the labels tend to disagree on how much of the tentacle to include in each bounding box. From here you can either edit, or delete and re-queue the label. Data labeling is an iterative process. Often our idea of what we think we should label evolves as we become more intimate with the project. For example, maybe at the onset it made sense to label the entire jellyfish. However, it is plausible that the very thin long whispery tentacles might confuse the model rather than improve its performance. The model might start identifying any sliver of a line, especially near the edges of the photo, with an occluded jellyfish when it very well might just be underwater plant debris or the tendril of sea anemone. Because this is not a well recognized problem, the ability to numerically and visually monitor and iterate on training data is often under valued. Consensus measures the rate of agreement between multiple annotators (human or machine). A consensus score is calculated by dividing the sum of agreeing labels by the total number of labels per asset. Consensus Consensus Workflow Enable consensus and customize the consensus parameters (Automatic) Random labels are distributed across labelers at random intervals Keep track of overall consistency and investigate any dips in quality by looking into individual labeler and label consensus scores Consensus Example To configure consensus, you can customize the percentage of training data and the number of labelers to test. Monitor overall consistency with the Consensus histogram. Breakdown the consensus score by asset. Here you can compare the labels of a particular image. This is a collaborative part of research and development and that enables AI teams to innovate on projects. Points of disagreement can be opportunities to think about the problem from other perspectives, which in turn can spark new solutions. Typically, there is an agile progression to data labeling. Often the most successful AI models in production have had labeling tasks that evolved over time to better fit their use cases. Meaning, as visual engineers get more familiar with the problem they are able break it down and solve it in steps with models curating images for other models down the line to drive efficiencies. In other words, effective modeling plays to the repetition and precision strengths of machines by creating a production line of atomic tasks. Review Having control of data with a tight feedback loop between humans and machines empowers teams to build great ML applications. Visualizing data is paramount, not only for trouble shooting quality issues, but also for developing a deep understanding of and a sophisticated solution for a machine learning problem. Review Workflow Reviewing is a manual rather than automated process. It is part of keeping a human in the loop. Choose which labels to review Review, modify or re-enqueue labels Review Example Filtering options help managers prioritize which labels to review. As depicted in the image below, available filters include labeler, consensus score, label contains, and more. Reviewers are often the top performing labelers or internal knowledge experts. To review a label, there are thumbs up and down icons. The reviewer also has the option to modify or correct the labels on the spot. Additionally, clicking on the three vertical dots allows you to delete and re-enqueue the label, view the benchmark when applicable, and copy the link to send to other collaborators. Conclusion Creating training data is often one of the most expensive components of building a machine learning application. Properly monitoring training data quality increases the chance of having a performant model the first time around. And, getting labels right the first time (first pass quality) is far cheaper than the cost of discovering and redoing work to fix the problem. With world class tooling at your fingertips, you can ensure your labeling maintains the level of quality you need to get the modeling results you want. With Quality Assurance processes data scientists can: Monitor overall consistency and accuracy of training data Quickly troubleshoot quality errors Improve labeler instructions, on-boarding, and training Better understand the specifics to their project on what and how to label Get Started with Labelbox Visit to explore Labelbox for free or about an enterprise solution for your business. www.labelbox.com speak to one of our team members

How to Measure Quality when Training Machine Learning Models

A Guide to Outsourcing Without Compromising Data Quality In order for data science teams to outsource annotation to a managed workforce provider — also known as a Business Process Outsourcer (BPO) — they must first have the tools and infrastructure to store and manage their training data. Data management tools and infrastructure should support R&D product management teams, outsourced labeling teams, and internal labeling and review teams working together in a single centralized place with fully transparent oversight. Scaling with Subject Matter Expertise There is a direct relationship between the volume of your training data and the size of your annotation team. The alternative to scaling your annotation workforce through outsourcing is hiring an internal team of labelers. While this is an expensive option, it is sometimes the only option. For example, scaling sensitive training data, such as medical data with HIPAA protection, might require a solely internal labeling workforce. Continuing with this example, medical data, such as CT scans, would need to be labeled by radiologists who have the necessary medical expertise to properly interpret the data. The concern with outsourcing labeling work requiring subject matter expertise is that a BPO will not be able to provide specialized labelers. While there is good reason to be skeptical about outsourcing complex or niche datasets, BPOs cover a surprisingly wide spectrum of subject matter expertise and with a little bit of research, you might find one that offers a specialized annotation service capable of labeling your dataset at a fraction of the cost it would take to hire an internal team. Grant Osborne, Chief Technology Officer at , a comprehensive esports community platform powered by AI, describes his decision-making process behind using Labelbox’s outsourcing feature to scale annotations within the competitive gaming industry. Gamurs is developing an AI coach for professional video game players. The AI coach will help to improve gamer performance by learning from similar examples in which players are underperforming and suggest ways to enhance the gamer’s performance. Grant originally considered crowdsourcing gamers from their large social media following to label their favorite games of choice. At first, he looked into a number of popular crowdsourcing tools but quickly rejected this option because their revenue generation comes from annotation. Gamurs “These tools charge for storage based on the number of bounding boxes. And since we will have millions of labels, this pricing structure is impractical.” He then considered building a cheap in-house tool and hiring an internal team of labelers, until he spoke with Brian Rieger, Co-founder and Chief Operating Officer at Labelbox. Gamurs needed a platform for uploading and regulating images of multiple games with object detection. In contrast to other commercial labeling tools, Labelbox’s pricing structure is based on a three-tier system: Free, Business, and Enterprise. The subscription tiers are categorized by number of ML projects and dataset size. These tiers vary in price and access to certain platform features. “My favorite part about Labelbox is the ease of the API. Having a developer focused API makes it effortless to productionize models.” Dota2 Annotations on Labelbox (photo: Gamurs) “We needed a machine learning pipeline solution and Labelbox was it!” — Grant Osborne, CTO at GAMURS Unsurprisingly, Grant was initially dubious about outsourcing specialized gaming actions on Dota2 or League of Legend to a BPO. “We wanted to have an internal labeling team because the computer actions are complicated. How are we going to have an external company used to labeling simple objects, like stop signs and trees, label our games? However, Labelbox’s BPO partners told us to just send over a manual and they’d handle getting a dedicated annotation team up to speed.”  “Labelbox recommended two BPOs that would best fit our needs and said there were more if we were interested. The BPOs estimated that it would take ~3–4 weeks to get everyone completely trained. While this estimation was a bit optimistic for how complicated the material is, they were able to finish the training cycle in ~4–5 weeks.” Despite the drastically different cost quotes from the two BPOs (with one at 1.5–2 cents per bounding box and the other at 10–12 cents per bounding box), Gamurs still decided to use a mixture of both BPOs with a 20 person labeling team from the first and a 10 person labeling team from the second. League of Legends Annotations on Labelbox (photo: Gamurs) “We will probably do a combination of BPOs based on their strengths per game. We will get them to do consensus and if one BPO is better at quality assurance but slower at labeling, we will use them to cross review the other team’s work.” Scaling with Data Quality The inverse misconception of outsourcing subject matter expertise is believing that all human labelers are equal when it comes to annotating an extremely simple dataset. This perspective often downplays the importance of data quality in labeling. Read the section to learn how training a deep convolutional object detection model to identify something as simple as a pumpkin is actually much more complex than you might guess. Even with simple labeling tasks, to ensure data quality, you must be able to oversee the consistency and accuracy of labels across annotators and over time.  Labeling at scale without compromising data quality requires transparency throughout your labeling pipeline. Teams of data scientists who are outsourcing on locally run in-house tools often send data to several different annotation services where labeling happens locally, sometimes in a variety of countries, and the data scientists must rely on these labelers to send the file via email or to do uploading acrobatics via Dropbox.  Consequentially, the data becomes fragmented, disorganized, and difficult to manage, leaving it vulnerable to problems in data security, data quality, and data management. In order to monitor labeling accuracy and consistency of outsourcing services in real time, companies, like switch from managing their annotation workforce on a homegrown tool to managing it through Labelbox. is best in the world for integrating your in-house labeling and review teams with your outsourcing team in one centralized place. What’s a Pumpkin? SomaDetect Labelbox Not all Labelers are Equal Factors that differentiate outsourcing goes far beyond only the subject matter expertise it services. Labelbox has hand selected the top BPO firms based on the following criteria: Pricing transparency Quality customer service Diversity in company size, regions of service, range of skills, and styles of engagement We spoke with Michael Wang, Computer Vision Engineer at Companion Labs, who spoke to us about his experience outsourcing on Labelbox with one of our recommended BPO partners. He explained why outsourcing with a dedicated team of labelers, as opposed to crowdsourcing random human labelers, produces higher quality training data. “Connecting directly with a dedicated team of outsourced labelers helps you and the clients figure out how to label the project and labelers get better over time. With random labelers, you have to start the learning curve from scratch every time. Dedicated teams of labelers come to understand your project and when you explain something it gets communicated across the entire team.” — Michael Wang Before choosing Labelbox, Companion Labs had compared Labelbox to a leading competitor by trying out both labeling service APIs in terms of quality metrics, time, and effort to label their project. Michael said that Labelbox has a higher quality outsourcing pool than the well-known competitor who crowdsources.  When asked how he chose who to work with amongst Labelbox’s partner BPOs, he explained that Labelbox provided two recommendations, which he evaluated both on quality and cost metrics. “Both providers were pretty amazing in terms of quality so choosing came down to the cost.” Outsource on Labelbox Managed workforce services are often an instrumental part of making an AI project successful. Therefore, we at Labelbox want to enable managed workforce providers to render their services in as frictionless a way as possible. With Labelbox, teams of data scientists, annotators, and product managers can transparently manage small projects and experiments to super large projects all on a single platform. Our focus is to make our customers as successful as they can be at their AI projects. Our customers are businesses of all sizes building and operating AI. We have worked with a lot of managed workforce providers and it is clear to us that the best providers stand out from the rest in the service they provide and the customer-centric nature of their business. We have hand selected BPO partners so that our customers can have high-quality labeling services rendered directly inside of their Labelbox projects.  On Labelbox your internal and outsourced labelers can work together on a labeling project. It’s so cohesive that there’s literally no seam between the two! seamlessly An Effortless Two-step Process Contact one of our Workforce Partners listed . here Share your Project with them by adding their “Firm Provider ID” (Provided by the Workforce Partner). That’s really it! Your Project will show up as a shared Project in the Workforce Partner’s Labelbox account where they will be able to add and manage their own labelers in your project. They will have access to annotate, review annotations, and manage their labelers. The best part is that your internal team will be able to monitor their performance with complete transparency. For more information . check out our docs Get Started with Labelbox Visit to explore for free or about an enterprise solution for your business. www.labelbox.com Labelbox speak to one of our team members Originally published at medium.com on December 13, 2018.

How to Scale Training Data

Ready to learn how to train your AI in less than a day without writing a single line of code? If you are sitting on a pile of data that you want to make useful, but AI seems too daunting of a task to get started, then this article is for you. Or if you have experienced the trenches of AI and are ready to make the model training process a whole lot simpler and faster, you will also find this article eye-opening. Off-the-shelf algorithms have gotten so good that they have ushered in a new paradigm of accessibility in machine learning. The conversation in machine learning has evolved from how to build the technical architecture of a prediction model to how we can most effectively train an off-the-shelf state-of-the-art model on a specific business application. To show you the simplicity and effectiveness of training an off-the-shelf model and to celebrate the fall season, will teach you how to train a deep convolutional object detection model called SSD MobileNet V1 to identify pumpkins. Labelbox What’s a Pumpkin? If you can answer this question then you can train a model that can, too. Instead of labeling images from scratch, we leveraged an off-the-shelf model that has already been trained to accomplish a similar task. SSD_mobilenet_V1 is an object detection model, which we fine-tune for customer specific object detection applications. Being able to visualize and correct a model’s erroneous initial predictions in Labelbox not only saves time and money but also teaches the model trainer to better define his/her objectives and better control data quality. Visually inspecting the model’s initial predictions of our pumpkin data forced us to think more deeply about what a pumpkin is and which pumpkins we wanted our model to recognize. In the United States, pumpkins usually refer to the round orange squash that is carved into jack-o’-lanterns for Halloween or baked into pies on Thanksgiving. As it turns out, pumpkin qualities, such as shape, size, texture, and color, vary widely making identifying pumpkins much more complicated than we originally thought. We began questioning whether we really knew what a pumpkin was and how to tell it apart from squash or gourds. False positives in the initial prediction In the side-by-side images above, the SSD MobileNet V1 model incorrectly identified the cat’s orange eyes and the dog’s orange fur to be pumpkins. This bias of orange suggests that the pumpkins in our dataset are disproportionately orange. In the image below, the model correctly identified the carved pumpkins but failed to identify the uncarved smaller pumpkins, suggesting that the pumpkins in our dataset are disproportionately jack-o’-lanterns. False negatives in the initial prediction It wasn’t just our own conception of a pumpkin that we had to think more deeply about, but also the extent to which we wanted our model to identify pumpkins. For example, the initial prediction model was able to identify a cartoon jack-o’-lantern, but not a black and white sketch of a pumpkin. Was the sketch a false negative or did it correctly interpret this sketch as not a pumpkin? The two images are similar in size, shape, ribbing, but differ in color and carving. Without the coloring, it is ambiguous as to whether the sketch is a pumpkin or a gourd. In addition to potential biases in our dataset, we noticed biases in our annotation process. Absence of a label will penalize predictions in those areas as false positives, biasing the model away from detecting these. For example, we labeled pumpkins in the foreground and ignored those in the background, we labeled pumpkins in focus and ignored blurry pumpkins, and we labeled well-lit pumpkins over those in dark shadows. There is no good or bad choice when deciding whether to label one, the other or both options, it depends on your desired outcome. The key is to be consistent in your labeling choices over time and across labelers. Build your own AI in 3 steps With the following steps, you can build your own AI in under 24 hours. Step 1. Import data Step 2. Configure interface Step 3. Iterate: label-train-evaluate Interface Configuration Labeling Environment The setup (step 1 and step 2) takes just a few minutes. We started by creating a new project in Labelbox and uploading 500 pumpkin images. Next, we configured the labeling interface by typing in the segmentation title “Pumpkin” and choosing “Rectangle” as the bounding geometry type. We need to use rectangles rather than another bounding option like polygons because SSD_mobilenet_V1 is trained on rectangular bounding boxes. The fine-tuning stage (step 3) is the iterative process of labeling, training and reviewing. With just a click of a button, we trained the model on 44 images, which resulted in an initial prediction accuracy of 43%. We corrected the model’s initial labeling errors and retrained it on 181 images of pumpkins which increased the accuracy to 66%. This process can be continued to further increase model accuracy. First and second iteration of the Automatic Model Builder Automatic Model Builder One-Click Model Training with the Automatic Model Builder Labelbox is soon releasing an Intelligent Labeling Assistance (ILA) feature, which functions behind the scenes and requires no technical background to get started. There are two key components to ILA: the Automatic Model Builder (AMB) and the . Essentially, AMB performs transfer learning to fine-tune a deep neural network on the data and ontology specific to your particular project. In other words, ILA automatically customizes pre-trained state-of-the-art computer vision models to fit the particular needs of your business application. ILA is split into two components so that Labelbox can support the same prediction features on your own model as it does an off-the-shelf one. predictions API AMB accelerates your training process by giving you a starting point way ahead of the game. When you import your data, the Automatic Model Builder tool will generate a prediction model that pre-labels your data immediately. Not having to train your model from scratch on basic feature extraction layers, such as differentiating the foreground from the background or understanding object edges, reduces the total amount of time that you have to spend in the iterative labeling and training process so that you can get your model up and running faster. By having a model make an initial prediction, you can quantify and visualize the initial accuracy of your model and spend your time and budget more effectively by skipping the work of initial training and jumping right into fine-tuning. Your model will correctly process the obvious use cases of your data set, alleviating your domain experts from rudimentary labeling, and freeing up their time for the more complicated or ambiguous cases. Being able to visualize how your model is interpreting your data during training will help you make the corrections you need to get the outcome you want. For more information on how to ensure high-quality training data check out our article . It All Boils Down to Training Data We are releasing AMB with “Single Shot Multibox Detector (SSD) with MobileNet” for visual object segmentation. This is just the beginning. Stay tuned as we extend our AMB feature to incorporate more pre-trained cutting-edge models from . Tensorflow detection model zoo Business Applications In conclusion, you can apply this Automatic Model Builder tool to your business in under a day and without writing any code. It took us only 2–3 hours to get our AI to identify pumpkins at 66% accuracy. With a bit more time and whole lot more data, you can quickly train your AI to have an extremely accurate outcome. Whether you are trying to identify tumors or have your self-driving car recognize barriers, Labelbox’s AMB feature will save you time and money by automatically transferring learning from pre-trained state-of-the-art models to your business application. Visit to explore Labelbox for free or about an enterprise solution for your business. www.labelbox.com speak to one of our team members Originally published at medium.com on November 20, 2018.

AI is More Accessible Than You Know

Is your model not performing well? Try digging into your data. Instead of getting marginal improvements in performance by searching for state-of-the-art models, drastically improve your model’s accuracy by improving the quality of your data. Since most data scientists are adapting off-the-shelf algorithms to specific business applications, one of the most difficult challenges that data scientists face today is creating a continuous workflow that consistently feeds high-quality training data into their algorithms. At the same time, your model is learning and you want to be able to leverage this intelligent model to label the rest of your data set. Building the infrastructure to do annotation that integrates with your model and managing the workflow is the most challenging part of machine learning. Iteration => Accuracy & Consistency The axiom of can be masked in training. Even when fed random noise, such as random labels or unstructured pixels, certain models are capable of overtraining to the point of attaining 0% training error ( ). This is because recent high-capacity models like deep neural networks can memorize even massive data sets. While these models do not commit errors during training, when tested, they perform no better than random guessing. garbage in garbage out Understanding Deep Learning Requires Rethinking Generalization Therefore, iteration and rigorous QA/QC processes are essential to a proper data labeling workflow. “Quality evaluation methods can be classified in three main families: (i) automatic, (ii) by direct inspection of the job provider and (iii) methods using the crowd itself as evaluator” ( ). Since, in most cases, automated evaluation without human input is either impossible or guarantees minimal quality, we will discuss how to implement QA/QC methods of the latter categories to help improve the confidence in the quality of your training data. Worker Ranking Determination in Crowdsourcing Platforms using Aggregation Functions Test questions Direct inspection Consensus Test questions and direct inspection are QA/QC methods that fit into category (ii) where the job provider, or data scientist, is directly responsible for evaluating quality. Test questions is a standard technique amongst companies. It refers to a set of data that is correctly labeled by the data scientist and then distributed randomly amongst labelers to test their accuracy. Direct inspection is the process of visually inspecting your labeled data to gauge accuracy. Visual screening is a basic functionality that everyone should have to preprocess data and post-label review for accuracy. In the article, Pete Warden recommends randomly browsing through your data. This basic practice can reveal valuable information about your data set, such as “unbalanced number of examples in different categories, corrupted data (for example PNGs labeled with JPG file extensions), incorrect labels, or just surprising combinations.” For more practical tips on improving your data quality, read his article . While most open source tools do not provide this essential feature, Labelbox is a repository of labeled data where you can visually browse and manage your data in one place. Why You Need To Improve Your Training Data, And How To Do It, here While the QA/QC methods of category (ii) are extremely useful, they have two inherent drawbacks. First, they are inherently unscalable since the resources of the job provider, or data scientist, to evaluate the accuracy of crowdsourced labels is finite. Second, in order to perform these methods, the correct answers must already be known. Consensus, on the other hand, is both inherently scalable and useful when the correct answers are unknown. Consensus requires multiple different annotators to provide labels for the same piece of data. With that information, consensus computes Intersection Over Union (IOU) to average out idiosyncrasies across labelers and get better attenuation of the signal. In other words, the answers to the same question are compared to determine the rate of agreement. High agreement is indicative of a high-quality data set, while low agreement typically points to poor data quality, but can also be indicative of ambiguous examples. Labelbox offers a built-in consensus tool so you can monitor your quality metrics in real-time. Read more about how the Labelbox Consensus tool works . here Consensus Diminishing Marginal Returns Google published a study that showed that even when you think you have enough data, adding more can make your model perform even better ( ). And yet, the answer is more complicated than more is always better. The Unreasonable Effectiveness of Data The core question to ask is, not whether you have enough data, but whether you have hit the efficient frontier where the marginal costs of labeling exceed the marginal gains in model performance. To visualize this, plot the model’s performance over time on held-out evaluation data. For example, start with 1000 samples to train your model and evaluate it on 200 held out samples to measure your starting accuracy. Then collect another 1000 samples and repeat the experiment with the second set. The model is expected to do better with 2000 examples because it is learning to see natural variations in the data and filter out idiosyncrasies while better attenuating to signal. Workflow Transparency It is common practice to use a labeling service where you outsource the data and get labeled data in return. However, if you are outsourcing your data labeling, but have no way of measuring the quality of the labeling service, you are essentially gambling with your investment. Outsourced labeling services can be a good go-to for basic object classification, like labeling cars or dresses. If you need to generate a large labeling task force on a specific subject matter, there are different Business Process Outsourcing (BPO) firms that can accommodate particular specialized knowledge categories. Through , you can connect with our partner BPOs, monitor the quality of your outsourced data labeling services, and create and manage your own workflow all on a single unified platform. Labelbox To Sum it Up, Clean it Up Your model is only as good as your training data. Now that you know how to ensure that your training data is consistent enough, accurately labeled, and sufficient in size, go clean it up! Visit to explore Labelbox for free or about an enterprise solution for your business. www.labelbox.com speak to one of our team members Originally published at medium.com on November 12, 2018.

It all Boils Down to the Training Data

September brought many new features and updates to Labelbox. But most notable is that the image annotation interface has undergone a massive upgrade. Fast & Intuitive Interface Configuration We’ve added a much simpler way for you to customize the image labeling interface. Introducing the Form-based Interface Configuration Tool. With this new interface configurator, you can easily set up your labeling task using all of the available segmentation & classification tools using an intuitive web form instead of coding JSON. Still prefer the JSON? It’s available by switching the configuration screen from ‘FORM’ to ‘JSON’ at the top. Annotate images with pixel level accuracy Sometimes you need pixel perfect annotations. Labelbox now offers advanced pixel-wise annotation tools, including a blazing fast (and browser side!) superpixel tool. Also available are the complimentary brush and eraser tools. Empower your annotation team with Labelbox’s best-in-class pixel-wise annotation toolkit. A pixel accurate brush tool is also available for use. It’s mechanics are consistent with most digital brush tools available in creativity software today. The brush tool is often used to do final touch-up after using the superpixel tool. Brush For many pixel level annotation tasks the goal is to label a homogenous grouping (or groups) of pixels. An example of this is detecting clouds in satellite imagery. For these tasks, the superpixel tool is often an extremely efficient method for annotation. Superpixels are computed groupings of pixels, and in the case of Labelbox are computed in the browser so that you can use this tool seamlessly on your private data. For optimal performance of superpixel, we recommend using the Google Chrome Browser. Superpixels The eraser tool removes the assigned class from the pixels it is applied to. Eraser Bounding boxes, polygons, lines, points and nested classifications Get Started with Labelbox Visit to explore Labelbox for free or about an enterprise solution for your business. www.labelbox.com speak to one of our team members

Introducing best-in-class image annotation tools for computer vision applications

The Critical Role of Data Annotation in Shaping the Future of Generative AI

Крытычна важная роля анатацыі даных у фарміраванні будучыні генератыўнага ІІ

Критичната роля на анотирането на данни при оформянето на бъдещето на генериращия ИИ

Den kritiske rolle for dataannotering i at forme fremtiden for generativ AI

El papel fundamental de la anotación de datos en la configuración del futuro de la IA generativa

Le rôle crucial de l’annotation des données dans la définition de l’avenir de l’IA générative

Il ruolo critico dell'annotazione dei dati nel plasmare il futuro dell'intelligenza artificiale generativa

生成型 AI の未来を形作るデータ注釈の重要な役割

생성 AI의 미래를 형성하는 데 있어서 데이터 주석의 중요한 역할

Karolo ye Bohlokwa ya Tlhaloso ya Datha go Bopeng Bokamoso bja AI ya Tlhagišo

Rolul critic al adnotării datelor în modelarea viitorului IA generativă

Rozhodujúca úloha anotácie údajov pri formovaní budúcnosti generatívnej AI

Den kritiska rollen för datakommentarer i att forma framtiden för generativ AI

As part of my continuous work to see how much I can do with just CSS (see other work such as the ), I decided to try and recreate Windows 98 using nothing else apart from CSS and HTML. Did anyone ask for this? Not really? Is it fun to try and see what you can accomplish with just CSS? Yes, sort of. Was it time-consuming? Unfortunately yes. CSS-only Minecraft Chicken As a quick note - this is a desktop Windows recreation, so it is of course optimized for desktop viewing. Ironically, though, it probably won't work on Windows 98, since you would have to use a very old version of Internet Explorer to view it. since it is better viewed in full-screen mode. In this demo the things which I thought were cool included: The link to the demo can be found here with just CSS - although no scorekeeping. Minesweeper and logout with the memory of who is logged in using the brand new CSS parent selector. Login Animated . update process . Minimizing, maximizing, and closing windows It is however worth noting that the things that are hard or just downright impossible to do with only CSS, such as: Drag and drop - not possible at all with CSS. Multiple conditions - for example, hard to say that a window should be both maximized and put on top at the same time. Click anywhere to close - you have to click the start button to close and open it. You can't just click anywhere else to close it. Multiple conditions in general - CSS doesn't really have an AND operator. How to build Windows 98 in CSS So, the first thing I wanted to get right about this version of Windows 98 was the look and feel. I'm using some pretty cool Windows 98 icons (which I do think should make a comeback), as well as the standard Windows 98 color scheme. To get the indented and out-dented feel I used quite a complicated box-shadow, as you can see here: .windows-box-shadow, .minesweeper .content > label {
    box-shadow: -2px -2px #e0dede, -2px 0 #e0dede, 0 -2px #e0dede, -4px -4px white, -4px 0 white, 0 -4px white, 2px 2px #818181, 0 2px #818181, 2px 0 #818181,  2px -2px #e0dede, -2px 2px #818181, -4px 2px white, -4px 4px black, 4px 4px black, 4px 0 black, 0 4px black, 2px -4px white, 4px -4px black;
}
.inverse-windows-box-shadow, .minesweeper .content > label:active {
    box-shadow: -2px -2px #818181, -2px 0 #818181, 0 -2px #818181, -4px -4px black, -4px 0 black, 0 -4px black, 2px 2px #e0dede, 0 2px #e0dede, 2px 0 #e0dede,  2px -2px #818181, -2px 2px #e0dede, -4px 2px black, -4px 4px white, 4px 4px white, 4px 0 white, 0 4px white, 2px -4px black, 4px -4px white;
} Everything else was relatively straightforward look and feel-wise. The key to making all of this work is checkboxes and radio buttons. Using Checkboxes and Radio buttons as a store of information in CSS Checkboxes and radios are the only way to store information in CSS. We can then use them to implement style changes. Checkboxes, when checked, allow us to enable or disable a single feature (like showing a window, maximizing a window, or clicking a minesweeper square). For things where there is only one option allowed to be active at a time (for example, which window should be on top) - we can use radio buttons. Both follow the same syntax in CSS, where we use the selector: :checked #windows-11:checked ~ .windows-11 .text {
    /* -- CSS here -- */
} Here, when the input is checked, it will affect its sibling's child - so we can apply some custom CSS. Importantly, since we can't easily style an HTML input, we use s to model the different features of Windows 98. #windows-11 .text label For example: <form id="windows">
    
    <input type="checkbox" id="login-screen-input" name="login-screen-input" />

    
    <label for="login-screen-input">Log Off</label>
</form> Here, the shown is associated with the checkbox . That means when you click the label, it will check the checkbox. This basically gives us free reign to track a user's clicks, and then use the checkbox status to show certain windows, in certain forms. The difficulty is you can only have label associated with input. label #login-screen-input :checked one one That means in a scenario where a button is supposed to open the window and place it on top of all other windows, you'd have to use Javascript since this will require tracking two states - the of the window, and whether it is open or closed. This is a major blocker in implementing CSS-only versions of complex UIs. z-index Using the parent selector to track who is logged in Since we have a login screen in a for when a user logs out, we can't use sibling selectors to easily track who is logged in. We can still use statuses to track this, but the inputs are too deep in our DOM to affect their parent's sibling's CSS. Fortunately, we can use the new CSS parent selector for just this task: div :checked #login-screen:has(#login-window .select-box #zark-muckerberg:checked) ~ #start-bar .zark-muckerberg,
#login-screen:has(#login-window .select-box #donald-trump:checked) ~ #start-bar .donald-trump,
#login-screen:has(#login-window .select-box #spiderman:checked) ~ #start-bar .spiderman {
    display: inline;
    padding-left: 0.5rem;
} Here, if has a div, we can use it to display the user name in the start bar, despite these checkboxes being deep within the DOM. This is pretty neat, and a useful way to use parent selectors if you ever wish to accomplish recreating a CSS-only version of a Windows operating system. #login-screen :checked There is no CSS AND Selector Much to my dismay, there was no way for me to create a CSS selector using chained checked boxes. For example, consider this situation where we apply some CSS based on a state: AND :checked #minesweeper-box-1-1:checked ~ .content > .minesweeper-box-1-1 {
    
} This works fine, but what if we want to check if two minesweeper boxes next to each other are checked, before applying CSS? I thought, logically, that the selector would only continue if both were checked - so tried this: #minesweeper-box-1-1:checked + #minesweeper-box-2-1:checked ~ .content > .minesweeper-box-1-1 {
    
} But unfortunately, that doesn't work. So while we have a way to track states in CSS, . That's disappointing, but it doesn't limit us too much for our Windows 98 implementation. it's quite hard to track multi-conditioned checkbox states to create logical statements and styles based on that Achieving Windows 98 Text Windows 98 text is anti-aliased. To remove anti-aliasing (at least for some browsers), and achieve that classic, crisp, Windows 98 finish, I used the following CSS: not body {
    -webkit-font-smoothing: none;
    -moz-osx-font-smoothing: grayscale;
} Recreating Minesweeper So one of the major undertakings in this project was recreating Minesweeper. I kept the grid relatively small (to maintain my sanity) - but I had to make my own Minesweeper map created out of labels. Each of these labels mapped to an input, which tracked if a cell had been clicked or not. If a mine is clicked, it's game over and you can't interact with the board anymore. As there were ~56 Minesweeper cells, we needed an equivalent of ~56 Minesweeper inputs. Tracking that all in CSS required of CSS, but the overall result is pretty cool looking. a lot Overall, this follows the same logic as the previously mentioned checkbox and radio trick - so conceptually it's not any more complicated than anything else we've done. Conclusion I hope you've enjoyed this guide. Doing this reminded me of how web development was about 10 years ago when things were a lot harder to accomplish and required a lot of manual creation of DOM elements. It's fun to see what can be achieved in CSS all these years on (including the parent selector). Is this a realistic way to create web applications? Not really in terms of speed, and not yet in terms of functionality, but CSS did a lot more than I thought it would be able to, and I'm pretty happy with the results. If you enjoyed this, . please consider following me on Twitter Also published . here

Recreate Windows 98 With CSS

70-Page Report on the COCO Dataset and Object Detection [Part 3]

Take a look at the to quickly find common resources and/or assets for a given dataset and a specific task, in this case dataset=COCO, task=object detection. We are building a dataset-first marketplace focusing on the end-to-end machine learning pipeline, one where data and assets can be shared and traded. The marketplace will contain all that the contains (and much more for a lot more datasets) report report I’m open to suggestions, questions, and criticism — let’s start a conversation. I have broken up the report into the following blogs: Part 1 (this one): COCO Summary Card. Each link will take you to the longer report where you can learn more. The next 3 parts represent a specific section in the report. Part 2: This part is about COCO and examples and tutorials of tools and platforms used to work with COCO (or object detection tasks). Part 3: Process: This part is about the tools and platforms that can be used for different phases of data preparate or data processing involved in vision, object detection, and specifically COCO-related tasks. It will also discuss synthetic data and data quality. Part 4: Models: This part is about a quick introduction to some pre-trained models and some corresponding readings. COCO Summary Card (Common Objects in Context) COCO COCO is a large-scale object detection, segmentation, and captioning dataset. The dataset classes include 80 pre-trained objects. Website: https://cocodataset.org/ Github: GitHub — cocodataset/cocodataset.github.io Paper: Microsoft COCO: Common Objects in Context dataset with evaluation metric Average Precision (AP) Benchmark : Object Detection, Panoptic Semantic Segmentation, Keypoint Detection, DensePose Tasks Tools and platforms used to work with COCO (or object detection tasks) : Explore with tutorials FiftyOne, DataTorch, Know Your Data (KYD), OpenCV, OpenVINO, CVAT, Roboflow, SuperAnnotate, OpenMMLab, Coral, Amazon, Facebook, Google, Microsoft, NVIDIA, Weights and Biases. Open-source tools and Paid platforms to perform the following steps. : Processing Data collection : CVAT, LabelImg, Label Studio, Makesense.AI, OpenLabeling, Dataloop, Hive Data, Labelbox, Scale AI, SuperAnnotate, V7, Ximilar Data labeling and annotation Data cleaning : Albumentations, AugLy, Augmentor, autoalbument, DALI, Data Augmentation for Object Detection, Imgaug, Kornia, MXNet, Tensorflow, Transforms (PyTorch) Data transformations Data visualization : Cvedia, Neurolabs, Synthesis AI, Unity, UnrealROX Synthetic data Data quality To answer what objects are in image X and where are they? (You’ll find a brief summary and readings.) : Models RCNN SPPNet Fast RCNN and Faster RCNN Pyramid Networks / FPN Mask RCNN YOLOv1 SSD RetinaNet YOLOv3 YOLOv4 YOLOv5 DINO (and DETR) Swin Transformer V2 Vision Transformer Adapter YOLOR If you have feedback please review this link ( ) and email me at samin@reasonets.com. Looking forward to starting a conversation. Marketplace — Coming Soon | ReasoNets Next, Part 2

70-Page Report on the COCO Dataset and Object Detection [Part 1]

The term ‘curation’ is commonly associated with museums or libraries, not data science. However, much like the work that’s done on rare paintings or books, data curation tools make the most important data easily accessible to engineers as they build complex machine learning models. Without curation, data is difficult to find, analyze, and interpret. Data curation tools provide meaningful insights and enduring access to all your data in one place. In this article, we’ll dive into the importance of data curation for computer vision specifically, as well as review the top data curation tools on the market today. What is data curation? Data curation is the act of organizing, enhancing, and preserving data for future use. In machine learning, data curation describes the management of data throughout its lifecycle: from its collection and initially storage, to the time it is archived for future re-use. This process is all the more important for computer vision engineers, who deal with massive amounts of visual data on a daily basis. Instead of using manual methods such as writing ETL jobs to extract insights, data curation tools provide a streamlined way to access the right data whenever you need to. The importance of data curation for machine learning Under the hood, data curation tools directly influence computer vision model performance. Using data curation tools, engineers can get a better understanding of the data they’ve collected, identify the most important subsets and edge cases, and curate custom training datasets to feed back into their models. The best data curation tools enable you to: Make it easy to obtain insights on key metrics, as well as the general distribution and diversity of your datasets regardless of sensor type and format. Visualize large scale data: Quickly search, filter, and sort through the entire data lake by making all features queryable and easily accessible. Enable data discovery and retrieval: Identify the most interesting segments within your dataset, and manipulate them within the tool to create completely customized training sets. Curate diverse scenarios: The tool should fit well within your existing workflows and toolset. Seamlessly integrate: What are the best data curation tools for computer vision? With an overwhelming amount of AI products and platforms popping up year after year, how do you know which will provide the most value? Read on below to find out which data curation tool is the best fit for your computer vision project. 1. Aquarium Learning Aquarium is a data management platform that aims to make it easy to identify labeling errors and model failures. With Aquarium, users can version and combine model predictions with their ground truth. Aquarium is especially focused on curating and maintaining training datasets, catering less to raw data management use cases. This is because data exploration in Aquarium is predominantly tied to model predictions and ground truth labels. Users can access Aquarium via their cloud platform or API. However, they currently do not offer on-premise or VPC deployments, and there are no external integrations. - Aquarium supports image, 3D, audio, and text data. They also support multiple annotation types, such as classification, detection, and segmentation. Wide range of use cases - Users can manipulate evaluation thresholds and obtain interactive visualizations to obtain required samples quickly. Interactive model evaluation - Users can collaborate with each other on the Aquarium platform to build data subsets, associate them with issues, and identify new data for annotation. Collaborative features 2. FiftyOne Developed by Voxel51, FiftyOne is an open-source tool to visualize and interpret computer vision datasets. The tool is made up of three components: the Python library, the web app (GUI), and the Brain. FiftyOne does not contain any auto-tagging capabilities, and therefore works best with datasets that have previously been annotated. Furthermore, the tool only supports image and video data, and does not work for multimodal sensor datasets at this time. Unlike other tools, FiftyOne is designed to be used by individual developers rather than teams, functioning like a programming IDE. Today, the platform lacks collaborative features; for example, a single instance cannot host multiple user accounts. FiftyOne taps into TF and Pytorch dataset zoos to provide access to a variety of open datasets and open-source models. Model & dataset zoo - - Via the Brain, a separate closed-source Python package, users can quantitatively assess the uniqueness, mistakenness, and hardness of data. Advanced data analysis - FiftyOne directly integrates with popular annotation tools such as . They also have tight integrations with Jupyter and Colab Notebooks making it easy for users to run FiftyOne through Python notebooks. External integrations LabelBox , 3. Scale Nucleus Launched in late 2020 by , Nucleus is one of the newest data curation tools to hit the market. The Nucleus platform allows users to collaboratively search through image data for model failures. As of now, Nucleus only supports image data, with no support for 3D sensor fusion, video, or text data. Scale Users can access Nucleus via their cloud platform, API or Python SDK. Currently, Nucleus does not support on-premise deployability. - Users can search for visually similar images based on one or multiple base samples and associate custom tags with them. Visual similarity - Using the Nucleus SDK, users can create flexible metadata schemas. Nucleus provides smart methods to detect and create schemas using the annotation format provided. Metadata schemas - Users can create model entities and associate corresponding runs with them. Hence, models can be versioned based on runs (dataset & predictions). Model versioning 4. Clarifai Clarifai is an end-to-end solution for labeling, searching & modeling unstructured data. One of the first AI startups, they provide a platform for modeling image, video, and text data. While Clarifai’s original focus was enabling users to build custom models, they’ve recently added several data curation features including auto-tagging, visual search, and annotations. Ultimately, Clarifai is more of a modelling platform and less of a developer tool. They are best suited for relatively inexperienced teams getting started with ML use cases. - Clarifai offers a broad library of pre-built AI models, including anything from food to facial recognition. Ready-to-use model gallery - The platform supports for image, video, and text data. Wide range of data types - With the platform, users can customize or retrain existing models or create new ones from scratch. Model customization - In addition to their modelling platform, Clarifai offers fully managed annotation services through their Scribe LabelForce data labeling service. Data annotation 5. SiaSearch is a data management platform for computer vision data. Consisting of a scalable metadata catalog and query engine, SiaSearch enables developers to easily search through visual data, add metadata to frames and sequences, as well as assemble custom subsets of data for training or testing. SiaSearch With deep roots in autonomous driving, the SiaSearch platform is used by many OEMs, Tier 1s and tech companies. Aside from autonomous driving, SiaSearch also has solutions for robotics, retail, and more. - One of the only tools that can support 3D sensor fusion data, SiaSearch can analyze large volumes of unstructured sensor data, providing insights at the frame and sequence level. Specialized in sensor data - SiaSearch employs a large catalog of pre-trained extractors to automatically add frame-level, contextual metadata to raw data. Additionally, SiaSearch provides a toolbox for quick extractor development, allowing developers to integrate their own extractors. Auto-tagging capabilities - The SiaSearch platform features a unique, proprietary architecture that combines numeric and sequence-based queries to enable noticeably faster performance. Fast performance - Users can access SiaSearch via their web-based GUI or programmatic API. SiaSearch also supports cloud or on-premise deployment for enterprise users. Flexible workflows & integrations Interested in data curation? The right data curation tool can dramatically reduce the time spent on manual processes, allowing engineers to focus on what really matters - building great models. Lead image via Tobias Fischer on Unsplash Originally published by Clemens Viernickel on: https://www.siasearch.io/blog/best-data-curation-tools-for-computer-vision and has been reposted with permission.

Stories, News & Mentions about LABELBOX

Labelbox’s Stories on HackerNoon