The Gartner Hype Cycle Report and the Future of Data

Gartner, a Connecticut-based technology consulting company with a well-established reputation for conducting research and gauging highly specialized industries, put out a new Hype Cycle report earlier this year.

A whole section of the report is dedicated to data management and provides an in-depth examination of how the world of today has come to rely heavily on innovations in data science and Machine Learning (ML).

Gartner identifies data labeling as one of the key factors responsible for the ongoing evolution of AI technology and rapid AI-powered product development. 

The report lists a handful of companies as today’s leaders of the data-labeling market – among them are Scale AI, Toloka, Appen, CloudFactory, Amazon MTurk, and Playment.

According to Gartner, these companies use both internal and external labelers to offer classification, segmentation, transformation, and augmentation procedures utilized to prepare training data for ML algorithms.

The report stipulates that the need for quality training data has skyrocketed over the past several years as more clients seek fast, reliable, and cost-effective labeling solutions. 

The Demand is on the Rise 

This ongoing demand is the result of both speedy growth of the AI industry itself, as well as the latest integration of new AI products into traditional business domains, from commercial shipping to agriculture.

The report explicitly states that many fields like search engine optimization, autonomous vehicles, Natural Language Processing, and e-commerce owe their success to human-handled data labeling.

Gartner concludes that human-in-the-loop models are of particular benefit because they do not require deep domain knowledge by the parties seeking labeling services.

The most important aspect of a successful labeling endeavor is dependable and easily scalable production pipelines, states the report.

The report further lays out a number of recommendations for those in need of data labeling:

• Choose between external or internal teams and leverage pros and cons of each one to get the highest quality results.
  
• When selecting a labeling provider, make sure your provider in question has reliable ways to test the labelers’ expertise and measure labeling accuracy.
  
• Seek transparent pricing models with costs estimated ahead of the projects.
  
• If you have engineers at your disposal, lighten their workload by letting them concentrate on working with code instead of labeling.
  
• Opt for those platforms that make use of real-time human-in-the-loop methodologies.

Choosing the Right Solution in the Global Market

Today, the data labeling market is expanding at an almost exponential pace. Last year, the $1 billion mark was reached, which is expected to be surpassed in the following five years, with the figures climbing all the way up to $7 billion by 2027.

This is an annual growth rate of almost 30%, and if this trend persists, some research powerhouses, including Grand View Research, project that the global data-labeling market will exceed $8 billion before 2030, which is roughly a quarter of the global AI market today.

The market’s growth is reliant on the evolution of specific data-labeling solutions that are used to annotate production-grade training data. A variety of solutions are available on the market today, with each one adhering to a particular data-labeling method.

All of these methods, in turn, belong in one of the three major categories: manual labeling (done by humans), automated labeling (done by computers), and hybrid labeling (a combination of the two). Currently, the most common labeling methods from these categories are as follows: 

Internal/in-house labeling (manual):

All labeling is handled by small, highly specialized teams. These teams consist of full-time staff, whose levels of expertise vary, so training may or may not be required. Control of the labeling process is maintained at every stage. 

Crowdsourcing (manual):

Considered the most economical method, it involves deploying a large group of labelers from across the globe, with each one contributing a small part. The tasks are completed, verified, and then aggregated to produce end results. Some task-specific training is usually required beforehand.

Outsourcing (manual and hybrid):

This approach entails hiring either a ready team or compiling your own team from scratch. In both cases, the work is carried out by external specialists.

When you hire a company, you get a turnkey solution with a ready set of tools, which is more expensive. When you hire labelers one by one, it’s cheaper but also takes more time and requires that you provide a set of labeling tools.

Synthetic Labeling (automatic or hybrid):

In this case, a training dataset is generated synthetically, which can subsequently be used as real data. While obtaining such a set takes little time and can be achieved with minimal supervision, a large amount of computing power is needed that most businesses do not possess.

Data Programming (Automatic):

This fully automated approach is when engineers prep and run code, and the machines do all of the labeling work. While it’s the least labor-intensive method, it tends to produce noisy datasets suitable only for weak supervision models or discriminative training.

At present, in-house labeling and crowdsourcing are arguably the two most popular approaches among the ones listed. Here’s their brief comparison:

If you would like more information on choosing the right strategy for your specific needs at hand, check out these useful resources that contain a more detailed analysis of the data-labeling solutions available in 2021:

• Data labeling guide from TheSequence
• Data labeling approaches and tools from KDnuggets
• eBook on data labeling approaches and use cases from Toloka