Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work 2 Related Work 3 Method and 3.1 Phase 1: Taxonomy Generation 3 Method and 3.1 Phase 1: Taxonomy Generation 3.2 Phase 2: LLM-Augmented Text Classification 3.2 Phase 2: LLM-Augmented Text Classification 4 Evaluation Suite and 4.1 Phase 1 Evaluation Strategies 4 Evaluation Suite and 4.1 Phase 1 Evaluation Strategies 4.2 Phase 2 Evaluation Strategies 4.2 Phase 2 Evaluation Strategies 5 Experiments and 5.1 Data 5 Experiments and 5.1 Data 5.2 Taxonomy Generation 5.2 Taxonomy Generation 5.3 LLM-Augmented Text Classification 5.3 LLM-Augmented Text Classification 5.4 Summary of Findings and Suggestions 5.4 Summary of Findings and Suggestions 6 Discussion and Future Work, and References 6 Discussion and Future Work, and References A. Taxonomies A. Taxonomies B. Additional Results B. Additional Results C. Implementation Details C. Implementation Details D. Prompt Templates D. Prompt Templates 2 RELATED WORK Taxonomy Generation. Prior work in taxonomy generation falls into manual and automatic approaches. Handcrafted taxonomies, beyond being expensive to construct, tend to be developed for specific downstream tasks (e.g., web search intent analysis [4, 20], chatbot intent detection [31]), or tied to the development of specific datasets [22, 25]. On the other hand, automated approaches scale better but either rely on term extraction from corpora to obtain labels, which may hinder interpretability and/or coverage [24, 33], or else require a set of seeds for the taxonomy in order to generate new labels [32]. TnT-LLM, in contrast, is automatic, abstractive (i.e., labels describe the corpus but need not be directly extracted from it), and does not require any seed labels. Moreover, TnT-LLM treats taxonomy generation and text classification as interrelated problems in an end-to-end pipeline, whereas prior work has tended to focus mainly on the quality of the taxonomy produced, without considering its downstream utility for classification. Taxonomy Generation TnT-LLM TnT-LLM Text Clustering and Topic Modeling. Text clustering and topic modeling “invert” the traditional approach of defining a label set, then applying the labels on the corpus. Given a set of documents, such approaches first group the documents into topical clusters using various definitions of textual similarity, then post-hoc label or summarize the clusters [1, 29]. While traditional approaches in theory accomplish the same goals as TnT-LLM, they suffer due to a lack of interpretability [5], as they typically do not assign intelligible labels to clusters. More recently, attempts have been made to overcome these problems by using LLMs for topic modeling [18, 30], though these approaches still require supervision through either a predefined taxonomy [30] or a seed set of topics [18]. Text Clustering and Topic Modeling. TnT-LLM LLMs as Annotators. Recent work has explored using LLMs to replace human annotation for labor-intensive tasks such as search relevance quality labeling [28], topic and stance detection [8], and various computational social science labeling tasks [34]. These studies have found that, in general, LLMs perform on par or even better than crowd-workers [15], often at a fraction of the cost. In the same vein, we explore using LLMs as annotators for text classification, although our main goal is to scale the process by distilling LLMs’ label-specific capabilities into more efficient, lightweight classifiers. LLMs as Annotators. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv Authors:
(1) Mengting Wan, Microsoft Corporation and Microsoft Corporation;
(2) Tara Safavi (Corresponding authors), Microsoft Corporation;
(3) Sujay Kumar Jauhar, Microsoft Corporation;
(4) Yujin Kim, Microsoft Corporation;
(5) Scott Counts, Microsoft Corporation;
(6) Jennifer Neville, Microsoft Corporation;
(7) Siddharth Suri, Microsoft Corporation;
(8) Chirag Shah, University of Washington and Work done while working at Microsoft;
(9) Ryen W. White, Microsoft Corporation;
(10) Longqi Yang, Microsoft Corporation;
(11) Reid Andersen, Microsoft Corporation;
(12) Georg Buscher, Microsoft Corporation;
(13) Dhruv Joshi, Microsoft Corporation;
(14) Nagu Rangan, Microsoft Corporation. Authors: Authors: (1) Mengting Wan, Microsoft Corporation and Microsoft Corporation; (2) Tara Safavi (Corresponding authors), Microsoft Corporation; (3) Sujay Kumar Jauhar, Microsoft Corporation; (4) Yujin Kim, Microsoft Corporation; (5) Scott Counts, Microsoft Corporation; (6) Jennifer Neville, Microsoft Corporation; (7) Siddharth Suri, Microsoft Corporation; (8) Chirag Shah, University of Washington and Work done while working at Microsoft; (9) Ryen W. White, Microsoft Corporation; (10) Longqi Yang, Microsoft Corporation; (11) Reid Andersen, Microsoft Corporation; (12) Georg Buscher, Microsoft Corporation; (13) Dhruv Joshi, Microsoft Corporation; (14) Nagu Rangan, Microsoft Corporation.

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Microsoft

What Does the Future of AI Model Training Hold?

Prior Approaches to Text Mining: Taxonomy, Clustering, and LLM Annotation

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

256K Tokens on One GPU? Jamba’s Engineering Magic Explained

TnT-LLM: Text Mining at Scale With Large Language Models

TnT-LLM: Automating Text Taxonomy Generation and Classification With Large Language Models

TnT-LLM: LLMs for Automated Text Taxonomy and Classification

Evaluating TnT-LLM: Automatic, Human, and LLM-Based Assessment

256K Tokens on One GPU? Jamba’s Engineering Magic Explained

TnT-LLM: Text Mining at Scale With Large Language Models

TnT-LLM: Automating Text Taxonomy Generation and Classification With Large Language Models

TnT-LLM: LLMs for Automated Text Taxonomy and Classification

Evaluating TnT-LLM: Automatic, Human, and LLM-Based Assessment

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps