Table of Links Abstract and 1 Introduction 2 Related Work 3 Method and 3.1 Phase 1: Taxonomy Generation 3.2 Phase 2: LLM-Augmented Text Classification 4 Evaluation Suite and 4.1 Phase 1 Evaluation Strategies 4.2 Phase 2 Evaluation Strategies 5 Experiments and 5.1 Data 5.2 Taxonomy Generation 5.3 LLM-Augmented Text Classification 5.4 Summary of Findings and Suggestions 6 Discussion and Future Work, and References A. Taxonomies B. Additional Results C. Implementation Details D. Prompt Templates 4.2 Phase 2 Evaluation Strategies To quantitatively evaluate text classification, we create a benchmark dataset with reliable ground-truth annotations as follows: Task and Annotation Reliability. We first assess the reliability of the label assignment task and the human annotations by involving multiple human annotators and calculating the interrater agreement (Cohen’s Kappa [6] between two raters and Fleiss’ Kappa [7] among multiple raters). We then resolve disagreements between human annotations by either voting or deliberation, and obtain a consensus human annotation for each instance. Then we use an LLM as an additional annotator to perform the same label assignment task, and measure the agreement between the LLM annotation and the consensus human label. Intuitively, this agreement captures how well the LLM is aligned with (the majority of) human annotators and how reliable it is for this label assignment task. Classification Metrics. We apply both human and LLM annotations on a small-scale corpus sample and calculate the conventional multiclass and multilabel classification metrics (e.g., Accuracy, F1) with human annotations as the ground truth. These metrics evaluate how the label classifier is aligned with human preferences on a small subset of the corpus. We then apply the LLM annotator on a larger-scale corpus sample and leverage the resulting annotations as the oracle to calculate the same classification metrics. These metrics enable a comprehensive diagnosis of the label classifier performance at scale on different aspects of the corpus, such as domains, languages, and time ranges. In practice, we recommend leveraging both human evaluation and LLM-based metrics as a holistic evaluation suite, while also taking into account the task and annotation reliability. This approach can help us identify and mitigate the possible bias that may arise from either method or be affected by the task complexity, and enable us to scale up the evaluation and annotation to a large corpus sample with confidence, thus obtaining more robust and informative evaluation results. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors:
(1) Mengting Wan, Microsoft Corporation and Microsoft Corporation;
(2) Tara Safavi (Corresponding authors), Microsoft Corporation;
(3) Sujay Kumar Jauhar, Microsoft Corporation;
(4) Yujin Kim, Microsoft Corporation;
(5) Scott Counts, Microsoft Corporation;
(6) Jennifer Neville, Microsoft Corporation;
(7) Siddharth Suri, Microsoft Corporation;
(8) Chirag Shah, University of Washington and Work done while working at Microsoft;
(9) Ryen W. White, Microsoft Corporation;
(10) Longqi Yang, Microsoft Corporation;
(11) Reid Andersen, Microsoft Corporation;
(12) Georg Buscher, Microsoft Corporation;
(13) Dhruv Joshi, Microsoft Corporation;
(14) Nagu Rangan, Microsoft Corporation. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work 2 Related Work 3 Method and 3.1 Phase 1: Taxonomy Generation 3 Method and 3.1 Phase 1: Taxonomy Generation 3.2 Phase 2: LLM-Augmented Text Classification 3.2 Phase 2: LLM-Augmented Text Classification 4 Evaluation Suite and 4.1 Phase 1 Evaluation Strategies 4 Evaluation Suite and 4.1 Phase 1 Evaluation Strategies 4.2 Phase 2 Evaluation Strategies 4.2 Phase 2 Evaluation Strategies 5 Experiments and 5.1 Data 5 Experiments and 5.1 Data 5.2 Taxonomy Generation 5.2 Taxonomy Generation 5.3 LLM-Augmented Text Classification 5.3 LLM-Augmented Text Classification 5.4 Summary of Findings and Suggestions 5.4 Summary of Findings and Suggestions 6 Discussion and Future Work, and References 6 Discussion and Future Work, and References A. Taxonomies A. Taxonomies B. Additional Results B. Additional Results C. Implementation Details C. Implementation Details D. Prompt Templates D. Prompt Templates 4.2 Phase 2 Evaluation Strategies To quantitatively evaluate text classification, we create a benchmark dataset with reliable ground-truth annotations as follows: Task and Annotation Reliability. We first assess the reliability of the label assignment task and the human annotations by involving multiple human annotators and calculating the interrater agreement (Cohen’s Kappa [6] between two raters and Fleiss’ Kappa [7] among multiple raters). We then resolve disagreements between human annotations by either voting or deliberation, and obtain a consensus human annotation for each instance. Then we use an LLM as an additional annotator to perform the same label assignment task, and measure the agreement between the LLM annotation and the consensus human label. Intuitively, this agreement captures how well the LLM is aligned with (the majority of) human annotators and how reliable it is for this label assignment task. Task and Annotation Reliability. Classification Metrics. We apply both human and LLM annotations on a small-scale corpus sample and calculate the conventional multiclass and multilabel classification metrics (e.g., Accuracy, F1) with human annotations as the ground truth. These metrics evaluate how the label classifier is aligned with human preferences on a small subset of the corpus. We then apply the LLM annotator on a larger-scale corpus sample and leverage the resulting annotations as the oracle to calculate the same classification metrics. These metrics enable a comprehensive diagnosis of the label classifier performance at scale on different aspects of the corpus, such as domains, languages, and time ranges. Classification Metrics. In practice, we recommend leveraging both human evaluation and LLM-based metrics as a holistic evaluation suite, while also taking into account the task and annotation reliability. This approach can help us identify and mitigate the possible bias that may arise from either method or be affected by the task complexity, and enable us to scale up the evaluation and annotation to a large corpus sample with confidence, thus obtaining more robust and informative evaluation results. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv Authors: (1) Mengting Wan, Microsoft Corporation and Microsoft Corporation; (2) Tara Safavi (Corresponding authors), Microsoft Corporation; (3) Sujay Kumar Jauhar, Microsoft Corporation; (4) Yujin Kim, Microsoft Corporation; (5) Scott Counts, Microsoft Corporation; (6) Jennifer Neville, Microsoft Corporation; (7) Siddharth Suri, Microsoft Corporation; (8) Chirag Shah, University of Washington and Work done while working at Microsoft; (9) Ryen W. White, Microsoft Corporation; (10) Longqi Yang, Microsoft Corporation; (11) Reid Andersen, Microsoft Corporation; (12) Georg Buscher, Microsoft Corporation; (13) Dhruv Joshi, Microsoft Corporation; (14) Nagu Rangan, Microsoft Corporation. Authors: Authors: (1) Mengting Wan, Microsoft Corporation and Microsoft Corporation; (2) Tara Safavi (Corresponding authors), Microsoft Corporation; (3) Sujay Kumar Jauhar, Microsoft Corporation; (4) Yujin Kim, Microsoft Corporation; (5) Scott Counts, Microsoft Corporation; (6) Jennifer Neville, Microsoft Corporation; (7) Siddharth Suri, Microsoft Corporation; (8) Chirag Shah, University of Washington and Work done while working at Microsoft; (9) Ryen W. White, Microsoft Corporation; (10) Longqi Yang, Microsoft Corporation; (11) Reid Andersen, Microsoft Corporation; (12) Georg Buscher, Microsoft Corporation; (13) Dhruv Joshi, Microsoft Corporation; (14) Nagu Rangan, Microsoft Corporation.

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

TnT-LLM for User Intent and Conversational Domain Labeling in Bing Copilot

Evaluating TnT-LLM Text Classification: Human Agreement and Scalable LLM Metrics

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

256K Tokens on One GPU? Jamba’s Engineering Magic Explained

TnT-LLM: Text Mining at Scale With Large Language Models

Prior Approaches to Text Mining: Taxonomy, Clustering, and LLM Annotation

TnT-LLM: Automating Text Taxonomy Generation and Classification With Large Language Models

TnT-LLM: LLMs for Automated Text Taxonomy and Classification

256K Tokens on One GPU? Jamba’s Engineering Magic Explained

TnT-LLM: Text Mining at Scale With Large Language Models

Prior Approaches to Text Mining: Taxonomy, Clustering, and LLM Annotation

TnT-LLM: Automating Text Taxonomy Generation and Classification With Large Language Models

TnT-LLM: LLMs for Automated Text Taxonomy and Classification

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps