Table of Links Abstract and 1. Introduction Abstract and 1. Introduction Related Work


Task, Datasets, Baseline


RQ 1: Leveraging the Neighbourhood at Inference
4.1. Methods
4.2. Experiments


RQ 2: Leveraging the Neighbourhood at Training
5.1. Methods
5.2. Experiments


RQ 3: Cross-Domain Generalizability


Conclusion


Limitations


Ethics Statement


Bibliographical References Related Work Related Work Related Work Task, Datasets, Baseline Task, Datasets, Baseline Task, Datasets, Baseline RQ 1: Leveraging the Neighbourhood at Inference
4.1. Methods
4.2. Experiments RQ 1: Leveraging the Neighbourhood at Inference 4.1. Methods 4.1. Methods 4.2. Experiments 4.2. Experiments RQ 2: Leveraging the Neighbourhood at Training
5.1. Methods
5.2. Experiments RQ 2: Leveraging the Neighbourhood at Training RQ 2: Leveraging the Neighbourhood at Training 5.1. Methods 5.1. Methods 5.2. Experiments 5.2. Experiments RQ 3: Cross-Domain Generalizability RQ 3: Cross-Domain Generalizability RQ 3: Cross-Domain Generalizability Conclusion Conclusion Conclusion Limitations Limitations Limitations Ethics Statement Ethics Statement Ethics Statement Bibliographical References Bibliographical References Bibliographical References 4. RQ 1: Leveraging the Neighbourhood at Inference In this section, we leverage the knowledge from semantically similar training instances directly during inference without extra training overhead. We interpolate the label distribution predicted by the baseline model with the distribution derived from the training instances similar to the test instance. This overcomes the problem of memorizing/learning rare patterns implicitly in the model parameters, thus enhancing the model’s ability to handle longtail cases (classes with few instances or rare patterns in frequent classes) especially in limited data settings. We explore three different methods to obtain the distribution from similar training instances. 4.1. Methods 4.1.1. Interpolation with kNN 4.1.1. Interpolation with kNN In this method, we construct a datastore of training instances and then retrieve the nearest neighbours to the test instance for computing the interpolated label distribution during the inference. Datastore Construction After training, we obtain contextualized representation ci of every sentence in each document of the training set using the trained model. We construct the datastore by a single forward pass over each training document. The datastore {K, V } is the set of all contextualized representation-rhetorical label pairs constructed from all the training examples D as: Datastore Construction Interpolation During inference time, we query the datastore using the contextualized representation of every sentence in the test document, to find the k-nearest neighbours N according to the euclidean distance. Then, we derive the distribution of labels pkNN using labels of the retrieved neighbours based on softmax of their negative distances, while aggregating probability mass for each label across all its occurrences in the retrieved neighbours (labels that do not appear in the retrieved N have zero probability). Intuitively, the closer a neighbor is to the test instance, the larger its weight is. Interpolation 4.1.2. Interpolation with Single Prototype 4.1.2. Interpolation with Single Prototype Instead of storing all the training instances in the datastore, we seek to store one prototype for each label, which captures the essential semantics of various sentences for each rhetorical role, significantly reducing the datastore’s memory footprint. To create a prototype for each label, we calculate the average of the contextualized embeddings of sentences that share the same rhetorical role. Intuitively these prototypes can be assumed to be the center of clusters for different labels, surrounded by sentences expressing the same label in the embedding space. The interpolation process closely resembles the kNN approach (Eq. 5), with the key difference being that interpolation directly involves the prototypes, rather than a prior retrieval step. 4.1.3. Interpolation with Multiple Prototypes 4.1.3. Interpolation with Multiple Prototypes Instead of using a single prototype for each rhetorical role, we suggest the use of multiple prototypes for each label. This choice is driven by the fact that instances with the same rhetorical role can exhibit distinct variations in expression, resulting in diverse contextual embeddings scattered across the embedding space. Averaging these embeddings into a single prototype might diminish specificity. Utilizing multiple prototypes allows us to effectively capture the intricate viewpoints within each label. To accomplish this, we cluster the instances belonging to each rhetorical role using k-means, yielding multiple prototypes for each label from the k centroids. The interpolation step remains similar (Eq.5), involving all these multiple prototypes without any retrieval step. Authors:
(1) Santosh T.Y.S.S, School of Computation, Information, and Technology; Technical University of Munich, Germany (santosh.tokala@tum.de);
(2) Hassan Sarwat, School of Computation, Information, and Technology; Technical University of Munich, Germany (hassan.sarwat@tum.de);
(3) Ahmed Abdou, School of Computation, Information, and Technology; Technical University of Munich, Germany (ahmed.abdou@tum.de);
(4) Matthias Grabmair, School of Computation, Information, and Technology; Technical University of Munich, Germany (matthias.grabmair@tum.de). Authors: Authors: (1) Santosh T.Y.S.S, School of Computation, Information, and Technology; Technical University of Munich, Germany (santosh.tokala@tum.de); (2) Hassan Sarwat, School of Computation, Information, and Technology; Technical University of Munich, Germany (hassan.sarwat@tum.de); (3) Ahmed Abdou, School of Computation, Information, and Technology; Technical University of Munich, Germany (ahmed.abdou@tum.de); (4) Matthias Grabmair, School of Computation, Information, and Technology; Technical University of Munich, Germany (matthias.grabmair@tum.de). This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

How Neighborhood Data Improves Legal Document Classification

Leveraging Deep Learning for Legal Text Analysis

Instancing.TECH

Too Long; Didn't Read

Make resilience your competitive advantage

Improving Legal Document Labeling by Comparing Similar Sentences

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

The Tortoise and the Hare: An Unexpected Scheduling Race Between MILP and CP Solvers

New AI Methods Help Machines Understand Legal Text Better

Datasets and Models Used to Analyze Legal Documents

Leveraging Deep Learning for Legal Text Analysis

How Neighborhood Data Improves Legal Document Classification

The Tortoise and the Hare: An Unexpected Scheduling Race Between MILP and CP Solvers

New AI Methods Help Machines Understand Legal Text Better

Datasets and Models Used to Analyze Legal Documents

Leveraging Deep Learning for Legal Text Analysis

How Neighborhood Data Improves Legal Document Classification

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps