Neural Entity Linking and How JPMorgan Chase Plans to Use it by@harshit158

Neural Entity Linking and How JPMorgan Chase Plans to Use it

JPMC published a paper in 2021 highlighting their approach to Entity Linking. It's the task of assigning a unique identity to ambiguous mentions of named entities in a text. This article summarizes the problem statement, solution, and other key technical components of the paper. The idea is to capture both Semantic distance(the distance of the mention or entity composition) and distance (the composition of the name composition) of the names and distance between the names are calculated by step 1: Obtaintaindings for Entions and embeddings.
image
Harshit Sharma HackerNoon profile picture

Harshit Sharma

ML Engineer @ Juniper Networks | https://medium.com/@harshit158


JPMC published a paper in 2021 highlighting their approach to Entity Linking. This article summarizes the problem statement, solution, and other key technical components of the paper*

image

What is Entity Linking?

It's the task of assigning a unique identity to ambiguous mentions of named entities in a text.

An example of an entity linking from Wikipedia

An example of an entity linking from Wikipedia

Here, “Paris” from the text is given a unique identity via a URL (the most common type of URI) “wikipedia.org/wiki/Paris”. Note that the type of URI used to identify the mentioned entity depends on the domain uniquely. For eg, Instead of a web address, we could have used ISBNs if we were to identify books from a text.

image

JPMC was interested in :

Mapping mentions of financial institutions from news articles to the entities stored in their internal knowledge base (stored as a Knowledge Graph)


An example is shown below:

Example of Entity Linking from the paper

Example of Entity Linking from the paper


There are two sub-problems that must be defined:

  1. Recognition:

    Extraction of mentions from financial news articles. JPMC has used Spacy for this.


  2. Linking:

    Choosing the correct entity from the internal Knowledge Graph to be linked to the extracted mentions in the previous step. The paper discusses this step.


A pictorial representation of this is shown below:

(Image by Author) Sub-problems as part of the overall solution

(Image by Author) Sub-problems as part of the overall solution



image

  1. String Matching

    These approaches capture the “morphological” structure of the entity names. The team experimented with

    (a) Jaccard

    (b) Levenshtein

    (c) Ratcliff-Obershelp (also known as Gestalt-Pattern-Matching)

    (d) **Jaro Winkler**(e) N-Gram Cosine Similarity


The con of these approaches is that they focus just on the “syntactics” of the names and not the semantics. An example of a failure case would be the matching of “Lumier” and “Lumier”. Even though they are exactly the same, they refer to two different entities.


2. Context Similarity Methods:

These methods take the contexts around mentions and entities to give a similarity score.

The Context for “mention” is text to the left and right of the mention, whereas

The Context for “entity” is all the data that's stored in the KG for this entity.

Finally, Cosine similarity / Jaccard similarity can be used on top of the context vectors.


3. ML Classification:

Naive Bayes, Logistic Regression, and SVM are trained on (mention, entity) pairs to find the ones that should be linked


4. Learn to Rank Methods (LTR)

These models work in tandem with ML approaches, which might give us multiple (mention, entity) pairs as the solutions. LTR approaches just narrow down to the most probable solution.


image

The idea is to capture both Semantic distance(the meaning that the mention or entity stands for) and Syntactic distance (character composition of the name) between the names and use a contrastive loss function to train a model.

image

We will see below how both of these distances are calculated step by step.

Step 1: Obtain embeddings for Entities and Mentions

To come up with both of the distances, the authors have proposed to use embeddings for mentions as well as entities in the KG.

To obtain Entity embeddings, the authors have used a Triplet Loss function (shown below)

Triplet Loss Function

Triplet Loss Function


For each entity, they used 10 positive and 10 negative samples, making 10 <entity, positive word, negative word> triplets.

Model for Entity embedding

Model for Entity embedding


Unlike Entity embeddings, which they had precomputed, the mention embeddings were trained using the on-the-go embeddings approach, where the embedding matrix is learned during the training itself.

Step 2: Calculate Syntactic Distance score

Before going further, it's worth mentioning “Wide & Deep” architecture which was introduced by Google in 2016. You can find their official blog here. We won’t go into the details, but to give a summary, it's an architecture that has two components — The Wide component and the Deep component.

Image from Google’s paper | Spectrum of Wide and Deep Models

Image from Google’s paper | Spectrum of Wide and Deep Models


Syntactic Distance score calculation is done using the WIDE part, which consists of a Linear Siamese Network.

Calculation of Syntax Distance score

Calculation of Syntax Distance score


The output of the siamese network is the vectors for both the entity and the mention, which are then compared to find the Euclidean distance.

image

Step 3: Calculate Semantic Distance score

Semantic Distance score calculation is done using the DEEP part

Calculation of Semantic Distance score

Calculation of Semantic Distance score


eₖ is the pre-trained embedding for the “Apache Corp” that was calculated in Step 1. To obtain the embedding for the mention, its left and right context words are fed into a Bi-LSTM network that trains the embeddings. The embedding vectors of mention (Vₘ) and the entity (Vₑ) are then used to find the Euclidean distance:

image

Step 4: Compute Contrastive Loss

Both Syntactic and Semantic distances are combined in a weighted fashion as follows:

image

The contrastive loss function is then combined as follows:

image

Contrastive Loss Function

where Y is the ground truth value, where a value of 1 indicates that mention m and entity e are matched, 0 otherwise.

Combining all the pieces, the final model framework is shown below:

JEL Model Framework

JEL Model Framework


image

At the time of writing this paper, JPMC was still in the process of deploying the model, which, once done, will help support users across JPMC in discovering relevant and curated news that matters to their business.


From the cost perspective, not all the mentions are fed through the JEL framework as that would be computationally expensive. JPMC has put another blocking layer to funnel out the mentions that share less than 2 bigrams with the entities from their internal KGs.



Once again, here is the paper link if you would like to read the full paper.



Also published here



Thanks for reading !! Follow Intuitive Shorts (my Substack newsletter), to read quick and intuitive summaries of ML/NLP/DS concepts. You can also follow me on Medium.


react to story with heart
react to story with light
react to story with boat
react to story with money
L O A D I N G
. . . comments & more!