If you have ever worked on a Spring application using Hibernate 5, you probably encountered the challenge of dealing with duplicated entities. It is an odd scenario where your collection contains multiple entries referencing the same entity. This is a quite known issue, and there are ways to address it whether using the keyword, Sets, or JPA EntityGraph. DISTINCT However, I happened to have quite disturbing behavior using EntityGraph returning duplicated entities! We will be diving a bit under the hood of Hibernate 5 to understand what happened. A Simple Spring Boot Application To illustrate Hibernate 5 behavior, let’s start with a basic Spring Boot project connected to a database: https://github.com/Shaolans/HibernateEntityGraphDeduplicationIssue The database schema is a simple model as follows: The Hibernate entities are defined as follows: Our focus will be on the queries generated by this JPA repository: @Repository
public interface AuthorRepository extends JpaRepository<Author, Integer> {

    @Query("SELECT a FROM Author a LEFT JOIN FETCH a.books")
    List<Author> findAuthorsLeftJoinFetchNoEntityGraph();

    @EntityGraph(attributePaths = "books")
    @Query("FROM Author a LEFT JOIN a.books")
    List<Author> findAuthorsLeftJoinWithEntityGraph();

    @Query("FROM Author a LEFT JOIN a.books")
    List<Author> findAuthorsLeftJoinWithoutEntityGraph();

    @EntityGraph(attributePaths = "books")
    @Query("FROM Author a INNER JOIN a.agent WHERE a.agent.name like '%John%'")
    List<Author> findAuthorsWithJoinNotFetchedAndEntityGraph();

    @EntityGraph(attributePaths = "books")
    @Query("SELECT a FROM Author a INNER JOIN a.agent WHERE a.agent.name like '%John%'")
    List<Author> findAuthorsWithJoinNotFetchedAndEntityGraphAndSelect();
} Entity Duplication Why Does It Happen? To understand the occurrence of entity duplication, let’s consider a use case where the objective is to retrieve all authors and their respective books. In order to avoid the issue I include in my JPQL query the keyword . N+1 query FETCH It enables the retrieval of both authors and their associated books in a single SQL query, eliminating the necessity to individually fetch books for each author. @Query("SELECT a FROM Author a LEFT JOIN FETCH a.books")
List<Author> findAuthorsLeftJoinFetchNoEntityGraph(); Invoking this method will result in the compilation of an SQL query equivalent to the following: SELECT A, B FROM AUTHOR A LEFT JOIN BOOK B ON A.ID = B.BOOK_AUTHOR_ID; In a case where you have 2 authors that have respectively 2 and 3 books, the result of the SQL query will yield the following table: Depending on the type of used, the association between the and tables are established based on the provided condition. join Author Book This will produce for each that matched an , a new row containing the columns of the table but also the columns of the table. Book Author Book Author Hibernate, upon retrieving this result set, intelligently resolves the entity without duplication. In our case, we end up with only two instances of the entity. However, it handles resolution on a row-by-row basis and does not inherently deduplicate the rows by default. Author This explains why the , returned by calling , returns the following collection: List findAuthorsLeftJoinFetchNoEntityGraph [Author@5640, Author@5640, Author@5641, Author@5641, Author@5641] being the instance representing J.R.R Tolkien entity and representing the Isaac Asimov. Author@5640 Author Author@5641 Author This duplication is mostly observed on and relationships since joining with these types of relations might match multiple rows of the associated table. @OneToMany @ManyToMany On the contrary, and relationships match at most only 1 entry on the associated table and will not induce duplication. @OneToOne @ManyToOne How to Deduplicate Although this duplication issue is quite common and often occurs during the development of applications, there are ways to address and resolve this issue. Set A straightforward solution involves using the returning type instead of : Set<Author> List<Author> @Query("FROM Author a LEFT JOIN FETCH a.books")
Set<Author> findAuthorsLeftJoinFetchNoEntityGraph(); When the results are collected, duplicated entities will be filtered through the . Set DISTINCT keyword Another approach is to use the keyword: DISTINCT @Query("SELECT DISTINCT a FROM Author a LEFT JOIN FETCH a.books")
List<Author> findAuthorsLeftJoinFetchNoEntityGraph(); It should be noted that using as two semantic meanings: DISTINCT : The conventional SQL meaning involves filtering duplicate values in the selected columns. SQL : It indicates a desire to filter duplications in the parent entity. Hibernate When the keyword is used in a JPQL query, it leads to the generation of an SQL query featuring the keyword but also instructing Hibernate to filter duplicated entities. DISTINCT DISTINCT However, using in SQL induces an overhead when executed in the database, and in some cases, the goal is to only filter duplicate entities induced by the join. For such cases, the Hibernate hint can be employed to avoid the generation of in the SQL query: DISTINCT HINT_PASS_DISTINCT_THROUGH DISTINCT @QueryHints(@QueryHint(name = HINT_PASS_DISTINCT_THROUGH, value = "false"))
@Query("SELECT DISTINCT a FROM Author a LEFT JOIN FETCH a.books")
Set<Author> findDistinctAuthorsLeftJoinFetchNoEntityGraph(); EntityGraph In JPA 2.1, EntityGraph was introduced as a powerful feature allowing developers to define the loading strategies of entities. In our case, we can just use the annotation to load the books along with the , as illustrated below: @EntityGraph Author @EntityGraph(attributePaths = "books")
@Query("FROM Author a LEFT JOIN a.books WHERE a.books.size > 2")
List<Author> findAuthorsLeftJoinWithEntityGraph(); This JPQL query is designed to retrieve all authors, along with their respective books, who possess more than 2 books. Even without explicitly including the keyword in the SQL query, the EntityGraph ensures the books are loaded and the parent entity remains non-duplicated. FETCH Author : This is a simple example of using EntityGraph; a more complete and sophisticated way of using it exists, but this is not the scope of this section. Note EntityGraph and Entity Duplication While it is established that EntityGraph handles entity deduplication, let's look into a more complicated scenario. Consider the entity, which has a relation with . Suppose the objective is to retrieve all authors and their books where the author's agent has a name containing "John", yet we have no intention of loading the entity for an arbitrary reason. In this case, we can accomplish this by the following code: Author @ManyToOne Agent Agent @EntityGraph(attributePaths = "books")
@Query("FROM Author a INNER JOIN a.agent WHERE a.agent.name like '%John%'")
List<Author> findAuthorsWithJoinNotFetchedAndEntityGraph(); Since we are using EntityGraph, we expect to avoid duplicated , but upon executing the method, the outcome is unexpected: Author Duplicated entities are retrieved despite the use of EntityGraph. To understand the underlying reasons for this behavior, let’s dive a bit under the hood of Hibernate! How Hibernate Resolves Entity Duplication Hibernate is a quite complete and complex ORM, it can be hard to grasp entirely its fine details, but in our case, since we want to understand deduplication mechanisms, we will be focusing on the method of list . QueryTranslatorImpl For the sake of clarity, some portions of the code have been omitted in order to highlight the most crucial aspects. Upon invoking a method from the , the execution will eventually reach this code snippet. Here is a breakdown of its functionality: AuthorRepository It computes a boolean called , and as the name implies, it identifies whether the result set needs to be deduplicated. needsDistincting We can observe when we do use a keyword, EntityGraph, or limit the size of retrieval, there is a operation in the JPQL query; the distinction needs to be applied. DISTINCT AND FETCH In our issue, it can be observed that whenever EntityGraph is used, the distinction will be applied since EntityGraph will edit the source query to “ ” child entities given in the attribute paths. FETCH This method invocation executes the query in the database, retrieves the result set, and resolves the entities. This list might contain duplicated entities. This is the most interesting part here; when it does need to distinguish entities, Hibernate uses an to filter out the duplicated entity. IdentitySet At this point, it is established that using EntityGraph should filter duplicated entities, but what happened to cause the unexpected outcome previously discussed? Why EntityGraph Returns Duplicates in Some Cases Let’s Compare In order to have a better understanding of the situation, let’s compare the results of when invoking the following methods: queryLoader.list( session, queryParametersToUse ) (2) @EntityGraph(attributePaths = "books")
@Query("FROM Author a LEFT JOIN a.books")
List<Author> findAuthorsLeftJoinWithEntityGraph(); and @EntityGraph(attributePaths = "books")
@Query("FROM Author a INNER JOIN a.agent WHERE a.agent.name like '%John%'")
List<Author> findAuthorsWithJoinNotFetchedAndEntityGraph(); Upon calling , the returned results contains: findAuthorsLeftJoinWithEntityGraph Given the EntityGraph is used, the duplication will be resolved through the , and at the end, the method will return a consisting of 3 authors/instances. IdentitySet List<Author> However, upon calling , the returned results contains the following: findAuthorsWithJoinNotFetchedAndEntityGraph In this scenario, the result is not a List of instances but rather a of distinct array instances of Objects; each encapsulating duplicated and instances. The reason behind the presence of duplicate entities becomes evident. Author List Author Agent Each array ( , , ..., ), being a unique instance, is not filtered during the distinction process with the . Object[2]@13551 Object[2]@135512 Object[2]@13555 IdentitySet Consequently, all arrays are considered as distinct instances, leading to the observed duplication. Explaining the Difference In order to understand the difference between obtaining a List of instances in the former case and a list of arrays containing and instances in the latter case, we need to explore deeper; let’s take a look at the class and especially the method . Author Author Agent QueryLoader initialize This initialization code has interesting parts, particularly the one highlighted in red. It creates an array of boolean where it checks which entities should be selected and be returned. includeInSelect Let’s take for example: @Query("SELECT a, b FROM Author a LEFT JOIN FETCH a.books b") The initialize method checks the content one by one which is and ; if the selected entity originates from a in the clause, then it will be omitted since is used to load children entities (but not in the case of the parent entity). Otherwise, the entity should be selected. SELECT a b FETCH FROM FETCH In this example, the array contains . includeInSelect [true (Author), false (Book)] In our case, of entity duplication with EntityGraph: @EntityGraph(attributePaths = "books")
@Query("FROM Author a INNER JOIN a.agent WHERE a.agent.name like 'John'")
List<Author> findAuthorsWithJoinNotFetchedAndEntityGraph(); We did not specify the clause, so by default, it will select all the entities from all the tables specified in the clause. SELECT FROM The logic returns an array where and (2 elements) should be selected because only is subject to a operation due to the use of EntityGraph. includeInSelect [true (Author), false (Book), true (Agent)] Author Agent Book FETCH This explains the difference between and . findAuthorsLeftJoinWithEntityGraph findAuthorsWithJoinNotFetchedAndEntityGraph The former returns only one value per row because the clause contained only and entities (the select of was generated by EntityGraph), however, since was subject to a , it was not taken into account in the . SELECT Author Book Book Book FETCH includeInSelect The latter returns an array per row because and were in the clause and are not subject to a operation unlike . Thus, returning an array consisting of 2 elements per row. Author Agent SELECT FETCH Book In summary, Hibernate consistently returns an array, and the content of this array is determined by the entities specified in the clause that are not associated with operations. SELECT FETCH However, if we follow the preceding logic, should also return an array but an array of a single element. This is accurate; it does. However, upon examining the implementation at ** **lines and , when the array consists of a single element, the element is just unwrapped. findAuthorsLeftJoinWithEntityGraph QueryLoader 476 620 How to Fix It To ensure correct duplicate filtering when applying a distinction, the result set should be a list of entities rather than a list of entity arrays. This behavior is determined by the entities selected. A straightforward solution is to include the desired entities in the clause: SELECT @EntityGraph(attributePaths = "books")
@Query("SELECT a FROM Author a INNER JOIN a.agent WHERE a.agent.name like '%John%'")
List<Author> findAuthorsWithJoinNotFetchedAndEntityGraph(); The clause will only contain and (the select being generated by the EntityGraph): SELECT Author Book Book Since is a operation, the will only select one element ( ), and instead of returning an array, it will return directly the instance which then will pass through the to be deduplicated. Book FETCH includeInSelect Author Author IdentitySet Conclusion Entity duplication occurs when tables are joined, and multiple rows are matched. There are several methods to address this issue, including using , the keyword, or EntityGraph to deduplicate the result set. Set DISTINCT In the last two cases, deduplication is achieved through a Hibernate flag that accumulates results into an . IdentitySet However, it may not work in certain scenarios due to the fact that, depending on your clause, it could return newly instantiated arrays of your selected entities for each row. This could undermine deduplication through the . SELECT IdentitySet Therefore, it is advisable, when writing JPQL queries, to always specify the clause. If not supplied, all entities in the clause that are not subject to a operation will be selected by default. SELECT FROM FETCH The issue of duplicated parent entities is addressed in Hibernate 6 (shipped in Spring Boot 3), as it has introduced automatic handling of duplicated entities. However, considering that the release of Hibernate 6 is relatively recent, many applications still run on Hibernate 5 or earlier versions, and migration from Hibernate 5 to 6 might be challenging. Thus, understanding how Hibernate 5 handles these situations remains relevant for the considerable number of applications still using it. Also published here

This story contains new, firsthand information uncovered by the writer.

Hibernate 5 and Entities Duplication: Let’s Dive Under the Hood

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

10 Ways to Optimize Your Database

10 Ways to Reduce Data Loss and Potential Downtime Of Your Database

10 Principles of Proper Database Benchmarking

10 Minute Guide to Fixing Damaged SQL Databases - No Recovery Required!

10 Best Online Courses to Learn Oracle and PL/SQL for Beginners

121 Stories To Learn About Databases

10 Ways to Optimize Your Database

10 Ways to Reduce Data Loss and Potential Downtime Of Your Database

10 Principles of Proper Database Benchmarking

10 Minute Guide to Fixing Damaged SQL Databases - No Recovery Required!

10 Best Online Courses to Learn Oracle and PL/SQL for Beginners

121 Stories To Learn About Databases

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps