Authors:
(1) Soyeong Jeong, School of Computing;
(2) Jinheon Baek, Graduate School of AI;
(3) Sukmin Cho, School of Computing;
(4) Sung Ju Hwang, Korea Advanced Institute of Science and Technology;
(5) Jong C. Park, School of Computing. Authors: Authors: (1) Soyeong Jeong, School of Computing; (2) Jinheon Baek, Graduate School of AI; (3) Sukmin Cho, School of Computing; (4) Sung Ju Hwang, Korea Advanced Institute of Science and Technology; (5) Jong C. Park, School of Computing. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2 Related Work 2 Related Work 3 Method and 3.1 Preliminaries 3 Method and 3.1 Preliminaries 3.2 Adaptive-RAG: Adaptive Retrieval-Augmented Generation 3.2 Adaptive-RAG: Adaptive Retrieval-Augmented Generation 4 Experimental Setups and 4.1 Datasets 4 Experimental Setups and 4.1 Datasets 4.2 Models and 4.3 Evaluation Metrics 4.2 Models and 4.3 Evaluation Metrics 4.4 Implementation Details 4.4 Implementation Details 5 Experimental Results and Analyses 5 Experimental Results and Analyses 6 Conclusion, Limitations, Ethics Statement, Acknowledgements, and References 6 Conclusion, Limitations, Ethics Statement, Acknowledgements, and References A Additional Experimental Setups A Additional Experimental Setups B Additional Experimental Results B Additional Experimental Results 3.2 Adaptive-RAG: Adaptive Retrieval-Augmented Generation We now introduce our adaptive retrieval-augmented LLMs, which are built upon three different strategies described in the previous section, and which are designed to select the most suitable strategy according to the complexity of queries. Adapting Retrieval-Augmented LLMs Note that in real-world scenarios, not all q from users have the same level of complexity, necessitating tailored strategies for handling each query. In other words, employing the most basic, non-retrieval based approach LLM(q) to respond to the complex query q would be also ineffective (Figure 2, A); conversely, using a more elaborate multi-step approach LLM(q, d, c) for simple q would be inefficient (Figure 2, B). Therefore, our adaptive framework is designed to dynamically adjust the query handling strategy of retrieval-augmented LLMs, which is achieved by determining the complexity of each query before attempting a solution. Notably, this framework can offer a robust middle ground with a range of solutions, from the simplest approach for the most straightforward queries, to the one-step approach for moderate queries, and up to the most comprehensive and rigorous approach for complex queries. In addition, since the operations of LLM and Retriever remain consistent regardless of inputs to them, our method can seeminglessly go back and forth across queries of different complexities, without changing the internal model architecture or parameters during adaption. Adapting Retrieval-Augmented LLMs Query Complexity Assessment To operationalize our adaptive retrieval-augmented LLM framework, we should determine the query complexity, and to achieve this, we propose to model a complexity classifier, whose goal is to return the appropriate complexity level of the given query. Specifically, given the query q, our classifier can be formulated as follows: o = Classifier(q), where Classifier is a smaller Language Model that is trained to classify one of three different complexity levels and o is its corresponding class label. In our classifier design, there are three class labels: ‘A’, ‘B’, and ‘C’, where ‘A’ indicates that q is straightforward and answerable by LLM(q) itself, ‘B’ indicates that q has the moderate complexity where at least a single-step approach LLM(q, d) is needed, and ‘C’ indicates that q is complex, requiring the most extensive solution LLM(q, d, c) [2]. Training Strategy The remaining step is to train the smaller Language Model for Classifier, to accurately predict its complexity o in response to the given query q. Yet, there is no annotated dataset available for query-complexity pairs. Hence, we propose to automatically construct the training dataset with two particular strategies. Training Strategy To be specific, we first aim at labeling the query complexity based on the results from three different retrieval-augmented LLM strategies, in order to determine the label by its needs. For example, if the simplest non-retrieval-based approach correctly generates the answer, the label for its corresponding query is assigned ‘A’. Also, to break the tie between different models in providing the label to the query, we provide a higher priority to a simpler model. In other words, if both single-step and multi-step approaches produce the same correct answer while the non-retrieval-based approach fails, we assign label ‘B’ to its corresponding query. However, this labeling strategy has a limitation in that not all the queries are assigned labels, since the three retrieval-augmented approaches may all fail to generate the correct answer. On the other hand, the benchmark datasets may already have meaningful inductive biases about the most appropriate retrieval-augmented LLM strategies for their queries, considering the ways they are created (e.g., QA datasets that require sequential reasoning usually necessitate a multi-step approach; while queries of those with labeled single documents can be ideally answerable with the single-step approach). Therefore, for those queries that remain unlabeled after the first labeling step, we assign ‘B’ to queries in single-hop datasets and ‘C’ to queries in multi-hop datasets. Finally, we train Classifier with these automatically collected query-complexity pairs[3], by using a crossentropy loss. Then, at inference, we can determine the complexity of the query, which is one of {‘A’, ‘B’, ‘C’}, by forwarding it to Classifier: o = Classifier(q). 4 Experimental Setups In this section, we explain datasets, models, metrics, and implementation details. We provide additional details in Appendix A. 4.1 Datasets In order to simulate a realistic scenario, where different queries have varying complexities, we use both the single-hop and multi-hop QA datasets simultaneously, in the unified experimental setting. Single-hop QA For simpler queries, we use three benchmark single-hop QA datasets, which consist of queries and their associated documents containing answers, namely 1) SQuAD v1.1 (Rajpurkar et al., 2016), 2) Natural Questions (Kwiatkowski et al., 2019), and 3) TriviaQA (Joshi et al., 2017). Single-hop QA 1) SQuAD v1.1 2) Natural Questions 3) TriviaQA Multi-hop QA To consider more complex query scenarios, we use three benchmark multi-hop QA datasets, which require sequential reasoning over multiple documents, namely 1) MuSiQue (Trivedi et al., 2022a), 2) HotpotQA (Yang et al., 2018), and 3) 2WikiMultiHopQA (Ho et al., 2020). 1) MuSiQue 2) HotpotQA 3) 2WikiMultiHopQA This paper is available on arxiv under CC0 1.0 DEED license. This paper is available on arxiv under CC0 1.0 DEED license. available on arxiv [2] We consider three levels of query complexity, and leave the exploration of more fine-grained complexities as future work.

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

RAGged Edge: LLMs on a Retrieval Rollercoaster

Dynamic Retrieval Strategies: Enhancing QA Across Query Types

Read My Stories

Too Long; Didn't Read

Make resilience your competitive advantage

Query Quirks: AI’s Witty Way of Sorting Brain Teasers

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

How to Build a Patient From Scratch (and Still Trust the Results)

Can Adaptive QA Frameworks Solve the Query Complexity Puzzle?

Dynamic Retrieval Strategies: Enhancing QA Across Query Types

RAGged Edge: LLMs on a Retrieval Rollercoaster

Adaptive Ascension: LLMs, Efficiency, and Query Complexity

How to Build a Patient From Scratch (and Still Trust the Results)

Can Adaptive QA Frameworks Solve the Query Complexity Puzzle?

Dynamic Retrieval Strategies: Enhancing QA Across Query Types

RAGged Edge: LLMs on a Retrieval Rollercoaster

Adaptive Ascension: LLMs, Efficiency, and Query Complexity

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps