Adaptive-RAG: Adaptive Retrieval-Augmented Generation

Table of Links

Abstract and 1. Introduction

2 Related Work

3 Method and 3.1 Preliminaries

3.2 Adaptive-RAG: Adaptive Retrieval-Augmented Generation

4 Experimental Setups and 4.1 Datasets

4.2 Models and 4.3 Evaluation Metrics

4.4 Implementation Details

5 Experimental Results and Analyses

6 Conclusion, Limitations, Ethics Statement, Acknowledgements, and References

A Additional Experimental Setups

B Additional Experimental Results

3.2 Adaptive-RAG: Adaptive Retrieval-Augmented Generation

We now introduce our adaptive retrieval-augmented LLMs, which are built upon three different strategies described in the previous section, and which are designed to select the most suitable strategy according to the complexity of queries.

Adapting Retrieval-Augmented LLMs Note that in real-world scenarios, not all q from users have the same level of complexity, necessitating tailored strategies for handling each query. In other words, employing the most basic, non-retrievalbased approach LLM(q) to respond to the complex query q would be also ineffective (Figure 2, A); conversely, using a more elaborate multi-step approach LLM(q, d, c) for simple q would be inefficient (Figure 2, B). Therefore, our adaptive framework is designed to dynamically adjust the queryhandling strategy of retrieval-augmented LLMs, which is achieved by determining the complexity of each query before attempting a solution. Notably, this framework can offer a robust middle ground with a range of solutions, from the simplest approach for the most straightforward queries, to the one-step approach for moderate queries, and up to the most comprehensive and rigorous approach for complex queries. In addition, since the operations of LLM and Retriever remain consistent regardless of inputs to them, our method can seeminglessly go back and forth across queries of different complexities, without changing the internal model architecture or parameters during adaption.

Query Complexity Assessment To operationalize our adaptive retrieval-augmented LLM framework, we should determine the query complexity, and to achieve this, we propose to model a complexity classifier, whose goal is to return the appropriate complexity level of the given query. Specifically, given the query q, our classifier can be formulated as follows: o = Classifier(q), where Classifier is a smaller Language Model that is trained to classify one of three different complexity levels and o is its corresponding class label. In our classifier design, there are three class labels: ‘A’, ‘B’, and ‘C’, where ‘A’ indicates that q is straightforward and answerable by LLM(q) itself, ‘B’ indicates that q has the moderate complexity where at least a single-step approach LLM(q, d) is needed, and ‘C’ indicates that q is complex, requiring the most extensive solution LLM(q, d, c) [2].

Training Strategy The remaining step is to train the smaller Language Model for Classifier, to accurately predict its complexity o in response to the given query q. Yet, there is no annotated dataset available for query-complexity pairs. Hence, we propose to automatically construct the training dataset with two particular strategies.

To be specific, we first aim at labeling the query complexity based on the results from three different retrieval-augmented LLM strategies, in order to determine the label by its needs. For example, if the simplest non-retrieval-based approach correctly generates the answer, the label for its corresponding query is assigned ‘A’. Also, to break the tie between different models in providing the label to the query, we provide a higher priority to a simpler model. In other words, if both single-step and multi-step approaches produce the same correct answer while the non-retrieval-based approach fails, we assign label ‘B’ to its corresponding query.

However, this labeling strategy has a limitation in that not all the queries are assigned labels, since the three retrieval-augmented approaches may all fail to generate the correct answer. On the other hand, the benchmark datasets may already have meaningful inductive biases about the most appropriate retrieval-augmented LLM strategies for their queries, considering the ways they are created (e.g., QA datasets that require sequential reasoning usually necessitate a multi-step approach; while queries of those with labeled single documents can be ideally answerable with the single-step approach). Therefore, for those queries that remain unlabeled after the first labeling step, we assign ‘B’ to queries in single-hop datasets and ‘C’ to queries in multi-hop datasets. Finally, we train Classifier with these automatically collected query-complexity pairs[3], by using a crossentropy loss. Then, at inference, we can determine the complexity of the query, which is one of {‘A’, ‘B’, ‘C’}, by forwarding it to Classifier: o = Classifier(q).

[2] We consider three levels of query complexity, and leave the exploration of more fine-grained complexities as future work.