How to Run a RAG Powered Language Model on Android With the Help of MediaPipe

A couple of months ago, I gave a talk about running on device SLMs in apps. It was well received and refreshing to be able to give a talk where mobile apps can gain an advantage from the rise in language model usage. running on device SLMs in apps After the talk, I had a few pieces of feedback from attendees along the lines of “It would be good to see an example of an app using an SLM powered by RAG.” These are fair comments. The tricky thing here is the lead time it takes to set up a model and show something meaningful to developers in a limited time. Fortunately, it does make for a great blog post! So, here it is! If you’re looking for steps to add a RAG-powered language model to your Android app, this is the post for you! What is RAG? If you’re unfamiliar with the term, RAG stands for Retrieval Augmented Generation. It’s a technique for language models to access external information that isn’t available in their dataset after training. This allows models to be aware of up-to-date information they can use to provide more accurate answers to prompts. Retrieval Augmented Generation Let’s look at a quick example. Imagine you have two language models, one model is using RAG to retrieve external information about capital cities in Europe, whilst the other is only relying on its own knowledge. You decide to give both models the following prompt: You are an expert on the geography of Europe. Give me the name of 3 capital cities. Also give me an interesting fact for each of them. You are an expert on the geography of Europe. Give me the name of 3 capital cities. Also give me an interesting fact for each of them. The result could look something like this: As the diagram illustrates, you’re more likely to get a useful answer from the language model using RAG, instead of the model relying on its own knowledge. You may also notice less hallucinations from the RAG-powered model, as it doesn’t need to make up information. That’s really all you need to know about RAG at a high level. Next, let’s go deeper and write some code to enable your own RAG-powered language model inside your app! Setting Up Your Android App Let’s say you want to create an app using a language model to play the game of Simon Says. You want the model to be Simon, and use RAG to access a datasource of tasks to help decide what to ask the user. How would you do that? The most straightforward way to do that on Android is with MediaPipe, a collection of tools to help your app use AI and machine learning techniques. To begin, add the MediaPipe dependency to your app build.gradle: MediaPipe build.gradle dependencies { implementation("com.google.mediapipe:tasks-genai:0.10.27") implementation("com.google.ai.edge.localagents:localagents-rag:0.3.0") } dependencies { implementation("com.google.mediapipe:tasks-genai:0.10.27") implementation("com.google.ai.edge.localagents:localagents-rag:0.3.0") } Next, you need to add a language model to your test device via your computer. For this example, we’ll use Google’s Gemma3-1B, a lightweight language model that holds 1 billion parameters worth of information. Gemma3-1B Side Note: Before downloading the model, you may have to sign upto Kaggle and agree to Google’s AI Terms and Conditions. Side Note: Before downloading the model, you may have to sign upto Kaggle and agree to Google’s AI Terms and Conditions. Side Note: Before downloading the model, you may have to sign upto Kaggle and agree to Google’s AI Terms and Conditions. Once the model is downloaded, it’s time to add the model to your device. You can do this via adb: $ adb shell mkdir -p /data/local/tmp/slm/ # Create a folder to store the model $ adb push output_path /data/local/tmp/slm/gemma3-1B-it-int4.task # Copy the model over to the file $ adb shell mkdir -p /data/local/tmp/slm/ # Create a folder to store the model $ adb push output_path /data/local/tmp/slm/gemma3-1B-it-int4.task # Copy the model over to the file Alternatively, you can use the File Explorer using Android Studio to create the folders yourself and drag the model onto your device. File Explorer With the model added, you can continue building the RAG pipeline to feed Gemma with information. Next, let’s look at adding the information Gemma will rely on to answer prompts. Creating Embeddings The information language models rely on to perform RAG is not the same information you pass into it. Models require information to be in a specific format called Embeddings. Embeddings These are mathematical strings that represent the text’s semantic meaning. When the model receives a prompt, it will use it to search for the most relevant information matching it and use it alongside its own information to provide an answer. These mathematical strings are created by a tool called an Embedder. Embedder Embeddings are a whole subject on their own; you are encouraged to read about them. For this post, you only need to know how to create them. You can do this on the device by using the Gecko Embedder. Gecko Embedder First, download the sentencepiece.model tokenizer and the Gecko_256_f32.tflite embedder model files to your computer. Then push them to your device: sentencepiece.model Gecko_256_f32.tflite $ adb push sentencepiece.model /data/local/tmp/slm/sentencepiece.model # Push the tokenizer to the device $ adb push Gecko_256_f32.tflite /data/local/tmp/slm/Gecko_256_f32.tflite # Push the embedder model to the device $ adb push sentencepiece.model /data/local/tmp/slm/sentencepiece.model # Push the tokenizer to the device $ adb push Gecko_256_f32.tflite /data/local/tmp/slm/Gecko_256_f32.tflite # Push the embedder model to the device With the embedder installed on your device, it’s time to provide a sample file to create the embeddings from. In Android Studio, in your app module assets folder, create a file called simon_says_responses.txt. Then, in the file, add the following text: assets simon_says_responses.txt Go for a walk Jump and down 10 times Sing your favourite song! Text your best friend a funny meme Do 10 press ups! Go for a walk Jump and down 10 times Sing your favourite song! Text your best friend a funny meme Do 10 press ups! The file contains a couple of different responses one could give in a game of Simon Says, each split up with a tag. This gives the embedder a signal to know how to separate each response when splitting the text into embeddings. This process is called chunking and can have a large effect on how well RAG performs via the language model. Experimentation with different-sized chunks and responses is encouraged to find the right combination for your needs! chunking One thing to consider is app storage. Remember, you’ve already installed a language model and an embedder onto your device. These take up gigabytes of space, so make sure not to further bloat a device by using a text file that is too large! You want to consider storing the sample file in the cloud and downloading it via the network to reduce issues with storage. With the text file in place, it’s time to initialize the embedder: private const val GeckoEmbedderPath = "/data/local/tmp/slm/gecko_256_f32.tflite" private const val TokenizerModelPath = "/data/local/tmp/slm/sentencepiece.model" private const val UseGpuForEmbeddings = true val embedder: Embedder = GeckoEmbeddingModel( GeckoEmbedderPath, TokenizerModelPath, UseGpuForEmbeddings, ) private const val GeckoEmbedderPath = "/data/local/tmp/slm/gecko_256_f32.tflite" private const val TokenizerModelPath = "/data/local/tmp/slm/sentencepiece.model" private const val UseGpuForEmbeddings = true val embedder: Embedder = GeckoEmbeddingModel( GeckoEmbedderPath, TokenizerModelPath, UseGpuForEmbeddings, ) The embedder takes three parameters: the path of the embedder and the tokenizer on the device, with a final parameter to set whether the embedder can use the device GPU when creating embeddings. Setting this to true will speed up the creation of the embeddings if a GPU is available. Make sure to check the device capabilities before deciding to enable this value. Next, create an instance of the language model: private const val GemmaModelPath = "/data/local/tmp/slm/gemma3-1B-it-int4.task" val llmInferenceOptions = LlmInferenceOptions.builder() .setModelPath(GemmaModelPath) .setPreferredBackend(LlmInference.Backend.CPU) // Change to GPU if you have a GPU powered device. .setMaxTokens(1200) .build() val llmInferenceSessionOptions = LlmInferenceSessionOptions.builder() .setTemperature(0.6f) .setTopK(5000) .setTopP(1f) .build() val languageModel = MediaPipeLlmBackend( context, // This is the application context languageModelOptions, languageModelSessionOptions) languageModel.initialize().get() private const val GemmaModelPath = "/data/local/tmp/slm/gemma3-1B-it-int4.task" val llmInferenceOptions = LlmInferenceOptions.builder() .setModelPath(GemmaModelPath) .setPreferredBackend(LlmInference.Backend.CPU) // Change to GPU if you have a GPU powered device. .setMaxTokens(1200) .build() val llmInferenceSessionOptions = LlmInferenceSessionOptions.builder() .setTemperature(0.6f) .setTopK(5000) .setTopP(1f) .build() val languageModel = MediaPipeLlmBackend( context, // This is the application context languageModelOptions, languageModelSessionOptions) languageModel.initialize().get() There are a lot of parameters involved above; don’t worry about these for now. We’ll come back to them later on. With the language model created, you can focus back on the embeddings. The model needs a place to retrieve the embeddings each time it receives a prompt. MediaPipe provides an SQLite Vector Store, which is a common tool to store embeddings. Let’s create one: Vector Store private const val PromptTemplate: String = """ You are Simon in a game of Simon Says. Your task is to ask the player to perform a task from the following list: {0}. Your response must only contain the task that the player must do. Your response must be based on the players request: {1}. Do not ask the player to do the same thing twice. You must not ask the player to do anything that is dangerous, unethical or unlawful. """ val chainConfig = ChainConfig.create( languageModel, PromptBuilder(PromptTemplate), DefaultSemanticTextMemory( SqliteVectorStore(768), embedder ) ) private const val PromptTemplate: String = """ You are Simon in a game of Simon Says. Your task is to ask the player to perform a task from the following list: {0}. Your response must only contain the task that the player must do. Your response must be based on the players request: {1}. Do not ask the player to do the same thing twice. You must not ask the player to do anything that is dangerous, unethical or unlawful. """ val chainConfig = ChainConfig.create( languageModel, PromptBuilder(PromptTemplate), DefaultSemanticTextMemory( SqliteVectorStore(768), embedder ) ) Here, everything begins to link together as the language model and embedder are both passed into the ChainConfig. The 786 is the size of each “vector” the database can store. The PromptBuilder is used to provide a prompt to help drive the RAG process. 786 PromptBuilder Finally, let’s load the text file we created earlier in the assets folder into the embedder. First, load the file from the device and split the text into a list of strings: // This is an extension function to read the file from disk. val gameResponses: List = context.getTextFromFile("simon_says_responses.txt") // This is an extension function to read the file from disk. val gameResponses: List = context.getTextFromFile("simon_says_responses.txt") Next, load the responses into the chainConfig created earlier. chainConfig.semanticMemory.getOrNull() ?.recordBatchedMemoryItems(ImmutableList.copyOf(gameResponses)) ?.get() chainConfig.semanticMemory.getOrNull() ?.recordBatchedMemoryItems(ImmutableList.copyOf(gameResponses)) ?.get() Depending on your device and the size of the text file, this can take a couple of seconds to a few minutes to complete. A good rule of thumb is to prevent the language model from being used whilst this occurs. You could also run this operation on an IO thread to avoid blocking the main thread. With that done, you’ve just converted your textfile into a set of embeddings kept in a vector store. You’ve also linked your language model to the store so it can now retrieve information! The next section will show how to use those embeddings by passing the language model your prompts. Passing Prompts to Your RAG Powered Model With most of the complex setup complete, passing a prompt into the language model is surprisingly easy. First, you need to create a RetrievalAndInferenceChain using the chain config and invoke it. RetrievalAndInferenceChain val retrievalAndInferenceChain = RetrievalAndInferenceChain(chainConfig) val retrievalAndInferenceChain = RetrievalAndInferenceChain(chainConfig) Next, create a request and pass it into the chain. val prompt = "Tell me something to do Simon involving jumping!" val retrievalRequest = RetrievalRequest.create( prompt, RetrievalConfig.create( 50, // topK 0.1f, // minSimilarityScore RetrievalConfig.TaskType.RETRIEVAL_QUERY ) ) val response = retrievalAndInferenceChain!!.invoke(retrievalRequest).get().text val prompt = "Tell me something to do Simon involving jumping!" val retrievalRequest = RetrievalRequest.create( prompt, RetrievalConfig.create( 50, // topK 0.1f, // minSimilarityScore RetrievalConfig.TaskType.RETRIEVAL_QUERY ) ) val response = retrievalAndInferenceChain!!.invoke(retrievalRequest).get().text With that, the language model will process the prompt. During processing, it will refer to the embeddings in your vector store to provide what it thinks is the most accurate answer. As you would expect, what you ask the model will result in different responses. As the embeddings contain a range of simon says responses, it’s likely you will get a good response! What about the cases where you receive an unexpected result? At this point, you need to go back and fine tune the parameters of your objects from the previous section. Let’s do that in the next section. Fine-Tuning Your RAG Powered Language Model If there’s one thing we’ve learned over the years from Generative AI, it’s that it isn’t an exact science. You’ve no doubt seen stories where language models have “misbehaved” and produced a less than desired result, causing embarassment and reputational damage to companies. This is the result of not enough testing being performed to ensure the responses from a model are expected. Testing not only helps avoid embarassment, it can also help to experiment with your language model output so it works even better! Let’s take a look at what levers we can adjust to fine tune our RAG powered language model. Starting with LLmInferenceOptions.Builder: LLmInferenceOptions.Builder LlmInferenceOptions.builder() .setModelPath(GemmaModelPath) .setPreferredBackend(LlmInference.Backend.CPU) // Change to GPU if you have a GPU powered device. .setMaxTokens(1200) .build() LlmInferenceOptions.builder() .setModelPath(GemmaModelPath) .setPreferredBackend(LlmInference.Backend.CPU) // Change to GPU if you have a GPU powered device. .setMaxTokens(1200) .build() The first parameter that can be changed is the setMaxTokens() parameter. This is the amount of “input” or “output” the language model can handle. The larger the value, the more data the language model can handle at once. Resulting in better answers. setMaxTokens() In our example, this means the model can handle a text input and generate an output that consists of 1200 tokens. If we wanted to handle less text and also generate a smaller response, we could set MaxTokens to a smaller value. MaxTokens Be careful with this value, as you could find your model unexpectedly handling more tokens that it expects. Causing an app crash. Let’s move onto LlmInferenceSessionOptions.Builder(): LlmInferenceSessionOptions.Builder() llmInferenceSessionOptions = LlmInferenceSessionOptions.builder() .setTemperature(0.6f) .setTopK(5000) .setTopP(1f) .build() llmInferenceSessionOptions = LlmInferenceSessionOptions.builder() .setTemperature(0.6f) .setTopK(5000) .setTopP(1f) .build() Here, you can set a couple of different parameters. .setTemperature(), .setTopK(), and .setTopP(). Let’s dive them into each of them. .setTemperature() .setTopK() .setTopP() .setTemperature() can be thought of how “random” the responses from the language model can be. The lower the value, the more “predictable” the responses from the model can be. The higher the value, the more “creative” the responses will be, leading to more unexpected responses. .setTemperature() For this example, it’s set to 0.6, meaning the model will provide semi-creative, but not unimaginable responses. 0.6 The temperature is a good value to experiment with, as you may find different values provide better responses depending on your use case. In a game of Simons Says, some creativity is welcome! .setTopK() is a way of saying “only consider the top K results to return to the user.” Language models, whilst processing a prompt, generate a number of responses, potentially thousands! .setTopK() Each of these responses are given a probability as to how likely they are to be the right answer. To limit the amount of responses to consider, the topK value can be set to help focus the model. If you’re happy with less probabable responses being considered, you would want to set this value high. topK Similar to the temperarture property, this is a good property to experiment with. You may find the model works better with less or more responses to consider, depending on your needs. For a game of Simon Says, we want the model to be thinking about alot of different responses to keep the game fresh. So 5000 seems like a good value. 5000 .setTopP() builds upon the limit set by topK by saying “only consider results that have a probability of P”. As mentioned earlier, language models assign a probablility to each response it generates as to how likely it can be. Setting topP means the model can easily discard any responses that don’t have the minimum probability needed. .setTopP() topK topP To show an example, if the model had a topP set to 0.4 and was considering the following responses: topP 0.4 Simons Says clap 5 times! = 0.7 // This response has a probablility of 0.7 of being correct Simons Says jump up and down! = 0.5 Find a car and drive it. = 0.3 Simons Says clap 5 times! = 0.7 // This response has a probablility of 0.7 of being correct Simons Says jump up and down! = 0.5 Find a car and drive it. = 0.3 The first two responses would be considered because the probability is higher than 0.4. The response about the car would be discarded, as it only has a probability of 0.3. 0.4 0.3 Similar to temperature and topK, topP allows you to define how creative your language model can be. If you want to consider less probable responses to prompts, setting a low P value will help. temperature topK topP In our example, it’s set to 1.0. That’s because we want the model to be absolutely sure about its responses. It’s playing a childrens game after all! 1.0 Experimenting with these values will generate very different results from your language models. Try them out and see what happens! Where To Go Next? I hope you’ve enjoyed this walkthrough of how to add a RAG powered language model to your Android App! If you’re looking to learn more about RAG and Android, here are a few links I recommend: Simons Says App: Clone and run the sample code for this blog post to see it in action. It shows how to setup the RAG pipeline with Gemma using Android architecture best practices. MediaPipe RAG: Check out the RAG section on the MediaPipe Google Developer docs - Highly recommended reading. Setting Temperature, TopK, and TopP in LLMs: Learn more about how setting the Temperature, TopK, and TopP values can help control the results from language models. Another highly recommended article. Simons Says App: Clone and run the sample code for this blog post to see it in action. It shows how to setup the RAG pipeline with Gemma using Android architecture best practices. Simons Says App: Clone and run the sample code for this blog post to see it in action. It shows how to setup the RAG pipeline with Gemma using Android architecture best practices. Simons Says App: sample code MediaPipe RAG: Check out the RAG section on the MediaPipe Google Developer docs - Highly recommended reading. MediaPipe RAG: Check out the RAG section on the MediaPipe Google Developer docs - Highly recommended reading. MediaPipe RAG Google Developer docs Setting Temperature, TopK, and TopP in LLMs: Learn more about how setting the Temperature, TopK, and TopP values can help control the results from language models. Another highly recommended article. Setting Temperature, TopK, and TopP in LLMs: Learn more about how setting the Temperature, TopK, and TopP values can help control the results from language models. Another highly recommended article. Setting Temperature, TopK, and TopP in LLMs help control