QA Wolf is on a mission to reinvent QA for software teams

QA Wolf

384 Stories To Learn About Technology Trends

183 Stories To Learn About Engineering

I started to summarize a dozen books by hand and found it was going to take me weeks for each summary. Then I remembered about this AI revolution happening and decided I was long past due to jump into these waters. . When I began exploring the use of large language models (LLM) for summarizing large texts, I found no clear direction on how to do so Some pages give example prompts to give GPT4 with the idea that it will magically know the contents of whatever book you want summarized. (NOT) Some people suggested i need to find a model with a large context that can process my whole text in one go. (Not Yet) Some open source tools are available that allow you to upload documents to a database and answer questions based on the contents of that database. (Getting Closer) Others have suggested that you must first divide the book into sections and feed them into the LLM for summarization one at a time. (Now we’re talking) Beyond that determination, there are numerous variables which must be accounted for when implementing a given LLM. I quickly realized, despite any recommendations or model rankings available, I was getting different results than what others have. Whether its my use-case, the model format, quantization, compression, prompt styles, or what? I don’t know. All I know is, do your own model rankings under your own working conditions. Don’t just believe some chart you read online. This guide provides some specifics into my process of determination and testing out the details of above mentioned variables. . Find the complete ranking data, walkthrough, and resulting summaries on GitHub Background Key Terms Some of these terms are used in different ways, depending on the context (no pun intended). : (AKA Model) A type of Artificial Intelligence that has  been trained upon massive datasets to understand and generate human language. Large Language Model (LLM) : OpenAI’s GPT3.5 and GPT4 which have taken the world by storm. (In our case we are choosing among open source and\or freely downloadable models found on .) Example Hugging Face : A technique, , of storing documents in a database that the LLM searches among to find an answer for a given user query (Document Q/A). Retrieval Augmented Generation (RAG) developed by Meta AI (AKA Prompt, or Context) is the query provided by the user. User Instructions: Example: “Summarize the following text : ” { text } Special instructions given before the user prompt, that helps shape the personality of your assistant. System Prompt: Example: “You are a helpful AI Assistant.” User instructions, and possibly a system prompt, and possibly previous rounds of question\answer pairs. (Previous Q/A pairs are also referred to simply as context). Context: : These are special character combinations that a LLM is trained with to recognize the difference between user instructions, system prompt and context from previous questions. Prompt Style Example: <s>[INST] {systemPrompt} [/INST] [INST] {previousQuestion} [/INST] {answer} </s> [INST] {userInstructions} [/INST] Indicates the number of parameters in a given model (higher is generally better). Parameters are the internal variables that the model learns during training and are used to make predictions. For my purposes, 7B models are likely to fit on a my GPU with 12GB VRAM. 7B: This is a specific format for LLM designed for consumer hardware (CPU/GPU). Whatever model you are interested in, for use in PrivateGPT, you must find its GGUF version (commonly made by ). GGUF: TheBloke When browsing the files of a GGUF repository you will see different versions of the same model. A higher number means less compressed, and better quality. The M in K_M means “Medium” and the S in K_S means “Small”. Q2-Q8 0, K_M or K_S: This is the memory capacity of your GPU. To load it completely to GPU, you will want a model smaller size than your available VRAM. VRAM: This is the metric LLM weighs language with. Each token consists of roughly 4 characters. Tokens: What is PrivateGPT? PrivateGPT (pgpt) is an that provides a user-interface and programmable API enabling users to use LLM with own hardware, at home. It allows you to upload documents to your own local database for RAG supported Document Q/A. open source project : PrivateGPT Documentation - Overview PrivateGPT provides an containing all the building blocks required to build . The API follows and extends OpenAI API standard, and supports both normal and streaming responses. That means that, if you can use OpenAI API in one of your tools, you can use your own PrivateGPT API instead, with no code changes, if you are running privateGPT in mode. API private, context-aware AI applications and for free local Overview I began by asking questions to book chapters, using the UI\RAG. PrivateGPT Then tried pre-selecting chunks of text for summarization. This inspired Round 1 rankings: Q/A vs Summarization. Next I wanted to find which models would do the best with this task, which led to Round 2 rankings, where was the clear winner. Mistral-7B-Instruct-v0.2 Then I wanted to get the best results from this model by ranking prompt styles, and writing code to get the exact prompt style expected. After that, of course, I had to test out various system prompts to see which would perform the best. Next, I tried a few, user prompts, to determine what is the exact best prompt to generate summaries that require the least post-processing, by me. Only once each model has been targeted to its most ideal conditions can they be properly ranked against each-other. Rankings When i began testing various LLM variants, came as part of PrivateGPT's default setup (made to run on your CPU). Here, I've preferred the Q8_0 variants. mistral-7b-instruct-v0.1.Q4_K_M.gguf While I've tried 50+ different Q8 GGUF for this same task, and haven’t found any better than . Mistral-7B-Instruct 0.2 Round 1 - Q/A vs Summary I quickly discovered when doing Q/A is that I get much better results when uploading smaller chunks of data into the database, and starting with a clean slate each time. So I began splitting PDF into chapters for Q/A purposes. For my first analysis I tested out 5 different LLM for the following tasks: Asking the same 30 questions to a 70 page book chapter. Summarizing that same 70 page book chapter, divided into 30 chunks. Question / Answer Ranking - My favorite, during these tests, but when actually editing the summaries I decided it was too verbose. Hermes Trismegistus Mistral 7b - Became my favorite of models tested in this round. SynthIA 7B V2 - Not as good as I’d like. Mistral 7b Instruct v0.1 Alot of filler and took the longest amount of time of them all. It scored a bit higher than mistral on quality\usefulness, but the amount of filler just made it less enjoyable to read. CollectiveCognition v1.1 Mistral 7b the answers were too short, and made its BS stand out a little more. A good model, but not for detailed book summaries. KAI 7b Instruct Shown, for each model Number of seconds required to generate the answer Sum of Subjective Usefulness\Quality Ratings How many characters were generated? Sum of context context chunks found in target range. Number of qualities listed below found in text generated: Filler (Extra words with less value) Short (Too short, not enough to work with.) BS (Not from this book and not helpful.) Good BS (Not from the targeted section but valid.) Model Rating Search Accuracy Characters Seconds BS Filler Short Good BS hermes-trismegistus-mistral-7b 68 56 62141 298 3 4 0 6 synthia-7b-v2.0 63 59 28087 188 1 7 7 0 mistral-7b-instruct-v0.1 51 56 21131 144 3 0 17 1 collectivecognition-v1.1-mistral-7b 56 57 59453 377 3 10 0 0 kai-7b-instruct 44 56 21480 117 5 0 18 0 Summary Ranking For this first round I split the chapter contents in to sections with a range of 900-14000 characters each (or 225-3500 tokens). NOTE: Despite the numerous large context models being released, for now, I still believe smaller context results in better summaries. I don’t prefer any more than 2750 tokens (11000 characters) per summarization task. - Still in the lead. It's verbose, with some filler. I can use these results. Hermes Trismegistus Mistral 7b - Pretty good, but too concise. Many of the answers were perfect, but 7 were too short\incomplete for use. SynthIA 7B - Just too short. Mistral 7b Instruct v0.1 - Just too short. KAI 7b Instruct - Lots of garbage. Some of the summaries were super detailed and perfect, but over half of the responses were a set of questions based on the text, not a summary. CollectiveCognition v1.1 Mistral 7b Not surprisingly, summaries performed much better than Q/A, since had a precisely targeted context. Name Score Characters Generated % Diff from OG Seconds to Generate Short Garbage BS Fill Questions Detailed hermes-trismegistus-mistral-7b 74 45870 -61 274 0 1 1 3 0 0 synthia-7b-v2.0 60 26849 -77 171 7 1 0 0 0 1 mistral-7b-instruct-v0.1 58 25797 -78 174 7 2 0 0 0 0 kai-7b-instruct 59 25057 -79 168 5 1 0 0 0 0 collectivecognition-v1.1-mistral-7b 31 29509 -75 214 0 1 1 2 17 8 Find the full data and rankings on or on GitHub: , . Google Docs QA Scores Summary Rankings Round 2: Summarization - Model Ranking Again, I prefer Q8 versions of 7B models. Finding that had been released was worth a new round of testing. Mistral 7b Instruct v0.2 I also decided to test the prompt style. PrivateGPT didn’t come packaged with the Mistral prompt, so I tried both of the defaults (llama2 and llama-index). - This model had become my favorite, so I used it as a benchmark. SynthIA-7B-v2.0-GGUF (Llama-index Prompt) Star of the show here, quite impressive. Mistral-7B-Instruct-v0.2 (Llama2 Prompt) Still good, but not good as using llama-index prompt Mistral-7B-Instruct-v0.2 as - Another by the same creator as Synthia v2. Good, but not good. Tess-7B-v1.4 as - worked ok, but slowly, with llama-index prompt. Just bad with llama2 prompt. (Should test again with Llama2 "Instruct Only" style) Llama-2-7B-32K-Instruct-GGUF Summary Ranking Only summaries, Q/A is just less efficient for book summarization. Model % Difference Score Comment Synthia 7b V2 -64.43790093 28 Good Mistral 7b Instruct v0.2 (Default Prompt) -60.81878508 33 VGood Mistral 7b Instruct v0.2 (Llama2 Prompt) -64.5871483 28 Good Tess 7b v1.4 -62.12938978 29 Less Structured Llama 2 7b 32k Instruct (Default) -61.39890553 27 Less Structured. Slow Find the full data and rankings on or on . Google Docs GitHub Round 3: Prompt Style In the previous round, I noticed was performing much better with default prompt than llama2. Mistral 7b Instruct v0.2 Well, actually, the mistral prompt is quite similar to llama2, but not exactly the same. llama_index (default) system: {{systemPrompt}}
user: {{userInstructions}}
assistant: {{assistantResponse}} llama2: <s> [INST] <<SYS>>
 {{systemPrompt}}
<</SYS>>

 {{userInstructions}} [/INST] mistral: <s>[INST] {{systemPrompt}} [/INST]</s>[INST] {{userInstructions}} [/INST] with the , then prompt styles. Next I went to work . I began testing output default llama2 coding the mistral template The results of that ranking gave me confidence that I coded correctly. Prompt Style % Difference Score Note Mistral -50% 51 Perfect! Default (llama-index) -42% 43 Bad headings Llama2 -47% 48 No Structure Find the full data and rankings on or on . Google Docs GitHub Round 4: System Prompts Once I got the prompt style dialed in, I tried a few different system prompts, and was surprised by the result! Name System Prompt Change Score Comment None -49.8 51 Perfect Default Prompt You are a helpful, respectful and honest assistant. \nAlways answer as helpfully as possible and follow ALL given instructions. \nDo not speculate or make up information. \nDo not reference any given instructions or context." -58.5 39 Less Nice MyPrompt1 "You are Loved. Act as an expert on summarization, outlining and structuring. \nYour style of writing should be informative and logical." -54.4 44 Less Nice Simple "You are a helpful AI assistant. Don't include any user instructions, or system context, as part of your output." -52.5 42 Less Nice In the end, I find that works best for my summaries without any system prompt. Mistral 7b Instruct v0.2 Maybe would have different results for a different task, or maybe better prompting, but this works good so I'm not messing with it. Find the full data and rankings on or on . Google Docs GitHub Round 5: User Prompt What I already began to suspect is that I’m getting better results with less words in the prompt. Since I found the best system prompt, for , I also tested which user prompt suits it best. Mistral 7b Instruct v0.2 Prompt vs OG score note Propmt0 Write concise, yet comprehensive, notes summarizing the following text. Use nested bullet points: with headings, terms, and key concepts in bold. Focus on essential knowledge from this text without adding any external information. 43% 11 Prompt1 Write concise, yet comprehensive, notes summarizing the following text. Use nested bullet points: with headings, terms, and key concepts in bold. Focus on essential knowledge from this text without adding any external information. 46% 11 Extra Notes Prompt2 Write comprehensive notes summarizing the following text. Use nested bullet points: with headings, terms, and key concepts in bold. 58% 15 Prompt3 Create concise bullet-point notes summarizing the important parts of the following text. Use nested bullet points, with headings terms and key concepts in bold, including whitespace to ensure readability. Avoid Repetition. 43% 10 Prompt4 Write concise notes summarizing the following text. Use nested bullet points: with headings, terms, and key concepts in bold. 41% 14 Prompt5 Create comprehensive, but concise, notes summarizing the following text. Use nested bullet points: with headings, terms, and key concepts in bold. 52% 14 Extra Notes Find the full data and rankings on or on . Google Docs GitHub Perhaps with more powerful hardware that can support 11b or 30b models I would get better results with more descriptive prompting. Even with Mistral 7b Instruct v0.2 I’m still open to trying some creative instructions, but for now I’m just happy to refine my existing process. Prompt2: Wins! Write comprehensive notes summarizing the following text. Use nested bullet points: with headings, terms, and key concepts in bold. In this case, comprehensive performs better than "concise", or even than "comprehensive, but concise". However, I do caution that this will depend on your use-case. What I'm looking for is a highly condensed and readable notes covering the important knowledge. Essentially, if I didn't read the original, I should still know what information it conveys, if not every specific detail. Even if I did read the original, I’m not going to remember the majority, later on. These notes are a quick reference to the main topics. Result Using knowledge gained from these tests, I summarized my first complete book, 539 pages in 5-6 hours!!! Incredible! Instead of spending weeks per summary, I completed my first 9 book summaries in only 10 days. Plagiarism You can see the results from below for each of the texts published, here. CopyLeaks Especially considering that this is not for profit, but for educational purposes, I believe these numbers are acceptable. Book Models Character Difference Identical Minor changes Paraphrased Total Matched Eastern Body Western Mind Synthia 7Bv2 -75% 3.5% 1.1% 0.8% 5.4% Healing Power Vagus Nerve Mistral-7B-Instruct-v0.2; SynthIA-7B-v2.0 -81% 1.2% 0.8% 2.5% 4.5% Ayurveda and the Mind Mistral-7B-Instruct-v0.2; SynthIA-7B-v2.0 -77% 0.5% 0.3% 1.2% 2% Healing the Fragmented Selves of Trauma Survivors Mistral-7B-Instruct-v0.2 -75% 2% A Secure Base Mistral-7B-Instruct-v0.2 -84% 0.3% 0.1% 0.3% 0.7% The Body Keeps the Score Mistral-7B-Instruct-v0.2 -74% 0.1% 0.2% 0.3% 0.5% Complete Book of Chakras Mistral-7B-Instruct-v0.2 -70% 0.3% 0.3% 0.4% 1.1% 50 Years of Attachment Theory Mistral-7B-Instruct-v0.2 -70% 1.1% 0.4% 2.1% 3.7% Attachment Disturbances in Adults Mistral-7B-Instruct-v0.2 -62% 1.1% 1.2% 0.7% 3.1% Psychology Major's Companion Mistral-7B-Instruct-v0.2 -62% 1.3% 1.2% 0.4% 2.9% Psychology in Your Life Mistral-7B-Instruct-v0.2 -74% 0.6% 0.4% 0.5% 1.6% Completed Book Summaries In parenthesis is the page count of the original. Instead of spending weeks per summary, I completed my first 9 book summaries in only 10 days. Anodea Judith (436 pages) Eastern Body Western Mind Stanley Rosenberg (335 Pages) Healing Power of the Vagus Nerve Dr. David Frawley (181 Pages) Ayurveda and the Mind Janina Fisher (367 Pages) Healing the Fragmented Selves of Trauma Survivors John Bowlby (133 Pages) A Secure Base Bessel van der Kolk (454 Pages) The Body Keeps the Score Steven Porges (37 pages) Yoga and Polyvagal Theory, from Polyvagal Safety Cynthia Dale (999 pages) Llewellyn's Complete Book of Chakras SECTION 1. CHAKRA FUNDAMENTALS AND BASIC PRACTICES s SECTION 2: CHAKRAS IN DEPTH. HISTORICAL, SCIENTIFIC, AND CROSS-CULTURAL UNDERSTANDINGS (54 pages) Fifty Years of Attachment Theory: The Donald Winnicott Memorial Lecture (477 Pages) Attachment Disturbances in Adults Dana S. Dunn, Jane S. Halonen (308 Pages) The Psychology Major's Companion (5 Pages) Walter Wink The Myth of Redemptive Violence Sarah Gison and Michael S. Gazzaniga (1072 Pages) Psychology In Your Life Walkthrough If you are interested to follow my steps more closely, check out the containing scripts and examples. walkthrough on GitHub, Conclusion Now that I have my processes refined, and feel confident working with prompt formats, I will conduct further tests. In fact, i already have conducted further tests and rankings (will publish those next), but of course will do more tests again and continue learning! I still believe if you want to get the best results for whatever task you perform with AI, you ought to run your own experiments and see what works best. Don’t rely solely on popular model rankings, but use them to guide your own research. Additional Resources ( ) Pressure-tested the most popular open-source LLMs (Large Language Models) for their Long Context Recall abilities u/ramprasad27 Part 2 / - 💢 Pressure testing the context window of open LLMs LeonEricsson llmcontext Chatbox Arena Leaderboard u/WolframRavenwolf 🐺🐦‍⬛ LLM Comparison/Test: Ranking updated with 10 new models (the best 7Bs)! u/WolframRavenwolf 🐺🐦‍⬛ LLM Prompt Format Comparison/Test: Mixtral 8x7B Instruct with 17 different instruct templates Vectara Hallucination leaderboard Also appears . here

PrivateGPT for Book Summarization: Testing and Ranking Configuration Variables

বই সংক্ষিপ্তকরণের জন্য ব্যক্তিগত জিপিটি: টেস্টিং এবং র‌্যাঙ্কিং কনফিগারেশন ভেরিয়েবল

PrivateGPT für die Buchzusammenfassung: Test- und Ranking-Konfigurationsvariablen

PrivateGPT para resumen de libros: variables de configuración de prueba y clasificación

PrivateGPT pour la synthèse de livres : variables de configuration de test et de classement

पुस्तक सारांश के लिए PrivateGPT: परीक्षण और रैंकिंग कॉन्फ़िगरेशन चर

書籍の要約のための PrivateGPT: 構成変数のテストとランク付け

도서 요약을 위한 PrivateGPT: 구성 변수 테스트 및 순위 지정

PrivateGPT para resumo de livros: teste e classificação de variáveis de configuração

PrivateGPT для обобщения книг: тестирование и ранжирование переменных конфигурации

Kitap Özetleme için PrivateGPT: Yapılandırma Değişkenlerini Test Etme ve Sıralama

PrivateGPT dành cho tóm tắt sách: Kiểm tra và xếp hạng các biến cấu hình

用于书籍摘要的 PrivateGPT：测试和排名配置变量

Anatomy, Zoology and Physiology

If you have a lot of private enterprise data, how can you use a ChatGPT-like AI system to help you search for relevant information faster? Or if you have lots of personal information, how can you find old memories without the exact keywords? In this practical guide, I will show you 5 simple steps to implement semantic search with the help of LangChain, vector databases, and large language models. Best of all, I will use all open-source components that can be run locally on your own machine. No need for any cloud SaaS or API keys, and your data will never leave your office or home. Use Case Analysis Our has a based forum that has been running for over 5 years. There is a wealth of information, but sometimes, it is hard for newcomers to find the right answers to their questions quickly and succinctly. community Discourse There are a few reasons, which are very common for Google/pagerank based search solutions: Search is based on keywords, and often, users do not know the right terms or exact wording. Even if the user figured out the best search terms for their particular question, the return result is a list of top discussion threads. And the answer to the question might not be in the first few posts or could be buried deep in a prolonged discussion thread. In general, search engines do not understand long sentences very well if you ask a long-winded question or two questions in one sentence. You might ask another question, “ChatGPT v4 is already so good, why don’t you ask questions directly from chatGPT?” Again, there are several strong reasons: ChatGPT (v3.5 or v4) model was trained on data before November 2021, thus, it does not have the latest information since then. The latest OpenAI website retrieval plug-in can search over the website for the latest info, but it is still quite slow and often stuck. It is neither efficient nor cost-effective for your enterprise's internal use. You don’t want OpenAI and Microsoft to access your proprietary enterprise data or your personal data. So, my goal is that if someone comes in and asks, “What is the easiest way to run an NKN node?”, he/she should be able to get a straight answer in a few sentences. I use this public NKN forum only as an example to show you the typical dataset and typical questions and answers for a special enterprise knowledge base. Design Approach After some research and experiments, I narrowed down my plan into 5 steps: Do a quick Proof of Concept using cloud service and API Clean and prep my data Convert my entire data set into a vector database Use vector database to find the best matches to my query Use LLM to distill the best answer to the original question If you want to pause here and learn some basics about LangChain, vector databases, large language models, and how they work together, I will recommend and . And below, is a short introduction to these concepts. this video this video Key Components : LangChain is a framework for developing applications powered by language models. It is the glue or middleware that chains all the different components together to accomplish complex tasks semi-autonomously. Langchain The most powerful and differentiated applications will not only call out to a language model via an API but will also: Be data-aware: Connect a language model to other sources of data. Be agentic: Allow a language model to interact with its environment. : A vector database is a specialized storage and retrieval system designed to handle high-dimensional vector data efficiently and is often used in machine learning, artificial intelligence, and data analytics applications. Vector database These databases enable users to perform similarity searches or nearest neighbor searches, allowing them to find the most similar items within the database based on their vector representations. Pinecone is one of the leading commercial vector databases. There are open-source alternatives such as FAISS, Chroma, and Supabase. : A large language model like GPT (Generative Pre-trained Transformer) is an advanced artificial intelligence model that utilizes deep learning techniques to process and generate human-like text. Large Language Model With billions of parameters, GPT-3/4 is trained on massive amounts of data from diverse sources, allowing it to understand context, semantics, and syntax. As a result, GPT-3/4 can perform various natural language processing tasks, such as text generation, translation, summarization, and question-answering, with remarkable accuracy and fluency. There are several community-driven LLM models based on Meta’s LLaMA, e.g., Alpaca, Vicuna, Koala, and GPT4All, which can be run locally on a laptop (read ). my tutorial here Step 1: A Proof of Concept Before you spend more effort on your custom solution, let’s first get familiar with the key framework LangChain and the components in a playground. Thanks to GUI-based , you can now play with this entire solution without any code. flowise.ai This step is optional, and you can skip to step 2 if you want to go straight into coding locally or if you don’t have API keys for OpenAI or Pinecone. Prerequisites: You obtain the and in advance. Both are useful not only for this exercise but also for other AI-related experiments. OpenAI API key Pinecone API key In Pinecone, you need to create a new index called “ ” with dimension . Note there is an environment name auto-generated by Pinecone to designate where this index instance is run: “ ” You can use the default metrics of “ ” test 1536 us-west1-gcp-free. Cosine. A small dataset in text file format (let’s say ): it could be as simple as copy-paste a few paragraphs from a Word or PDF document. In addition, for privacy reasons, you want to use a very small data set and preferably publicly available information. Since both APIs are cloud-based. test.txt Now. head to , download the code from their , and install and run locally (you need to have Node Javascript installed). You can access it from a browser: . Flowise AI Github http://localhost:3000/marketplaces And from Marketplace, you can choose a template called “ ”, and use this template. The UI is self-explanatory: you fill in the OpenAI API key twice (one for embedding and one for LLM), the Pinecone API key, index name, environment, and metrics. Conversational QA Retrieval Chain And upload the data set text file. It might take a couple of minutes depending on the size of your text film. Then you click the little chat bubble on the top right corner, and you can start to ask any questions! So this will give you a visual understanding of the data processing pipeline of our forum use case, and help you implement your own solution by replacing all the cloud API with the local service using open-source software. LangChain will stay put since it is an open-source software and the master glue we need. Large language model: -> OpenAI LLaMA/Alpaca Vector database: -> Pinecone FAISS But before we do that, we first have to clean up and prepare your enterprise data for ingestion into the vector database. Step 2: Clean and Prep Data The NKN forum uses Discourse software, a popular open-source discussion and community forum solution. It uses the Postgres SQL database as its core data store, which has two tables that are especially relevant for our Q&A retrieval use case topics posts. For example, the database schema of “ ” is listed partially below. The most relevant columns for us are: “ ” (matching to the topics table), “ ” (original post content in the text without formatting), and “ ” (how popular or well-read is this post). posts topic_id raw score It is quite natural to map “ ” to questions (or “ ”), and “ ” to answers (“ ”). The terms in parentheses are for GPT-like LLMs. There are many implementation considerations and optimizations here, which would suffice for another article. topics prompts posts::raw completion But I would like to keep it simple and choose only the best post out of the entire thread to match the original topic. So first of all, we need to get both tables out of the database and into JSON files. To do that, I did a DB dump in Discourse, and then import it into a standalone Postgres SQL database running on my Macbook Pro M1. From the standalone Postgres SQL, I can export JSON files for the two tables: and . posts topics Then I wrote some simple Python script to do the following: For each topic, use SQL to select the posts with the highest “score.” The resulting “topic” and “post::raw” become one “prompt” and “completion” pair. Save the above into a JSON file called training.json (you can use it for many other training purposes, e.g., finetuning an LLM model, using GPT-3’s finetuning API, etc.) Write into a text file called training.txt: one line for prompt, and one line for completion, and separate each pair with two new lines. Now your file should look a bit like this: Step 3: Convert My Entire Data Set Into a Vector Database I pasted the Python script below, which is mostly self-explanatory. A few notes: You need to download or find the weights for your local LLM model. In my case, I used Alpaca 7B parameter 4-bit quantized version. You can find them on huggingface. There are many similar ones that are fine-tuned based on Meta’s LLaMA 7B/13B models, e.g., Alpaca, Vicuna, Koala, GPT4All. You can use any of them. Optionally, you have followed and installed Llama.cpp locally and successfully ran Llama.cpp with the weights from above. So you know the weights are good. my instruction The text splitter will split your long text file into chunks, each having a max length of . 1,000 Then, this chunk of text is fed into the LLM for embedding, which basically tokenizes the words, and converts them into vectors based on their meaning. Each chunk, after tokenization, might become up to 1,400 tokens. Since on average, 3 words create about 4 tokens. This is also the reason I have (default is 512) in order to have a large enough context window for embedding. n_ctx = 2048 Finally, the vectors are inserted into the FAISS vector database and indexed. The database is saved as a local file in a directory named “faiss_index”. from langchain.embeddings import LlamaCppEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.vectorstores import FAISS
from langchain.document_loaders import TextLoader

embeddings = LlamaCppEmbeddings(model_path="./ggml-alpaca-7b-q4.bin", n_ctx= 2048)

from langchain.document_loaders import TextLoader
loader = TextLoader('./training_full_clean.txt')
documents = loader.load()
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size = 1000, 
    chunk_overlap = 50,
    length_function = len)
docs = text_splitter.split_documents(documents)

db = FAISS.from_documents(docs, embeddings)
db.save_local("faiss_index")

# Now let's test it out
query = "Who is Whitfield Diffie?"
docs = db.similarity_search(query)
for doc in docs:
    print(doc.page_content) I highly recommend you start with a very small dataset text file to test first. If your dataset is large, this will take a long time. My locally run Llama.cpp can embed about 20 tokens per second, so each chunk of 1,000 text words could take about 1 minute to do embedding. My vector DB has about 7,500 entries, and it will take me about 100 hours on a Macbook Pro M1. The good news is that you only need to do this once. [Author note: please see comments to use different embedding module such as huggingface for much faster performance on large datasets] Step 4: Use a Vector Database to Find the Best Matches to My Query Again, I provide some simple Python code to do the query and summarization. Since we use a template chain called “ ”, some details might not be obvious. But under the hood, it involved the following steps: RetrievalQA When a user types in a query, the large language model converts the text into embedding and a vector (basically an array of floating point numbers) representing the query. The vector database then does a similarity search to find the candidate result vectors that are closest related to the query vector. Now, we return those candidate result vectors, which are already encoded in the way that LLM understands, to LLM for final processing. from langchain.vectorstores import FAISS
from langchain.llms import LlamaCpp
from langchain.embeddings import LlamaCppEmbeddings
from langchain.callbacks.base import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain.chains import RetrievalQA

# Load DB from local file
embeddings = LlamaCppEmbeddings(model_path="./ggml-alpaca-7b-q4.bin")
new_db = FAISS.load_local("faiss_index", embeddings)

# Use llama-cpp as the LLM for langchain
callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])
llm = LlamaCpp(
    model_path="./ggml-alpaca-7b-q4.bin",
    n_ctx= 2048,
    callback_manager=callback_manager, 
    verbose=True,
    use_mlock=True
)

retriever = new_db.as_retriever()

# Conversational QA retrieval chain
qa = RetrievalQA.from_chain_type(llm=llm, chain_type="stuff", retriever=retriever)

while True:
    user_input = input("You: ")
    if user_input.lower() == "exit":
        break

    response = qa.run(user_input)
    print(f"AI: {response}") Step 5: Use LLM to Distill the Best Answer to the Original Question In the final step and again under the hood, the “ ” template automatically constructs a special , which includes the following: RetrieverQA prompt : Instruct the LLM what kind of task it needs to do, and in this case, “ ” System message Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer. : Add the best candidate result vectors as context, as well as the query vector at the end. User message Once LLM receives such a prompt, it will try its best to sift through the candidate results and find the best answer to the original question. It summarizes, rephrases, and even finds separate responses to two separate questions in the original query (e.g., “Who is Stephen Wolfram, and how is he related to NKN?”). Sometimes, we can be quite amazed at how capable LLMs have become to handle this type of Q&A task. What Next? Now, you are well on your way to your office conference room to show off your new demo! And what can you do next? Congratulations, you have just successfully built your first semantic search tool on your private data using an entirely open-source toolkit! Try alternative components for vector databases or large language models that fit your data’s needs best. Optimize, scale, and productize your semantic search solution for your entire company. Dig deeper into LangChain and its infinite capabilities to solve other pressing work or life-related problems. Maybe you can even contribute your own chain as a template for other people to use. Happy hacking! 🙂

A Practical 5-Step Guide to Do Semantic Search on Your Private Data With the Help of LLMs

ECommerce Replatforming: How to Survive the Switch and Succeed in the Process?

Cambio de plataforma de comercio electrónico: ¿Cómo sobrevivir al cambio y tener éxito en el proceso?

Replateforming du commerce électronique : comment survivre au changement et réussir le processus ?

ईकामर्स रीप्लेटफॉर्मिंग: स्विच से कैसे बचे और प्रक्रिया में सफल हों?

e コマースの再プラットフォーム化: 移行を生き延び、プロセスを成功させるには?

Replatforming de comércio eletrônico: como sobreviver à mudança e ter sucesso no processo?

Tái tạo nền tảng thương mại điện tử: Làm thế nào để sống sót sau quá trình chuyển đổi và thành công trong quá trình này?

电子商务平台重塑：如何在转变中幸存下来并在此过程中取得成功？

This paper is under CC 4.0 license. available on arxiv Authors: (1) Jinrui Yang, School of Computing & Information Systems, The University of Melbourne (Email: jinruiy@student.unimelb.edu.au); (2) Timothy Baldwin, School of Computing & Information Systems, The University of Melbourne and Mohamed bin Zayed University of Artificial Intelligence, UAE (Email: (tbaldwin,trevor.cohn)@unimelb.edu.au); (3) Trevor Cohn, School of Computing & Information Systems, The University of Melbourne. Table of Links Abstract and Intro Background and Related Work Multi-EuP Experiments and Findings Language Bias Discussion Conclusion, Limitations, Ethics Statement, Acknowledgements, References, and Appendix 6 Conclusion In this paper, we introduce Multi-EuP, a novel dataset for multilingual information retrieval across 24 languages, collected from European Parliament debates. The demographic information provided by the Multi-EuP dataset serves a dual purpose: not only does it contribute to multilingual retrieval tasks, but it also holds significant potential for advancing research in the realm of fairness and bias. This dataset can play a pivotal role in investigating issues of equitable representations and mitigation of biases within document ranking settings. Multi-EuP facilitates diverse information retrieval (IR) scenarios, encompassing one-vs-one, one-vs-many, and many-vs-many settings. We demonstrated the utility of Multi-EuP as a benchmark for evaluating both monolingual and multilingual IR. Our study reveals the presence of language bias in multilingual IR when employing BM25. We further validate the effectiveness of mitigating this bias through the strategic implementation of whitespace as a language tokenizer. We propose to conduct future work in three main areas. First, we intend to expand our investigation of language bias to encompass a broader range of ranking methods, including neural methods such as mDPR (Zhang et al., 2021), mColBERT (Lawrie et al., 2023) and PLAID-X(Santhanam et al., 2022). Second, we will expand the dataset by developing an automated API to retrieve data published by the European Parliament (EP), thereby ensuring realtime synchronization of our dataset. Lastly, our current experiments have explored language bias only, but we plan to further investigate gender bias, age bias, and nationality bias. Limitations The limitations of the Multi-EuP dataset are notable but navigable. Primarily, the temporal coverage of the dataset is confined to the past three years. This temporal constraint arises due to the fact that, preceding 2020, documents released by the EU were predominantly available in mono-lingual versions only. However, a potential remedy lies in the amalgamation of the Europarl (Koehn, 2005) collection, enabling a more comprehensive and holistic MultiEuP dataset. Furthermore, it is worth noting the domain skew of the dataset, in that Multi-EuP inevitably centers on political matters. While this presents challenges, particularly in terms of the intricate nuances of political language, it inherently serves as an excellent foundational stepping stone for delving into the intricacies of multilingual retrieval. We believe, however, that this dataset can serve as a launching pad for broader explorations encompassing crossdomain and open-domain transfer learning scenarios, thus contributing to the broader landscape of language understanding and retrieval. Ethics Statement The dataset contains publicly-available EP data that does not include personal or sensitive information, with the exception of information relating to public officeholders, e.g., the names of the active members of the European Parliament, European Council, or other official administration bodies. The collected data is licensed under the Creative Commons Attribution 4.0 International licence. [8] Acknowledgements This research was funded by Melbourne Research Scholarship and undertaken using the LIEF HPCGPGPU Facility hosted at the University of Melbourne. This facility was established with the assistance of LIEF Grant LE170100200. We would like to thank George Buchanan for providing valuable feedback. References Luiz Henrique Bonifacio, Israel Campiotti, Roberto de Alencar Lotufo, and Rodrigo Frassetto Nogueira. 2021. mMARCO: A multilingual version of MS MARCO passage ranking dataset. CoRR, abs/2108.13897. Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. 2021. MultiEURLEX - a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6974–6996, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. 8 https://eur-lex.europa.eu/cont Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for opendomain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over BERT. CoRR, abs/2004.12832. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79–86, Phuket, Thailand. Dawn Lawrie, Eugene Yang, Douglas W. Oard, and James Mayfield. 2023. Neural approaches to multilingual information retrieval. arXiv cs.IR 2209.01335. Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, JhengHong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. https://github.com/ castorini/pyserini. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. CoRR, abs/1611.09268. Ella Rabinovich, Raj Nath Patel, Shachar Mirkin, Lucia Specia, and Shuly Wintner. 2017. Personalized machine translation: Preserving original author traits. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1074–1084, Valencia, Spain. Association for Computational Linguistics. Razieh Rahimi, Azadeh Shakery, and Irwin King. 2015. Multilingual information retrieval in the language modeling framework. Information Retrieval Journal, 18:246–281. Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. 2022. PLAID: An efficient engine for late interaction retrieval. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM ’22, page 1747–1756, New York, NY, USA. Association for Computing Machinery. Jörg Tiedemann and Santhosh Thottingal. 2020. OPUSMT – building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479–480, Lisboa, Portugal. European Association for Machine Translation. Eva Vanmassenhove and Christian Hardmeier. 2018. Europarl datasets with demographic speaker information. In Proceedings of the 21st Annual Conference of the European Association for Machine Translation, page 391, Alicante, Spain. Denny Vrandeciˇ c and Markus Krötzsch. 2014. ´ Wikidata: A free collaborative knowledge base. Communications of the ACM, 57:78–85. Konrad Wojtasik, Vadim Shishkin, Kacper Wołowiec, Arkadiusz Janz, and Maciej Piasecki. 2023. BEIRPL: Zero shot information retrieval benchmark for the Polish language. arXiv cs.IR 2305.19840. Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the use of lucene for information retrieval research. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1253–1256. Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021. Mr. TyDi: A multi-lingual benchmark for dense retrieval. arXiv cs.CL 2108.08787. A. Appendix [8] https://eur-lex.europa.eu/content/ legal-notice/legal-notice.html

Multi-EuP: Analysis of Bias in Information Retrieval - Conclusion, Limitations, and Ethics Statement

260 Stories To Learn About Team Productivity

QA WOLF

qawolf.com

-- employees

Since n.d.

COMPANY RANKING

QA WOLF

EVERGREEN INDEX #8828

Representing this company? Click here to claim and customize this page!

QA Wolf WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

QA WOLF

qawolf.com

-- employees

Since n.d.

COMPANY RANKING

QA WOLF

EVERGREEN INDEX #8828

RELATED COMPANIESRANK

STORIES ABOUT QA WOLF

Representing this company? Click here to claim and customize this page!

QA Wolf WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!