“The potential of today’s new era of AI can only be fully realized, in my opinion, if organizations have a strategy to unify data from multiple sources in near real-time without creating numerous copies of data and going through constant iterations of ...

IBM Unleashes the Potential of Data and AI with its Next-Gen IBM Storage Scale System 6000

Its fall 2023 platform release jump-starts GenAI innovation by transforming Domino’s AI Project Hub into an AI ecosystem hub with contributions from cutting-edge AI companies, reduces time to value with expanded data ... to build machine learning models ...

Domino Data Lab Unveils New Generative AI Capabilities in Latest Platform Release

Using data harvested mainly from more than one million mobile phones, the map gave the soldiers a real-time assessment of how many Gazans had heeded Israel’s demand. The Israel Defense Forces ...

Tracking Cellphone Data by Neighborhood, Israel Gauges Gaza Evacuation

To overcome the drawbacks of conventional macro-variable-driven techniques, a team of Japanese researchers developed a big data-driven ... various real-time data sources to enhance the precision ...

Unveiling real-time economic insights with search big data

A growing number of AI tools are being used to predict everything from sepsis to strokes, with the hope of accelerating the delivery of life-saving care. But over time, new research suggests ...

The ‘model-eat-model world’ of clinical AI: How predictive power becomes a pitfall

Empire's website claims to integrate AI machine learning into its automation ... What is different this time is the content produced by AI can be so real," Constellation Research VP and Principal ...

Generative AI is everything, everywhere, all at once

AI can enable business transformation to deliver positive outcomes for clients and propel sustainability goals, according to Sandeep Davé, chief digital and technology officer at CBRE. In ...

Using data, AI, and cloud to transform real estate

The new integration opens various use cases for AI in real-time data processing. For instance, users can analyze and rate the sentiment of product reviews as they stream in or automatically ...

Confluent Introduces Data Streaming for AI + New Partnerships

Students will pursue an applied or computational track and will gain competency in fundamental methods and techniques in artificial intelligence and machine learning. Their foundational understanding will be applied to real data sets and data analysis ...

Master’s (MS) in Machine Learning and Artificial Intelligence

Who is the Applied AI/ML for Data Science Certificate for? Drexel College of Computing & Informatics' Post-Baccalaureate/Graduate Certificate in Applied Artificial Intelligence & Machine Learning for Data Science provides the quantitative foundations ...

Graduate Certificate in Applied Artificial Intelligence and Machine Learning for Data Science

DataStax is the real-time data company for building production GenAI applications.

DataStax

How ColBERT Helps Developers Overcome the Limits of RAG

Want to Learn Warhammer 40K? Build This Chatbot with Vectors and RAG on Commodity Hardware

Achieving Relevant LLM Responses By Addressing Common Retrieval Augmented Generation Challenges

A look at two databases that have made claims to the Kubernetes native label: TiDB and DataStax Astra DB. The cloud computing revolution has inspired and benefitted from multiple interrelated trends. The availability of self-service, public cloud infrastructure has helped to drive the adoption of microservice architectures and DevOps practices, including automation and observability. The drive toward containerization and container orchestration has led to the widespread adoption of Kubernetes as an environment for managing cloud-native applications. But one of the lagging areas in this revolution has been data and data infrastructure. For too long, data has been something that has lived outside of Kubernetes, leading to a lot of extra effort and complexity for developers in deploying cloud-native applications. One oft-repeated axiom in the early years of Kubernetes was that it was not yet ready for stateful workloads. Thankfully, a major shift has been quietly underway and has reached a point of maturity. The transformation happened slowly initially, beginning with efforts to containerize existing databases. This worked relatively well in small databases that ran on a single compute node, or databases that had been designed in a cloud-native world, like Apache Cassandra and DynamoDB, but challenges remained. Over the past two to three years, a new generation of databases has emerged. These “Kubernetes native” databases have been designed from the ground up to run on this open-source orchestration system. Here, we’ll define the qualities that make a database Kubernetes native and the benefits of adopting a Kubernetes native database. To do that, we’ll look at two databases claiming the Kubernetes native label: TiDB and DataStax Astra DB. Kubernetes Native MySQL with TiDB First, let’s examine a database with a relational emphasis: (short for Titanium Database). TiDB is an open-source system built by PingCAP that provides a and a columnar database to support hybrid transactional and analytic processing (known as HTAP, for short). TiDB MySQL-compatible database As shown in Figure 1 below, TiDB has a microservice design. The TiDB query layer, TiKV MySQL databases, TiFlash columnar databases, Spark nodes, and metadata management are each deployed as scalable microservices in their clusters. This design separates compute-intensive work from storage-intensive work, as the query and database layers are independently scalable. One critical commitment the TiDB creators made was that the database only runs on Kubernetes. Is that enough to make it Kubernetes native? Let’s dig a bit deeper. First, TiDB is deployed and managed by a Kubernetes operator using (CRDs). The TiDB CRDs include the TiDBCluster, which enables you to specify the scaling and configuration of each microservice and how the database layer components use storage through Kubernetes Persistent Volumes. Additional CRDs are used to deploy monitoring tools and manage operational tasks like backup and restore. custom resources that interfaces with the default K8s scheduler to make more application-aware scheduling decisions. This emphasis on using existing Kubernetes capabilities where available is the mark of a Kubernetes native database. TiDB also has an optional scheduler extension Kubernetes Native Cassandra with DataStax Astra DB Now, look at another Kubernetes native database and note some similarities and differences. that was one of the first to claim to be cloud native, but what does it look like to deploy Cassandra in Kubernetes? is a highly scalable NoSQL database Cassandra that has been factored into microservices, as shown in Figure 2. is a version of Cassandra DataStax Astra DB Like TiDB, the database includes microservices concerned with query processing and data storage, as well as services for identity and access control, data repair, and backup/restore. The data services are particularly interesting in their use of , with Kubernetes Persistent Volumes used only for caching and object storage used for longer-term persistence. Separating compaction into its service enables this compute-intensive processing to happen in the background without affecting the performance of data services serving read and write traffic. storage Astra DB is offered as a managed service available in multiple cloud regions. Each region contains a data plane consisting of the services mentioned above, managed by a Kubernetes operator, as well as infrastructure services, including the stack for observability and for metadata management. Kube-Promethus etcd The data planes are managed by a control plane that can run in one or more clouds to manage customer accounts and databases and provision Kubernetes clusters in new regions. in which multiple user databases can share the same microservices and supporting infrastructure, lowering unit economics for smaller-scale users. One novel aspect of Astra DB is its multi-tenant architecture As users grow their applications, they can move to dedicated resources to achieve optimal performance at scale, all on a “ ” basis. pay-as-you-go Kubernetes Native Database Principles Based on our observations of TiDB and Astra DB, we can derive some ideas of what makes a database Kubernetes native. Many of these correspond to a list of principles for cloud-native data, which I described in an : earlier article First, a database broken into constituent microservices enables each service to be scaled independently. Some types of compute-intensive processing may even be scaled to zero for a true serverless solution, especially when combined with a multitenant design. Composable microservice architecture: Microservices composed of a Kubernetes native database should make maximum usage of Kubernetes APIs for managing the fundamental resources of cloud-native applications: compute resources such as , Kubernetes ingress and services for exposing network access to data and more. This includes leveraging capabilities already present in Kubernetes, such as etcd for metadata management, instead of bringing along components with duplicative functionality. Treat compute, network, and storage as commodities: StatefulSets and deployments for managing workloads, the Persistent Volume subsystem for storage Following common patterns for Kubernetes applications will yield multiple operational benefits, for example, . By default, Kubernetes itself sets a great example that databases should follow for how to be secure: using Kubernetes Secrets to distribute security credentials, only exposing ports as needed, and so on. Leverage Kubernetes best practices: exposing liveness and readiness checks on each microservice to help availability and exposing metrics via the Prometheus PromQL API for observability A Kubernetes native database should embody the Kubernetes principles of declarative management via operators and custom resources, rather than relying on legacy database management UIs and CLIs. When necessary, . The goal is a clean separation of data plane functionality (managing data) from control plane functionality (managing the database). Declarative management via operators: Kubernetes extension points, such as scheduler extensions, can be used to add application-specific behavior Databases and other data infrastructures that faithfully adopt these principles will yield benefits, including a , lower operational complexity resulting in faster time to market, and standards-compliant solutions meeting today’s high availability and security demands. high performance for optimal cost at all scales The Future of Kubernetes Native Data Infrastructure Much progress is still to be made, and it’s not limited to databases alone. Kubernetes native principles can be applied to other types of data infrastructure, including streaming, analytics, and machine learning. Kubernetes native solutions will continue to make strides in multicluster and multi-cloud deployments to scale globally and will adopt multitenancy and serverless principles for better cost optimization. Kubernetes itself has room for improvement in adding more flexibility to StatefulSets and support for multicluster federation. The key to continued progress is open collaboration. The is a highly active group of data geeks bringing together builders of data-intensive applications and the infrastructure that supports them. Data on Kubernetes Community Join us to talk about ideas like developing reusable operators that can manage multiple databases or defining a common set of CRDs for concepts like backup/restore and data loading. Together we’ll continue to push the horizon of cloud computing for the benefit of all. Learn more about Kassandra native databases and more at the digital summit on March 14, 2023. Cassandra Forward This article is based on Chapter 7, “The Kubernetes Native Database,” from the O’Reilly book “ Managing Cloud Native Data on Kubernetes ” by Jeff Carpenter and Patrick McFadin. [ By Jeff Carpenter, DataStax Jeff Carpenter has worked as a software engineer and architect in multiple industries and as a developer advocate at DataStax, helping engineers succeed with Apache Cassandra. He's involved in multiple open source projects in the Cassandra and Kubernetes ecosystems, including Stargate and K8ssandra. He is co-author of the O’Reilly books "Cassandra: The Definitive Guide" and "Managing Cloud Native Data on Kubernetes."

Kubernetes-Native Database: TiDB Vs. DataStax Astra DB

Base de datos nativa de Kubernetes: TiDB vs. Base de datos DataStax Astra

Base de données native Kubernetes : TiDB Vs. DataStax Astra DB

कुबेरनेट्स-नेटिव डेटाबेस: TiDB बनाम। डेटास्टैक्स एस्ट्रा डीबी

Kubernetes ネイティブ データベース: TiDB Vs. DataStax アストラ DB

Banco de dados nativo do Kubernetes: TiDB vs. Data Stax Astra DB

Cơ sở dữ liệu gốc Kubernetes: TiDB Vs. DataStax Astra DB

Kubernetes 原生数据库：TiDB 对比。 DataStax Astra 数据库

Retrieval augmented generation (RAG) is by now of generative artificial intelligence (AI) applications. Supplementing your application prompt with relevant context retrieved from a vector database can dramatically increase accuracy and reduce hallucinations. This means that increasing relevance in vector search results has a direct correlation to the quality of your RAG application. a standard part There are two reasons RAG remains popular and increasingly relevant even : as large language models (LLMs) increase their context window LLM response time and price both increase linearly with context length. LLMs still struggle with both and across massive contexts. retrieval reasoning But isn’t a magic wand. In particular, the most common design, dense passage retrieval (DPR), represents both queries and passages as a single embedding vector and uses straightforward to score relevance. This means DPR relies heavily on the embeddings model having the breadth of training to recognize all the relevant search terms. RAG cosine similarity Unfortunately, off-the-shelf models struggle with unusual terms, including names, that are not commonly in their training data. DPR also tends to be hypersensitive to chunking strategy, which can cause a relevant passage to be missed if it’s surrounded by a lot of irrelevant information. All of this creates a burden on the application developer to “get it right the first time,” because a mistake usually results in the need to rebuild the index from scratch. Solving DPR’s challenges with ColBERT is a new way of scoring passage relevance using a language model that substantially solves the problems with DPR. This diagram from shows why it’s so exciting: ColBERT BERT the first ColBERT paper This compares the performance of ColBERT with other state-of-the-art solutions for the MS-MARCO dataset. (MS-MARCO is a set of Bing queries for which scored the most relevant passages by hand. It’s one of the better retrieval benchmarks.) Lower and to the right is better. Microsoft In short, ColBERT handily outperforms the field of mostly significantly more complex solutions at the cost of a small increase in latency. To test this, I created and indexed over 1,000 Wikipedia articles with both ada002 DPR and ColBERT. I found that ColBERT delivers significantly better results on unusual search terms. a demo The following screenshot shows that DPR fails to recognize the unusual name of William H. Herndon, an associate of Abraham Lincoln, while ColBERT finds the reference in the Springfield article. Also note that ColBERT’s No. 2 result is for a different William, while none of DPR’s results are relevant. ColBERT is often described in dense machine learning jargon, but it’s actually very straightforward. I’ll show how to implement ColBERT retrieval and scoring on with only a few lines of Python and (CQL). DataStax Astra DB Cassandra Query Language The big idea Instead of traditional, single-vector-based DPR that turns passages into a single “embedding” vector, ColBERT generates a contextually influenced for each token in the passages. ColBERT similarly generates vectors for each token in the query. vector (Tokenization refers to breaking up input into fractions of words before processing by an LLM. .) Andrej Karpathy, a founding member of the OpenAI team, just released an outstanding video on how this works Then, the score of each document is the sum of the maximum similarity of each query embedding to any of the document embeddings: def maxsim(qv, document_embeddings):
    return max(qv @ dv for dv in document_embeddings)

def score(query_embeddings, document_embeddings):
    return sum(maxsim(qv, document_embeddings) for qv in query_embeddings) (@ is the PyTorch operator for dot product and is the most common .) measure of vector similarity That’s it — you can implement ColBERT scoring in four lines of Python! Now you understand ColBERT better than 99% of the people posting about it on X (formerly known as Twitter). The rest of the ColBERT papers deal with: How do you fine-tune the BERT model to generate the best embeddings for a given data set? How do you limit the set of documents for which you compute the (relatively expensive) score shown here? The first question is optional and out of scope for this writeup. I’ll use the pretrained ColBERT checkpoint. But the second is straightforward to do with a vector database like DataStax Astra DB. ColBERT on Astra DB There is a popular Python all-in-one library for ColBERT called ; however, it assumes a static dataset. One of the powerful features of RAG applications is responding to dynamically . So instead, I’m going to use Astra’s vector index to narrow the set of documents I need to score down to the best candidates for each subvector. RAGatouille changing data in real time There are two steps when adding ColBERT to a RAG application: ingestion and retrieval. Ingestion Because each document chunk will have multiple embeddings associated with it, I’ll need two tables: CREATE TABLE chunks (
    title text,
    part int,
    body text,
    PRIMARY KEY (title, part)
);

CREATE TABLE colbert_embeddings (
    title text,
    part int,
    embedding_id int,
    bert_embedding vector<float, 128>,
    PRIMARY KEY (title, part, embedding_id)
);

CREATE INDEX colbert_ann ON colbert_embeddings(bert_embedding)
  WITH OPTIONS = { 'similarity_function': 'DOT_PRODUCT' }; After installing the ColBERT library ( ) and downloading , I can load documents into these tables: pip install colbert-ai the pretrained BERT checkpoint from colbert.infra.config import ColBERTConfig
from colbert.modeling.checkpoint import Checkpoint
from colbert.indexing.collection_encoder import CollectionEncoder

from cassandra.concurrent import execute_concurrent_with_args
from db import DB


def encode_and_save(title, passages):
    db = DB()
    cf = ColBERTConfig(checkpoint='checkpoints/colbertv2.0')
    cp = Checkpoint(cf.checkpoint, colbert_config=cf)
    encoder = CollectionEncoder(cf, cp)

    # encode_passages returns a flat list of embeddings and a list of how many correspond to each passage
    embeddings_flat, counts = encoder.encode_passages(passages)

    # split up embeddings_flat into a nested list
    start_indices = [0] + list(itertools.accumulate(counts[:-1]))
    embeddings_by_part = [embeddings_flat[start:start+count] for start, count in zip(start_indices, counts)]

    # insert into the database
    for part, embeddings in enumerate(embeddings_by_part):
        execute_concurrent_with_args(db.session,
                                     db.insert_colbert_stmt,
                                     [(title, part, i, e) for i, e in enumerate(embeddings)]) (I like to encapsulate my DB logic in a dedicated module; you can access the full source in my .) GitHub repository Retrieval Then retrieval looks like this: def retrieve_colbert(query):
    db = DB()
    cf = ColBERTConfig(checkpoint='checkpoints/colbertv2.0')
    cp = Checkpoint(cf.checkpoint, colbert_config=cf)
    encode = lambda q: cp.queryFromText([q])[0]

    query_encodings = encode(query)
    # find the most relevant documents for each query embedding. using a set
    # handles duplicates so we don't retrieve the same one more than once
    docparts = set()
    for qv in query_encodings:
        rows = db.session.execute(db.query_colbert_ann_stmt, [list(qv)])
        docparts.update((row.title, row.part) for row in rows)
    # retrieve these relevant documents and score each one
    scores = {}
    for title, part in docparts:
        rows = db.session.execute(db.query_colbert_parts_stmt, [title, part])
        embeddings_for_part = [tensor(row.bert_embedding) for row in rows]
        scores[(title, part)] = score(query_encodings, embeddings_for_part)
    # return the source chunk for the top 5
    return sorted(scores, key=scores.get, reverse=True)[:5] Here is the query being executed for the most-relevant-documents part ( ): db.query_colbert_ann_stmt SELECT title, part
FROM colbert_embeddings
ORDER BY bert_embedding ANN OF ?
LIMIT 5 Beyond the basics: RAGStack This article and the linked repository briefly introduce how ColBERT works. You can implement this today with your own data and see immediate results. As with everything in AI, best practices are changing daily, and new techniques are constantly emerging. To make keeping up with the state of the art easier, DataStax is rolling this and other enhancements into , our production-ready RAG library leveraging LangChain and LlamaIndex. Our goal is to provide developers with a consistent library for RAG applications that puts them in control of the step-up to new functionality. Instead of having to keep up with the myriad changes in techniques and libraries, you have a single stream, so you can focus on building your application. You can use RAGStack today to incorporate best practices for LangChain and LlamaIndex out of the box; advances like ColBERT will come to RAGstack in upcoming releases. RAGStack By Jonathan Ellis, DataStax Also appears . here

কিভাবে ColBERT ডেভেলপারদের RAG এর সীমা অতিক্রম করতে সাহায্য করে

Wie ColBERT Entwicklern hilft, die Grenzen von RAG zu überwinden

Cómo ColBERT ayuda a los desarrolladores a superar los límites de RAG

Comment ColBERT aide les développeurs à surmonter les limites de RAG

कैसे ColBERT डेवलपर्स को RAG की सीमाओं को पार करने में मदद करता है

ColBERT が開発者による RAG の限界の克服にどのように役立つか

ColBERT가 개발자가 RAG의 한계를 극복하도록 돕는 방법

Como ColBERT ajuda os desenvolvedores a superar os limites do RAG

Как ColBERT помогает разработчикам преодолеть ограничения RAG

ColBERT, Geliştiricilerin RAG Sınırlarını Aşmasına Nasıl Yardımcı Olur?

ColBERT giúp nhà phát triển vượt qua giới hạn của RAG như thế nào

ColBERT 如何帮助开发人员克服 RAG 的限制

When building a generative AI application that needs to call a large language model (LLM) multiple times to complete a task, a common problem is that repeated queries to the LLM can be both expensive and unpredictable. Large models such as GPT-3.5/4 are incredibly resource-intensive to train and run inference; this is reflected in the API charges as well as occasional disruptions to service. ChatGPT was originally released as a research preview and wasn’t intended to be used for production applications. However, its usefulness across a vast array of applications is indisputable, so has exploded. interest in LLMs Since the inception of ChatGPT, users have been looking for ways around the lack of privacy and the inability to control uptime or inference settings when using GPT. This has led to the popularity of free, public models such as Meta’s and later the creation of quantized and lower-parameter versions of Llama that can run on consumer hardware. These public models are capable of providing much of the same functionality as GPT for much less computing power, though at the cost of fewer parameters and fewer verbose outputs. Llama 2 If your application doesn’t necessarily depend on processing excessively large contexts or producing verbose outputs, then hosting your own inference on instances you control can be a more cost-effective option. And when it comes to real-world applications of (RAG), the cost differences may become even more significant. retrieval augmented generation I’ll demonstrate a simple method for combining vector stores, lexical search, and prompt engineering to perform accurate RAG on commodity hardware. Using this method, you can both reduce the complexity of large volumes of information and make running generative AI applications more accurate, efficient, and cost-effective at scale. By using RAG on specific stores of information, you can gain the ability to eliminate hallucinations and create effective and knowledgeable agents from any source material without needing to pay for third-party APIs. To get started, you will need either a DataStax Enterprise 7 instance or to store the vectors and text data, as well as an LLM and a sentence transformer model to generate responses and encode your data with vectors. Depending on the complexity of your data or the user prompts, you may also consider combining this with a DataStax Enterprise 6.8 database that can perform searches to match wider ranges of data, which is what I used in this example. DataStax is continually working on improvements to enable all of these operations with a single database, but for now, I use two databases. DataStax Astra DB Solr Solving for Hallucinations Regardless of which LLM you choose, they all still suffer from . For now, that limitation needs to be resolved by feeding truthful information into the context of prompts to the LLM, otherwise known as RAG. The method by which you locate your information and transform it for the prompts completely depends on your data model, but you can find more pertinent information in a more efficient way by using vector databases. hallucinations Say, for instance, that you have a collection of e-books on a subject you’d like to explore, such as how to play . Under normal circumstances, it would take years to read through the supporting literature and gain enough gameplay experience to reach an expert level. Warhammer 40,000 A targeted question such as, “What can you tell me about Morvenn Vahl of Adepta Sororitas?” would be best answered by a veteran player or any employee at a Warhammer store. While ChatGPT can answer many questions about the game, it unfortunately has no training data that covers this particular character: Compare this to a Llama 2 13B parameter LLM hosted on a consumer workstation with an Nvidia RTX A4000 graphics card. Similarly, the model can demonstrate basic knowledge of the Warhammer universe, but because of the tuning, the model doesn’t care that the character isn’t found and provides instead a best-effort hallucination: If you want to build a chatbot that can help both newcomers and veterans play Warhammer 40,000, then these outputs are unacceptable. To be an effective game guide, the chatbot needs to know the rules of the game, the rules for each unit, some bits of the lore, and some strategy and commentary. Luckily, all of that information on the 10th-edition rules is available for free from Games Workshop and fan websites, and all you need to do is make it searchable to your chatbot app. Compare this to the same 13B Llama model, where with RAG, it is asked to compare a couple of sources on Morvenn Vahl and devise a relevant answer based on the user prompt. This time, the chatbot has access to a database full of all the public information on how to play Warhammer 40,000, 10th Edition: search database and a vector What a difference! Not only does it find pertinent information on this niche character, but it also keeps its outputs in line with the context of how to play the game with the 10th-edition rules. The hardest part in all of this is performing an effective search to find the relevant pages to feed into the LLM. This is where vector databases can be particularly useful. Applying vectors In this example, we’ll use DSE 7 and DSE 6.8 running in Docker instances to satisfy the database requirements of the chatbot application, which needs to be able to compare vectors and perform lexical searches. DSE 7 and Astra DB have introduced the ability to store vectors and perform vector searches, as well as filtering by text matches. We only need to search a few dozen books for this example, so running DSE instances in Docker will be sufficient for most consumer hardware. Using vectors in your databases will help to find documents that are similar to a given query, or they can be used to compare results retrieved from another search. This can help you to overcome the limitations of lexical search and to improve the effectiveness of data models. For instance, something like e-book PDFs can benefit from being encoded with sentence transformers like , and the vectors can be used to run a similarity comparison between a query and a given source. In this case, a sentence transformer model is used to create embeddings of a page’s text in an e-book, and this can enable you to compare to the user’s prompt to figure out if a result is relevant to the query. Relevant pages should contain one or more instances of terms that are similar to the user’s query and result in better similarity scores from the model’s standpoint. miniLM That said, the vectors are best applied as a supplement to an existing lexical search model. If you search by vectors only, then you might end up unexpectedly retrieving unrelated documents and providing them as context where they don’t apply. In this example, the query “What can you tell me about Morvenn Vahl of Adepta Sororitas?” can be transformed by an LLM to a set of simple search terms: Morvenn, Vahl, Adepta, Sororitas The first step in finding relevant documents would be to search for documents that contain those basic terms. This can be done by first filtering for text matches in the database to find keywords in the page text matching such a query. The reason for using an LLM to generate keywords is to provide a wider range of possible keywords to search, as it often attempts to add more keywords that are related but are not in the text of the original prompt. Be careful with this, however, as LLMs can also generate special characters and odd sequences that you will need to sanitize. Once you have at least one result, you can vectorize the user’s query and compare it to the vectors of the lexical search, creating scores of how relevant each result is. This allows you to check the search results for accuracy to the query and set a threshold for rejecting unrelated results when it comes to finally presenting your results to the LLM. In this case, the first step should match pages that specifically show Morvenn Vahl’s index card or gameplay mechanics because those describe the character’s unit in terms of how it plays in the game. If the page meets a certain relevance threshold to the user query, determined by the application, then it gets summarized and placed in a list of results. Finally, the search results can be compiled into a list and fed back to the LLM, where it is asked to use the most relevant contexts to answer the original query. Here is a visualization of the flow: As you can see, the LLM gets called quite frequently for this flow. The LLM is responsible for transforming the user prompt into keywords, summarizing applicable results, and choosing which context best answers a query. Each source to check adds another LLM invocation, which can be quite expensive when making queries to GPT. But if you already have the information you need and just want to summarize it or transform it, then you may not need to use such a large model. In fact, switching to smaller models can provide a number of benefits. By using a smaller LLM, you can reduce the computational cost of each query, which can lead to significant savings over time. This can also result in faster response times for your users, which can improve their overall experience. In this example, where RAG is performed using a small LLM and small databases, all hosted on the same GPU instance, it takes about 30 seconds to retrieve 15 sources, analyze them for relevance, and provide a final answer. And the shorter the prompts (sources), the faster the outputs can be returned. Additionally, this method allows for increased security and scalability. With prompt engineering and a pipeline of calls to the LLM, you’re in is accessed and what the users will get in their responses. In terms of resource usage, the example 13B parameter model only consumes a little over 8GB of VRAM and still provides relevant answers. Depending on needs, this shows potential for even running RAG on myriad other platforms, such as user workstations and mobile devices. full control of how the data Controlling the output Prompt engineering is key to making RAG do exactly what you want. You are in control of how the chatbot interprets the data and the context under which it should be thinking. In this example, we want to ensure that the chatbot knows we are specifically seeking Warhammer information, so we can first ask it to help provide supporting context to the user’s query: Query: “<user query>” Give me a minimal, comma-separated list of Warhammer 40K keywords for a search engine. Respond with only the query. Do not use emojis or special characters. Answer: Warhammer 40,000 is full of terms and names that might appear in other, unrelated popular culture, so it is important to set the context of the RAG in the very first query. This context should be something available to select or modify if your application covers multiple contexts, such as if you need to cover multiple editions of the Warhammer game rules or combine them with the official lore books. Note that the user’s query is always encapsulated with quotes for this experiment. This helps the LLM to distinguish between the query that it’s attempting to directly answer and the separate prompt-engineering instructions, which it must not directly answer. The question/answer portion of the prompt can be adjusted to fit a particular context, but mostly, all you need to be able to do is inform the LLM what it should and should not directly respond to and how to respond. In this case, it’s safe to assume that the LLM has a general knowledge of the game universe since the series is reasonably popular and general information is available for free. The output of this first query helps generate some keywords to use in lexical search without us having to build a cheat sheet into our application. The lexical and vector comparisons can then be performed in the background, and a list of results is compiled for review by the LLM. Because the user’s original prompt is never directly answered with inference at the first step, the LLM only transforms what is found in a search and can easily be stopped from answering queries outside of its guard rails or knowledge base. If there are relevant results from the search: Query: “<user query>” Review these search results and use them to answer the query. Result 1 Result 2 etc. Answer: If there are no relevant results from the search: Query: “<user query>” Politely tell me you searched but could not find an answer for the query. Answer to the best of your knowledge instead. Answer: For added security, you can completely reject or redirect the request when it cannot be served. Query: “<user query>” Politely tell me you searched but could not find an answer to the query. Instruct me to reach out to the Customer Support team for assistance instead. Answer: You can even make the outputs lengthier by asking for more details. As long as you can fit your source material within the context window, the LLM can transform it for you. Query: “<user query>” Review these search results and use them to answer the query. Be as detailed as possible and cite the sources. Result 1 Result 2 etc. Answer: Limitations The LLM has a limited context window and will fail to process exceptionally large pages of text. Consider placing limits on row size so your and easier for the LLM to process. For example, cutting pages into chunks of about 1,000 characters seems to work well, and try to avoid feeding more than four or five detailed answers into the prompt. data is more manageable The LLM has no memory of a conversation aside from what you can fit in the context window. It is possible to build a permanent store of conversation data, but it is not possible for an LLM to fit excessively large conversations or detailed context into a prompt; there is an upper limit to what it can transform. This means that no matter what, at a certain point, you will notice that the LLM seems to “forget” certain details even when they are provided as context; this is just an inherent limitation of the tool. It is best to rely on it for short conversations only and focus on transforming small amounts of text at a time to minimize hallucinations. Randomness in the LLM can be a problem. Testing and tuning will be necessary to determine which prompts work best for your data set and to find which model works best for your use case. In my testing with a 13B parameter model, there was a lot of unpredictability regarding which search keywords were generated from the first prompt, especially as the prompt length increased. For best results, stick to shorter prompts. Conclusion In summary, leveraging RAG by combining models allows for more effective finding and sorting of relevant results and generating agent outputs that are far less prone to hallucinations. The smaller the searchable context, the more precise and accurate the responses. Constructing your own custom pipeline of LLM calls offers far more flexibility in tuning responses toward your desired level of accuracy and guard rails. vector and lexical search While it cannot process excessively large amounts of data within the limited context window, it does offer the ability to create effective assistants on limited knowledge bases, as well as run more concurrent agents on the same or lesser hardware than before. This could open up more possibilities for virtual assistants for applications such as tabletop gaming or even cover more complex topics for use by government, legal and accounting firms, scientific research, energy, and more. If you’re ready to get started building, you can . Create your database and start loading your RAG sources today, with no cloud or database Ops experience required. try Astra DB for free By Mario Charnell-Delgado, DataStax Also published . here

Warhammer 40K শিখতে চান? কমোডিটি হার্ডওয়্যারে ভেক্টর এবং RAG দিয়ে এই চ্যাটবট তৈরি করুন

Möchten Sie Warhammer 40K lernen? Erstellen Sie diesen Chatbot mit Vektoren und RAG auf Commodity-Hardware

¿Quieres aprender Warhammer 40K? Cree este chatbot con vectores y RAG en hardware básico

Vous voulez apprendre Warhammer 40K ? Construisez ce chatbot avec des vecteurs et RAG sur du matériel de base

क्या आप वॉरहैमर 40K सीखना चाहते हैं? कमोडिटी हार्डवेयर पर वेक्टर और आरएजी के साथ इस चैटबॉट का निर्माण करें

ウォーハンマー 40K を学びたいですか? Vectors と RAG を使用してコモディティ ハードウェア上でこのチャットボットを構築する

워해머 40K를 배우고 싶으십니까? 상용 하드웨어에서 벡터와 RAG를 사용하여 챗봇 구축

Quer aprender Warhammer 40K? Construa este chatbot com vetores e RAG em hardware de commodities

Хотите изучить Warhammer 40K? Создайте этого чат-бота с помощью Vector и RAG на стандартном оборудовании

Warhammer 40K'yı Öğrenmek İster misiniz? Ticari Donanımda Vektörler ve RAG ile Bu Chatbot'u Oluşturun

Bạn muốn học Warhammer 40K? Xây dựng Chatbot này với Vector và RAG trên phần cứng hàng hóa

想学习战锤40K吗？在商品硬件上使用向量和 RAG 构建此聊天机器人

Developing generative AI applications with (RAG) and vector databases can be challenging. You’ll often need to troubleshoot RAG implementations that rely on vector databases to ensure that relevant context is retrieved; this is critical, because that context is then included in a prompt to a large language model to provide more accurate results. retrieval augmented generation In , we looked at the embedding pipeline, which populates the with embeddings, and considered three areas that can lead to poor results: suboptimal embedding models, inefficient chunking strategies and lack of metadata filtering. the previous article in this series vector database Here, we’ll look at the actual interaction with the LLM and examine a couple common problems that can lead to poor results: imprecise prompts and insufficient generation approaches. Prompt refinement Even the best embedding and chunking strategies can’t replace the need for quality . This involves making the prompts more explicit, specific and aligned with the intended output. Varied prompt formats, lengths and lexical choices should be tested to fine-tune the RAG process. prompt engineering Specifically, there are several things you should consider when building your prompts for RAG applications. These include: When interacting with LLM agents like ChatGPT, they will default to acting like a helpful chatbot. However, you can alter the nature of responses that will be generated by instructing the LLM to act in a particular way. Examples could be things like “you are an attorney evaluating if either party is in breach of an agreement” or “you are a customer service agent for an internet service provider; your job is to help people troubleshoot their internet issues” or whatever makes sense for your particular situation. Tell the LLM its role: : Make it clear to the LLM that you are providing context, and you want the generated response to reflect that context. You can do this by saying something like “your response should take into account the following context” followed by the context. Use examples: In the just-mentioned scenario of asking the LLM to act as an attorney evaluating contracts, you may want to include several examples in the prompt. For instance, you could provide an example of a contract that says payment is due within 30 days, but the buyer didn’t send their payment until 40 days after the contract was signed and therefore violated the agreement. You could provide additional examples of rights to remedy and possible ways to resolve contract violations. Explicitly tell the LLM to use the provided context If your use case requires specific output, you can specify the format that the generated output should adhere to. You can combine this technique along with the above tip to provide examples to make it clear to the LLM both how you want it to respond, as well as the key points of information you’re expecting in the generated response. Specify the output format: : For use cases where reasoning is involved to determine the appropriate response, you can consider using a technique called to help explain the steps that you want the LLM to follow to arrive at a generated response. For example, in the legal contracts case, you may want to walk the LLM through the logical steps that a person would follow to determine if a contract term has been violated. For instance, when dealing with legal contracts you might tell the LLM to first look for clauses that specify payment terms, then identify the amount of time the buyer had to submit payment, then calculate the number of days between the time payment was received and the date the contract was signed. Then, if the payment took longer than the agreed upon time frame, the buyer violated the agreement. Use chain of thought chain of thought Leveraging these techniques to improve your prompt engineering can have a significant impact on the quality of the generated results you’re able to produce in your RAG applications. However, sometimes you will need to use techniques that involve multiple exchanges with the LLM to achieve an acceptable response. Advanced Patterns FLARE Forward-looking active retrieval, or , is an example of a multi-query RAG technique that involves iteratively calling the LLM with custom instructions in your prompt that ask the LLM to provide additional questions about key phrases that would help it generate a better answer. Once the LLM has context with no gaps, it terminates with the final response. The process adds a loop between the LLM and the AI agent (the AI chatbot in the diagram) to facilitate these iterations: FLARE You can see an example of how FLARE works in the . LangChain cookbook example of FLARE RAG Fusion By generating similar queries to the user’s query and retrieving relevant context for both the original query as well as the generated similar queries, we are able to increase the likelihood that we have selected the most useful context to generate accurate results. The process, called “RAG fusion,” looks like this: The key step here is to use a to further refine the ANN results to identify the most relevant context to use to generate a response. reciprocal rank function Conclusion RAG is an approach that’s simple to get started with, but often leaves developers frustrated with unsatisfactory results for unclear reasons. The relevance of outputs in RAG-enabled generative AI applications can be affected by several factors. By meticulously selecting embedding models, formulating chunking strategies and designing prompts, you can significantly elevate the quality and precision of generated responses in systems powered by LLMs like ChatGPT. Hopefully these tips will help you create more useful RAG applications that deliver the experience and value you’re aiming for. . , the only vector database for building production-level AI applications on real-time data Try DataStax Astra DB By Chris Latimer, DataStax Also appears . here

সাধারণ পুনরুদ্ধার অগমেন্টেড জেনারেশন চ্যালেঞ্জ মোকাবেলা করে প্রাসঙ্গিক এলএলএম প্রতিক্রিয়া অর্জন করা

Erreichen relevanter LLM-Antworten durch Bewältigung häufiger Herausforderungen im Bereich Retrieval Augmented Generation

Lograr respuestas relevantes de LLM abordando los desafíos comunes de la generación aumentada de recuperación

Obtenir des réponses LLM pertinentes en relevant les défis courants de génération augmentée de récupération

सामान्य पुनर्प्राप्ति संवर्धित पीढ़ी चुनौतियों का समाधान करके प्रासंगिक एलएलएम प्रतिक्रियाएं प्राप्त करना

一般的な検索拡張生成の課題に対処することで、関連する LLM 応答を実現する

일반적인 검색 증강 생성 문제를 해결하여 관련 LLM 응답 달성

Alcançando respostas LLM relevantes abordando desafios comuns de geração aumentada de recuperação

Достижение соответствующих ответов LLM путем решения общих проблем поиска дополненной генерации

Ortak Erişim Artırılmış Üretim Zorluklarını Ele Alarak İlgili Yüksek Lisans Yanıtlarına Ulaşmak

Đạt được các phản hồi LLM có liên quan bằng cách giải quyết các thách thức chung về thế hệ tăng cường truy xuất

通过解决常见的检索增强生成挑战来获得相关的法学硕士学位

In the first installment of this two-part series, we look at how suboptimal embedding models, inefficient chunking strategies, and a lack of metadata filtering can make it hard to get relevant responses from your LLM. Here’s how to surmount these challenges. Building generative AI applications that use (RAG) can pose a host of challenges. Let’s look at troubleshooting RAG implementations that rely on vector databases to retrieve relevant context that’s then included in a prompt to a large language model to provide more relevant results. retrieval augmented generation We will break this process down into two main parts. The first, which we’ll address in this first article in the series, is the embedding pipeline, which populates the with embeddings: vector database Here, we’ll consider three main areas that can lead to poor results: suboptimal embedding models, inefficient chunking strategies, and a lack of metadata filtering. (In the upcoming article, we’ll look at the actual interaction with the LLM and examine some common problems that crop up there and can lead to poor results.) Selecting an Appropriate Embedding Model Your choice of an embedding model will have a significant impact on the overall relevance and usability of your RAG application. As such, it requires a nuanced understanding of each model’s capabilities and an analysis of how those capabilities align with your application’s requirements. If you’re relatively new to RAG and embeddings in general, one of the best resources you should be aware of is the . We focus on retrieval use cases in this post, but embeddings can, of course, be used for many other applications, including classification, clustering, and summarization. MTEB (Massive Text Embedding Benchmark) embedding leaderboard The leaderboard can help you identify the models that will perform best for your specific use case. One of the most common reasons for poor RAG performance is that developers new to this space do a Google search to find examples of embedding generation. They often find samples that use embedding models such as Word2Vec, sBERT, and RoBERTa that are poor choices for retrieval use cases. If you found this article because you’re debugging poor relevance results and you used something like sBERT to generate your embeddings, then we’ve likely identified the cause of your relevance problems. If so, the next question you will likely have is which embedding models you can use to improve your similarity search results. Without knowing the particulars of your use case, the three we would recommend are: text-embedding-ada-002 (Ada v2) is probably the most common starting point for most RAG applications simply because so many developers start off with Open AI’s APIs. Ada v2 performs admirably in retrieval use cases and was built to handle different types of content, including text and code. Ada v2 from OpenAI With a maximum input sequence length of up to 8,192 tokens, it also allows you to create embeddings for much longer pieces of text than alternative models. This is both a blessing and a curse. Having a large sequence size simplifies the process of creating embeddings for more of your text content, and it allows the embedding model to identify relationships across words and sentences in a bigger body of text. However, this also results in similarity searches that can become more fuzzy when comparing the similarity of two long documents when what you’re looking for is relevant chunks of context to facilitate the generation process. There are two big drawbacks of Ada v2. The first is that it can’t be run locally. You must use OpenAI’s API to create the embedding. This can not only introduce bottlenecks for cases where you want to create embeddings for many pieces of content, but it also adds a cost of $0.0001 per 1,000  tokens. The second is that the embeddings created from the Open AI model are 1,536 dimensions each. If you are using a cloud vector database, this can considerably add to your vector storage costs. You want a simple solution that only requires an API call, you potentially need to vectorize large documents, and cost is not an issue. When to choose: jina-embeddings-v2 (Jina v2) Jina v2 is a new open-source embedding model that gives you the same 8,000 input sequence support as Ada v2 and actually scores slightly better in retrieval use cases. Jina v2 provides an antidote to the problems of Ada v2. It’s under Apache License 2.0 and can be run locally, which, of course, is also a drawback if you aren’t looking to run your own code to do this. It also produces an embedding vector with half the dimensions of Ada v2. open source So not only do you get slightly better retrieval performance on benchmark use cases, but you also get those improved results with lower storage and compute requirements from a vector database perspective. You want to use an open-source solution and potentially need to vectorize large documents and are comfortable running embedding pipelines locally. You want to reduce vector database costs with lower-dimension embeddings. When to choose: bge-large-en-v1.5 is open-sourced under the MIT license and is currently the top-ranked embedding model on the MTEB leaderboard for retrieval use cases. With a smaller input sequence, it will require you to give more thought to your chunking strategy but ultimately provides the best all-around performance for retrieval use cases. bge-large-en-v1.5 When to choose: You want to use an open-source solution and are willing to spend more time on chunking strategies to stay within the input size limitations. You are comfortable running embedding pipelines locally. You want the best-performing embedding model for retrieval use cases. While outside the scope of this article, you might want to dig deeper into the 15 benchmarks in the MTEB leaderboard to identify the one that most closely resembles your specific situation. While there are definitely patterns in terms of how well various embedding models perform across the different benchmarks, there are often specific models that stand out in each. If you need to further refine your embedding selection, this is a possible area of further investigation. Optimizing Your Chunking Strategy The segmentation or, “chunking” of input text, is a pivotal factor that significantly influences the relevance and accuracy of the generated output. Various chunking strategies offer unique advantages and are suited for specific types of tasks. Here, we delve into these methodologies and provide guidelines for their application, incorporating some key considerations: Fixed-length chunking - Unless your content itself is highly structured and of fixed length, you usually want to rely on a more useful chunking strategy like the ones that follow. When to use - While very simple to implement, this chunking strategy is generally going to lead to poor results in RAG applications. Technical consideration If you are using a fixed-length strategy with your RAG application and having trouble retrieving relevant context, you should consider switching to a different chunking approach. Additional insight Sentence-level chunking - This strategy is effective when each sentence in the input text is rich in meaning and context. It allows the model to concentrate on the intricacies within each sentence, thereby generating more coherent and contextually relevant responses. You’ll rarely rely on sentence-level chunking for RAG use cases. When to use Sentence-level chunking often involves tokenization based on sentence boundaries, which can be achieved using natural language processing (NLP) libraries. Technical consideration - Sentence-level chunking can be particularly useful when you’re searching for specific statements, such as in a transcript of a meeting where you’re trying to find semantically similar statements to a given piece of text. Additional insight - Paragraph-level chunking - Employ this strategy when the input text is organized into distinct sections or paragraphs, each encapsulating a separate idea or topic. This enables the model to focus on the relevant information within each paragraph. When to use Identifying paragraph boundaries usually involves detecting newline characters or other delimiters that signify the end of a paragraph. Technical consideration - Paragraph-level chunking can be useful when you have documents that cover many different aspects of the same topic. For example, a page of product documentation might introduce a product feature, explain when to use it, talk about how to configure it, and give examples of different configurations. Additional insight - Using paragraph-level chunking can help you identify the most relevant part of the document to provide to the LLM as context. Content-aware chunking Opt for this strategy when the relevance of specific sections within the text is paramount. For instance, in legal documents, segmenting the text based on clauses or sections can yield more context-specific responses. When to use - This approach may require advanced NLP techniques to understand the semantic boundaries within the text. Technical consideration - Content-aware chunking is especially useful when dealing with structured or semi-structured data, as specific chunks can be combined with metadata filtering for more precise retrieval. Additional insight - For example, in a legal document, you might want to extract all warranty or indemnification clauses, and when you store embeddings for chunks in a vector database, you can use metadata to make it easier to search for content of a given type when building a RAG use case. Recursive chunking Recursive chunking divides data into smaller and smaller pieces, using a hierarchical approach. For example, when chunking a text document, you might divide the text into paragraphs first, then into sentences, and finally, words. When to use - Once the data has been divided into the first set of chunks, you can then recursively apply the chunking process to each of the smaller chunks, repeating until you reach the smallest chunk size you’re interested in. Implementing recursive chunking might involve a multilevel parsing strategy where chunks are further divided into sub-chunks based on additional criteria. If you’re using , its recursive implementation is a bit simpler than what is described here. Technical consideration - LangChain This approach enables the model to understand the context at multiple levels, from high-level themes to detailed nuances, making it particularly useful for complex documents like academic papers, technical manuals, or legal contracts. This brings flexibility benefits since similarity searches can identify similar text for both broader and shorter queries. Additional insight - However, this also means that there is a possibility that similar chunks from the same source document could end up being overrepresented in similarity searches as well, especially if you opt for a longer overlap between chunks in your text splitter configuration. As a general approach, before you try chunking up a big corpus and vectorizing it, you should consider doing some ad-hoc experimentation with your data. Manually inspect the documents you would want to retrieve for a given query, identify the chunks that represent the ideal context you would want to provide the LLM, and then experiment with chunking strategies to see which one gives you the chunks you feel would be most relevant for the LLM to have. Context Window Consideration The available context window of an LLM is an important factor in selecting a chunking strategy. If the context window is small, you’ll need to be more selective in the chunks you feed into the model to ensure that the most relevant information is included. Conversely, a larger context window allows for more flexibility, enabling the inclusion of additional context that may enhance the model’s output, even if not all of it is strictly necessary. By experimenting with these chunking strategies and taking these considerations into account, you can evaluate their impact on the relevance of the generated outputs. The key is to align the chosen strategy with the specific requirements of your RAG application, preserve the semantic integrity of the input, and offer a comprehensive understanding of the context. This will enable you to find the right chunking process for optimal performance. Metadata Filtering As the number of embeddings in your search index grows, approximate nearest neighbors (ANN) become less helpful when looking for relevant context to include in your prompts. Let’s say you have indexed embeddings for 200 articles in your knowledge base. If you can identify the top nearest neighbor with an accuracy of 1%, you are likely to find pretty relevant results because 1% represents the top two articles out of those 200, and you’re going to get one of those two. Now, consider a search index containing every article on Wikipedia. That would amount to approximately 6.7 million articles. If your nearest neighbor is in the top 1% of most similar articles, that means that you’re getting one of the 67,000 most similar articles. With a corpus like Wikipedia, this means that you could still end up being very far off the mark. Metadata filtering gives you a way to narrow down the pieces of content by first filtering the documents and then applying the nearest neighbor algorithm. In cases where you're dealing with a large number of possible matches, this initial pre-filtering can help you narrow the possible options before retrieving the nearest neighbors. Coming up next, we’ll dive into the interaction with the LLM and examine some common problems that can lead to poor results. . Try , the only vector database for building production-level AI applications on real-time data DataStax Astra DB By Chris Latimer, DataStax Also published here

Need More Relevant LLM Responses? Address These Retrieval Augmented Generation Challenges

আরো প্রাসঙ্গিক LLM প্রতিক্রিয়া প্রয়োজন? এই পুনরুদ্ধার অগমেন্টেড জেনারেশন চ্যালেঞ্জের সমাধান করুন (পর্ব 1)

Benötigen Sie relevantere LLM-Antworten? Bewältigen Sie diese Herausforderungen im Bereich Retrieval Augmented Generation (Teil 1)

¿Necesita respuestas de LLM más relevantes? Aborde estos desafíos de la generación aumentada de recuperación (Parte 1)

Besoin de réponses LLM plus pertinentes ? Relever ces défis de génération augmentée de récupération (partie 1)

अधिक प्रासंगिक एलएलएम प्रतिक्रियाओं की आवश्यकता है? इन पुनर्प्राप्ति संवर्धित पीढ़ी चुनौतियों का समाधान करें (भाग 1)

さらに関連性のある LLM 応答が必要ですか?これらの検索拡張生成の課題に対処する (パート 1)

더 많은 관련 LLM 응답이 필요하십니까? 이러한 검색 증강 세대 문제 해결(1부)

Precisa de respostas LLM mais relevantes? Aborde esses desafios de geração aumentada de recuperação (Parte 1)

Нужны более релевантные ответы LLM? Решите эти проблемы поиска дополненной генерации (часть 1)

Daha İlgili LLM Yanıtlarına mı ihtiyacınız var? Bu Erişim Artırılmış Nesil Zorluklarının Ele Alınması (Bölüm 1)

Cần thêm phản hồi LLM có liên quan? Giải quyết những thách thức về thế hệ tăng cường truy xuất này (Phần 1)

需要更多相关的法学硕士回复？应对这些检索增强生成挑战（第 1 部分）

With the rapid integration of generative AI into the application development process we are seeing an increasing need to be able to integrate our private data with public data that is being used to train (LLMs). The challenge this presents is that most private data is unstructured, siloed, and not in a format that can be readily accessible by LLMs. large language models In a recent webinar on , we explored how LLMs can be used for applications beyond ChatGPT and how private data needs to be used to augment the public data that generally available LLMs are trained on. This is where solutions like LlamaIndex come into play as they provide an orchestration framework for building LLM apps using built-in tools to ingest and query private data. Large Language Models for the Enterprise In this post, we’ll explain how LlamaIndex can be used as a framework for data integration, data organization, and data retrieval for all your private data generative AI needs. What is LlamaIndex? As stated earlier, LlamaIndex is an orchestration framework or “data framework” that simplifies building LLM applications. It provides the ability to perform data augmentation of private data, enabling it to be incorporated into LLMs for knowledge generation and reasoning. At the heart of all generative AI functionality is data. Enterprise applications need to be able to access more than just the public data that LLMs are trained on and need to incorporate structured, unstructured, and semi-structured data from all their internal and external data sources for building applications. It is this integration of data the LlamaIndex provides. Bringing in data from multiple unique sources. that data as vectors. Storing that newly vectorized data in a vector database. Ultimately, allowing for that data to be used by applications to perform complex operations with low latency response times like vector search. Embedding Benefits Simplified data ingestion connecting existing data sources like APIs, PDFs, SQL, NoSQL, documents, etc. for use with LLM applications. Natively store and index private data for use across different application use cases, with native integration with downstream vector store/vector databases. Built-in query interface, providing the ability to return knowledge-augmented responses from input prompts on your data. Use Cases Building natural language chatbots that provide real-time interaction with your product documentation for natural customer engagement. Building cognitively aware knowledge agents that can respond to changing decision trees based on a constantly growing knowledge basis. Interact with large volumes of structured data using natural language and human interaction. Augment public data with private knowledge corpus providing application-specific engagement. How does LlamaIndex work? LlamaIndex, formerly known as GPT Index, is a framework that provides the tools needed to manage the end-to-end lifecycle for building LLM-based applications. The challenge with building LLM-based applications is that they need data, typically from multiple different sources, and unless there is strong adherence to a common data representation the data required is in many different formats, some highly structured, some unstructured, and some in between. That is where LlamaIndex provides the toolbox to unlock this data with tools for data ingestion and data indexing. Once ingested and indexed, (RAG) applications can use the LlamaIndex query interface for accessing that data and powering LLMs. retrieval augmented generation Ingestion LlamaIndex has hundreds of data loaders that provide the ability to connect custom data sources to LLMs. It connects pre-built solutions like Airtable, Jira, Salesforce, and more to generic plugins for loading data from files, JSON documents, simple csv, and unstructured data. A complete list of data loaders can be found on the . Llama Hub Indexing Once data is ingested, it needs to be mathematically represented so that it can be easily queried by an LLM. With LlamaIndex, an index simply provides the ability to represent data mathematically in multiple different dimensions. Indexing data isn’t a new concept. However, with machine learning, we can expand the granularity of indexing from one or two dimensions (key/value representation for example) to hundreds or thousands of dimensions. The most common approach to indexing data for machine learning and LLMs is called a vector index; once data has been indexed the mathematical representation of the data is called a vector embedding. There are many types of indexing and embedding models but once data has been embedded the mathematical representation of the data can be used to provide semantic search as things like text with similar meanings will have a similar mathematical representation. For example, king and queen might be highly related if the query is royalty but not highly related if the query is gender. Querying This is where some of the real power of LlamaIndex and LLMs comes into play. Because querying data using LlamaIndex isn’t a complex series of commands to merge/join and find the data, it is represented as natural language via a concept called . The simplest way to view interaction with your data once you have ingested and indexed it is that querying becomes a process of asking questions and getting responses. prompt engineering What are the different indexes in LlamaIndex? LlamaIndex offers several different indexing models that are designed to provide optimizations around how you want to explore and categorize your data. This is ultimately where a lot of gains can be achieved, if you know the type of operation your application needs to perform on the data, leveraging a specific type of index can provide significant benefit to the application using the LLM and instantiating the query. List index A list index is an approach that breaks down the data and represents the data in the form of a sequential list. The advantage this has is that while the data can be explored in a multidimensional manner the primary optimization to querying the data is via a sequential pattern. This type of index works well with structured objects that occur over time so things like change logs where you want to query how things have changed over time. Tree index When using a tree index, LlamaIndex takes the input data and organizes it into a binary tree structure where data is organized as parent and leaf nodes. A tree index provides the ability to traverse large amounts of data and construct responses where you need to extract specific segments of the texts based on how the search traverses the tree. Tree indexing works best for cases where you have a pattern of information that you want to follow or validate like building a natural language processing chatbot on top of a support/FAQ engine. Vector store index When using the vector store index type, LlamaIndex stores data notes as vector embeddings. This is probably the most common indexing type as it provides the ability to use the representation of the data in multiple different ways including vector or similarity search. When data is indexed with a vector store index, it can be leveraged locally for smaller datasets and by a single application or for larger datasets and/or to be used across multiple different LLMs/applications it can be stored in a high-performance vector database like . Astra DB Keyword index Keyword indexing is more of the traditional approach of mapping a metadata tag, i.e. a keyword to specific nodes that contain those keywords. This mapping builds a web of relationships based on keywords, because a keyword may map to multiple different nodes and a node may be mapped to multiple different keywords. This indexing model works well if you are looking to tag large volumes of data and query it based on specific keywords that can be queried across multiple different datasets. For example legal briefings, medical records, or any other data that needs to be aligned based on specific types of metadata. LlamaIndex versus LangChain: Key comparisons One of the big questions that come up is how do LlamaIndex and LangChain compare, do they provide similar functionality or do they complement each other? The reality is that LlamaIndex and LangChain provide two sides of the same coin. While they are both designed to provide an interface to LLMs and machine learning in your application, LlamaIndex is designed and built specifically to provide indexing and querying capabilities for intelligent searching of data. On the other side of that coin is the ability to interact with data either via natural language processing, i.e. building a chatbot to interact with your data, or using that data to drive other functions like calling code. LlamaIndex provides the ability to store the data you have in a variety of different formats and pull that data from a bunch of different sources, ultimately providing the how for your generative AI application. LangChain provides the ability to do something with that data once it has been stored, generate code, provide generative question answers, and drive decisions, ultimately providing the what for your generative AI application. What projects can you build with LlamaIndex? With LlamaIndex you have an easy-to-use data/orchestration framework for ingesting, indexing, and querying your data for building generative AI applications. While we provide a simple example above to get started, the real power of LlamaIndex comes from the ability to build data-driven AI applications. You don’t need to retrain models, you can use LlamaIndex, and a highly scalable vector database to create custom query engines, conversational chatbots, or powerful agents that can interact with complex problem-solving by dynamically interpreting the data coming in and make contextual decisions in real-time. Build real-time, generative AI apps with vector search on Astra DB So when it comes time to build a generative AI application that requires the ability to leverage your private data and incorporate that into an application’s ability to interact and respond to that data, LlamaIndex is a great place to start for ingestion, indexing, and querying. But don’t repeat the mistakes of the past and silo the data you are using, embedding, and accessing for AI applications. Build out a complete end-to-end solution that includes storing those embeddings and indexes in a highly scalable vector store like Astra DB. To get started with LlamaIndex and to see how DataStax and LlamaIndex are better together, check out the recent DataStax blog post, “ ” Building Petabyte-Scale GenAI Apps Just Got Easier. You can find more information on how to set up and deploy Astra DB on one of the world’s highest-performing vector stores built on Apache Cassandra which was designed for handling massive volumes of data at scale. To get started for free, . register here - By Bill McLane, DataStax Also published here.

What Is LlamaIndex? A Comprehensive Exploration of LLM Orchestration Frameworks

LlamaIndex কি? এলএলএম অর্কেস্ট্রেশন ফ্রেমওয়ার্কের একটি ব্যাপক অনুসন্ধান

Was ist LlamaIndex? Eine umfassende Untersuchung der LLM-Orchestrierungs-Frameworks

¿Qué es LlamaIndex? Una exploración exhaustiva de los marcos de orquestación de LLM

Qu’est-ce que LlamaIndex ? Une exploration complète des cadres d'orchestration LLM

लामाइंडेक्स क्या है? एलएलएम ऑर्केस्ट्रेशन फ्रेमवर्क का व्यापक अन्वेषण

ラマインデックスとは何ですか? LLM オーケストレーション フレームワークの包括的な調査

LlamaIndex란 무엇입니까? LLM 오케스트레이션 프레임워크에 대한 포괄적인 탐색

O que é LlamaIndex? Uma exploração abrangente de estruturas de orquestração LLM

Что такое LlamaIndex? Комплексное исследование фреймворков оркестрации LLM

LlamaIndex Nedir? LLM Düzenleme Çerçevelerinin Kapsamlı Bir İncelemesi

Llama Index là gì? Khám phá toàn diện về Khung điều phối LLM

什么是 LlamaIndex？ LLM编排框架的全面探索

Large language models have become extremely powerful today; they can help provide answers to some of our hardest questions. But they can also lead us astray: they tend to hallucinate, which means that they give answers that seem right, but aren’t. hallucinate when they encounter queries that aren’t part of their training data set – or when their training data set contains erroneous information (this can happen when LLMs are trained on internet data, which, as we all know, can’t always be trusted). LLMs also don’t have memory. Finally, “fine-tuning” is often regarded as a way to reduce hallucinations by retraining a model on new data—but it has its drawbacks. LLMs Here, we’ll look at three methods to stop LLMs from hallucinating: Retrieval-augmented generation (RAG), reasoning, and iterative querying. Retrieval-augmented generation With , a query comes into the knowledge base (which, in this case, is a ) as a semantic vector – a string of numbers. The model then retrieves similar documents from the database using , looking for documents whose vectors are close to the vector of the query. RAG vector database vector search Once the relevant documents have been retrieved, the query, along with these documents, is used by the LLM to summarize a response for the user. This way, the model doesn’t have to rely solely on its internal knowledge but can access whatever data you provide it at the right time. In a sense, it provides the LLM with “long-term memory” that it doesn’t possess on its own. The model can provide more accurate and contextually appropriate responses by including proprietary data stored in the vector database. An alternate RAG approach incorporates fact checking. The LLM is prompted for an answer, which is then fact checked and reviewed against data in the vector database. An answer to the query is produced from the vector database, and then the LLM in turn uses that answer as a prompt to discern whether it's related to a fact. Reasoning LLMs are very good at a lot of things. They can predict the next word in a sentence, thanks to advances in “transformers,” which transform how machines understand human language by paying varying degrees of attention to different parts of the input data. LLMs are also good at boiling down a lot of information into a very concise answer, and finding and extracting something you’re looking for from a large amount of text. Surprisingly, LLMS can also plan – they can literally gather data and plan a trip for you. And maybe even more surprisingly, LLMs can use reasoning to produce an answer, in an almost human-like fashion. Because people can reason, they don’t need tons of data to make a prediction or decision. Reasoning also helps LLMs to avoid hallucinations. An example of this is “ .” chain-of-thought prompting This method helps models to break multi-step problems into intermediate steps. With chain-of-thought prompting, LLMs can solve complex reasoning problems that standard prompt methods can’t (for an in-depth look, check out the blog post from Google). Language Models Perform Reasoning via Chain of Thought If you give an LLM a complicated math problem, it might get it wrong. But if you provide the LLM with the problem as well as the method of solving it, it can produce an accurate answer – and share the reason behind the answer. A vector database is a key part of this method, as it provides examples of questions similar to this and populates the prompt with the example. Even better, once you have the question and answer, you can store it back in the vector database to further improve the accuracy and usefulness of your generative AI applications. There are a host of other reasoning advancements you can learn about, including , , , and . tree of thought least to most self consistency instruction tuning Iterative querying The third method to help reduce LLM hallucinations is interactive querying. In this case, an AI agent mediates calls that move back and forth between an LLM and a vector database. This can happen multiple times iteratively, in order to arrive at the best answer. An example of this forward-looking active retrieval generation, also known as FLARE. You take a question, query your knowledge base for more, similar questions. You’d get a series of similar questions. Then you query the vector database with all the questions, summarize the answer, and check if the answer looks good and reasonable. If it doesn’t, repeat the steps until it does. Other advanced interactive querying methods include , , and . AutoGPT Microsoft Jarvis Solo Performance Prompting There are many tools that can help you with agent orchestration. is a great example that helps you orchestrate calls between an LLM and a vector database. It essentially automates the majority of management tasks and interactions with LLMs, and provides support for memory, vector-based similarity search, advanced prompt templating abstraction, and a wealth of other features. It also helps and supports advanced prompting techniques like chain-of-thought and FLARE. LangChain Another such tool is , which was developed by DataStax as an abstraction on top of our Astra DB vector database, with the idea of making data and memory first class-citizens in . CassIO is a Python library that makes the integration of Cassandra with generative artificial intelligence and other machine learning workloads seamless by abstracting the process of accessing the database, including its capabilities, and offering a set of ready-to-use tools that minimize the need for additional code. CassIO generative AI vector search Putting it all together: SkyPoint AI is a SaaS provider specializing in data, analytics, and AI services for the senior care and living industry. The company leverages generative AI to enable natural and intuitive interactions between seniors, caregivers, and software systems. By simplifying complex applications and streamlining the user experience, to access information and insights effortlessly, which helps enhance care. SkyPoint AI SkyPoint AI empowers seniors and caregivers The company pulls from a wide variety of data that is both structured and unstructured to provide AI-generated answers to prompts like “How many residents are currently on Medicare?” SkyPoint CEO Tisson Mathew told me recently. This helps care providers make informed decisions quickly, based on accurate data, he said. Getting to that point, however, was a process, Mathew said. His team started by taking a standard LLM and fine-tuning it with SkyPoint data. “It came up with disastrous results – random words, even,” he said. Understanding and creating prompts was something SkyPoint could handle, but it needed an AI technology stack to handle generating accurate answers at scale. SkyPoint ended up building a system that ingested structured data from operators and providers, including electronic healthcare record and payroll data, for example. This is stored in a columnar database; RAG is used to query it. Unstructured data, such as policies and procedures and state regulations, is stored in a vector database: . DataStax Astra DB Tisson posed a question as an example: What if a resident becomes abusive? Astra DB provides an answer that is assembled based on state regulations and the users context and a variety of different documents and in natural language that’s easy for a senior care facility worker to understand, vector embeddings , “These are specific answers that have to be right,” Tisson said. “This is information an organization relies on to make informed decisions for their community and for their business.” Conclusion SkyPoint AI illustrates the importance of mitigating the risk of AI hallucinations; the consequences could be potentially dire without the methods and tools available to ensure accurate answers. With RAG, reasoning, and iterative querying approaches such as FLARE, generative AI – particularly when fueled by proprietary data – is becoming an increasingly powerful tool to help enterprises serve their customers efficiently and effectively. By Alan Ho, DataStax Learn more about how DataStax helps you . build real-time, generative AI applications Also published . here

Can AI Hallucinations Be Stopped? A Look at 3 Ways to Do So

এআই হ্যালুসিনেশন কি বন্ধ করা যায়? এটি করার 3 টি উপায়ের দিকে নজর দিন

Können KI-Halluzinationen gestoppt werden? Ein Blick auf 3 Möglichkeiten, dies zu tun

¿Se pueden detener las alucinaciones de la IA? Una mirada a tres formas de hacerlo

Peut-on arrêter les hallucinations de l’IA ? Un aperçu de 3 façons de le faire

क्या AI मतिभ्रम को रोका जा सकता है? ऐसा करने के 3 तरीकों पर एक नज़र

AIの幻覚は止められるのか？そうするための 3 つの方法を見てみましょう

AI 환각을 멈출 수 있을까? 이를 수행하는 3가지 방법 살펴보기

As alucinações de IA podem ser interrompidas? Uma olhada em três maneiras de fazer isso

Можно ли остановить галлюцинации ИИ? Взгляните на 3 способа сделать это

Yapay Zeka Halüsinasyonları Durdurulabilir mi? Bunu Yapmanın 3 Yoluna Bir Bakış

Ảo giác AI có thể bị dừng lại? Xem xét 3 cách để làm như vậy

AI幻觉可以停止吗？看看 3 种方法

is a critical component of generative AI tooling because of how retrieval augmented generation (RAG) like . At the same time, vector search is a feature, not a product – you need to query vectors as they relate to the rest of your data, not in isolation, and you shouldn’t need to build a pipeline to sync the rest of your data with a vector store to do that. Vector search FLARE helps LLMs incorporate up-to-date, customized information while avoiding hallucinations 2023 has seen an explosion in vector search products and projects, making selecting among them a serious effort.  As you research the options, you’ll need to consider the following hard problems and the different approaches to solving them. Here, I’ll walk you through these challenges and describe how DataStax tackled them for our implementation of vector search for DataStax Astra DB and Apache Cassandra. Content Overview The curse of dimensionality Problem 1: Scale-out Problem 2: Efficient garbage collection Sidebar: Cloud application workloads Problem 3: Concurrency Problem 4: Effective use of disk Problem 5: Composability Wrap up The curse of dimensionality At the core of these hard problems lies what researchers call “ .”  What this means in practice is that algorithms that work for exact vector search in 2-D or 3-D spaces, like , fall apart when you get to 10s or 100s or 1000s of dimensions in your vectors. The result is that there is no shortcut for exact similarity search with high-dimensional vectors; to get logarithmic-time results, we need to use approximate nearest neighbor (ANN) algorithms, which bring with them challenges in the following areas. the curse of dimensionality k-d trees Problem 1: Scale-out Many vector search algorithms were designed for datasets that fit in memory on a single machine, and this is still the only scenario tested by . (Ann-benchmarks further restricts its testing to a single core!) This might be okay for academic work of a million documents or rows, but it’s been many years since that could be considered representative of real-world workloads. ann-benchmarks As with any other domain, scale-out requires both replication and partitioning, as well as subsystems to handle replacing dead replicas, repairing them after a network partition, and so forth. This was an easy one for us: scale-out replication is Cassandra’s bread and butter, and combining that with the new-in-Cassandra-5.0 SAI (Storage-Attached Indexing – see for how it works, and for how to use it) gave our vector search implementation powerful scale-out capabilities effectively for free. CEP-7 the SAI documentation Problem 2: Efficient garbage collection By “garbage collection,” I mean removing obsolete information from the index – cleaning out deleted rows, and dealing with rows whose indexed vector value has changed. This might not seem worth mentioning – it’s been a more or less solved problem for forty years in the relational database world – but vector indexes are unique. The key problem is that There are lots of other vector indexing algorithms available – implements many of them – but all of them are either too slow to build, too slow to search, or offer recall that’s too low (and sometimes all three!) to be a general-purpose solution. That’s why every production vector database that I know of uses a graph-based index, the simplest of which is HNSW.  Hierarchical Navigable Small World graphs were ; the paper is quite readable and I highly recommend it. More on HNSW below. all the algorithms we know of that provide both low-latency searches and high-recall results are graph-based. FAISS introduced by Yury Malkov et al in 2016 The challenge with graph indexes is that you can’t just rip the old (vector-associated) node out when a row or document changes; if you do that more than a handful of times, your graph will no longer be able to perform its purpose of directing BFS (breadth-first search) toward areas of greater similarity to a query vector. So you’ll need to rebuild your indexes at some point to perform this garbage collection, but when—and how—do you organize it? If you always rebuild everything when changes are made, you will massively increase the physical writes performed; this is called write amplification. On the other hand, if you never rebuild then you'll do extra work filtering out obsolete rows at query time (“read amplification”). This is another problem space that Cassandra has been working on for years. Since SAI indexes are attached to the main storage lifecycle, they also participate in Cassandra , which logarithmically increases the storage units to provide a healthy balance between reads and writes. compaction Sidebar: Cloud application workloads DataStax Astra DB builds on Apache Cassandra to provide a platform for cloud application workloads. That means workloads that are: Massively concurrent thousands to millions of requests per second, usually to retrieve just a few rows apiece.  This is why you couldn’t run Netflix on Snowflake, even if you could afford it: Snowflake and similar analytical systems are designed to handle only a few concurrent requests that each run for many seconds to minutes or even longer. Larger than memory  If your dataset fits in memory on a single machine, it almost doesn’t matter what tools you use. Sqlite, MongoDB, MySQL – they’ll all work fine.  Things get more challenging when that’s not the case – and the bad news is that vector embeddings are usually several KB, or around an order of magnitude larger than typical database documents, so you’ll get to larger-than-memory relatively quickly. Core to the application  If you don’t care if you lose your data, either because it’s not that important or because you can rebuild it from the actual source of record, then again it kind of doesn’t matter what tools you use.  Databases like Cassandra, and Astra DB, are built to keep your data available and durable no matter what. Problem 3: Concurrency I mentioned above that the well-known comparison limits all algorithms to a single core. While this levels the playing field, it also handicaps those who can take advantage of the major source of hardware improvement over the last two decades. ann-benchmarks A related problem is that ann-benchmarks only performs one type of operation at a time: first, it builds the index, then it queries it. Dealing with updates interleaved with searches is optional – and likely even a handicap; if you know that you don’t need to deal with updates, you can make a lot of simplifying assumptions that look good on artificial benchmarks. If you care about being able to do multiple concurrent operations with your index or update it after it’s built, then you need to look a little deeper to understand how it works and what tradeoffs are involved. First, all general-purpose vector databases that I know of use graph-based indexes. That’s because you can start querying a graph index as soon as the first vector is inserted. Most other options require you to either build the entire index before querying it or at least do a preliminary scan of the data to learn some statistical properties. However, there are still important implementation details even within the graph index category. For example, we thought at first that we could save time by using Lucene’s HNSW index implementation, the way MongoDB Elastic and Solr do. But we quickly learned that Lucene only offers single-threaded, non-concurrent index construction. That is, you can neither query it as it is being built (which should be one of the primary reasons to use this data structure!) nor allow multiple threads to build it concurrently. The HNSW paper suggests that a fine-grained locking approach can work, but we went one better and built a non-blocking index. This is open-sourced in . JVector JVector scales concurrent updates linearly to at least 32 threads. This graph is log-scaled on both the x and y axes, showing that doubling the thread count halves the build time. More importantly, JVector’s non-blocking concurrency benefits more realistic workloads by mixing searches with updates. Here is a comparison of Astra DB’s performance (using JVector) compared to Pinecone across different datasets. While Astra DB is about 10% faster than Pinecone for a static dataset, it is 8x to 15x faster while also indexing new data. We picked the best available pod tier with Pinecone (Pod Type: p2 and Pod Size: x8, with 2 pods per replica) based on their recommendations for higher throughput and lower latency. Pinecone does not disclose what physical resources this corresponds to. On the Astra DB side, we picked the default PAYG deployment model and did not have to worry about choosing the resources as it is serverless. Tests were performed using . NoSQLBench Astra DB does this while maintaining higher recall and precision as well ( : ) F1 is a combination of recall and precision Problem 4: Effective use of disk We started with the because it’s fast to build the index, fast to query, highly accurate, and straightforward to implement. However, it has a well-known downside: it requires a lot of memory. HNSW graph indexing algorithm An HNSW index is a series of layers, where each layer above the base layer is roughly 10% as many nodes as the previous. This enables the upper layers to act as a skip list, allowing the search to zero in on the right neighborhood of the bottom layer that contains all of the vectors. However, this design means that (in common with all graph indexes) you can’t get away with saying “The disk cache will save us,” because, unlike normal database query workloads, every vector in the graph has an almost equal chance of being relevant to a search. (The exception is the upper layers, which we can and do cache.) Here’s what a profile of serving a 100M vector dataset of Wikipedia article chunks (about 120GB on disk) looked like on my desktop with 64GB of RAM, back when we were using Lucene: Cassandra is spending almost all of its time waiting to read vectors off of the disk. To solve this problem, we implemented a more advanced algorithm called DiskANN and open-sourced it as a standalone embedded vector search engine, . (Specifically, JVector implements the incremental version of DiskANN described in the paper.) Briefly, DiskANN uses a single-level graph with longer edges than HNSW and an optimized layout of vectors and neighbors to reduce disk IOPS and keeps a compressed representation of vectors in memory to speed up similarity computations. This results in tripling the throughput of the Wikipedia workload. JVector FreshDiskANN Here is what HNSW versus DiskANN looks like in a purely embedded scenario with no client/server components. This measures the speed of searching the Deep100M dataset under Lucene (HNSW) and JVector (DiskANN).  The Lucene index is 55GB, including the index and the raw vectors.  The JVector index is 64GB.  The search was performed on my 24GB MacBook, which has about one-third as much memory as it would take to hold the index in RAM. Problem 5: Composability Composability in the context of database systems refers to the ability to seamlessly integrate various features and capabilities in a coherent manner. This is particularly significant when discussing the integration of a new category of capability, like vector search. Non-toy applications will always require both classic database features as well as the new vector search. CRUD Consider the simple sample application for Astra DB. This is about as pure of a RAG application as you will find, using vector search to surface appropriate documentation to the LLM to respond to user questions. However, even a simple demo like this still needs to make “normal,” non-vector queries to Astra DB to retrieve the conversation history, which must also be included with every request to the LLM so that it can “remember” what has already taken place. So the key queries include: AI chatbot Find the most relevant documents (or document fragments) for the user’s question Retrieve the last twenty messages from the user’s conversation In a more realistic use case, one of our solutions engineers was recently working with a company in Asia that wanted to add semantic search to their product catalog, but also wanted to enable term-based matching. For example, if the user searches for [“red” ball valve] then they want to restrict the search to items whose description matches the term “red”, no matter how semantically similar the vector embeddings are. The key new query then (on top of classic functionality like session management, order history, and shopping cart updates) is thus: Restrict the products to those that contain all quoted terms, then find the most similar to the user’s search This second example makes it clear that not only that applications need both classic query functionality and vector search, but they often need to be able to use elements of each in the same query. The current state of the art in this young field is to try to do what I’m calling classic queries in a “normal” database, vector queries in a vector database, and then stitching the two together in an ad hoc fashion when both are required at the same time. This is error-prone, slow, and expensive; its only virtue is that you can make it work until you have a better solution. In Astra DB, we’ve built (and open-sourced) that better solution, on top of Cassandra SAI. Because SAI allows for creating custom index types that are all tied to the Cassandra stable and compaction life cycle, it is straightforward for Astra DB to allow developers to mix and match boolean predicates, term-based search, and vector search, with no overhead of managing and synchronizing separate systems. This gives developers building generative AI applications more sophisticated query capabilities that drive greater productivity and faster time-to-market. Wrap up Vector search engines are an important new database feature with multiple architectural challenges, including scale-out, garbage collection, concurrency, effective use of disk, and composability. I believe that in building vector search for Astra DB we have been able to leverage Cassandra’s capabilities to deliver a best-in-class experience for developers of generative AI applications. , or if you want to get up close and personal with vector search algorithms, check out . Learn more about Astra DB here JVector - By Jonathan Ellis, DataStax Also published here. Lead image . source

5 Hard Vector Search Problems and How We Solved Them In Apache Cassandra

5 হার্ড ভেক্টর অনুসন্ধান সমস্যা এবং কিভাবে আমরা Apache Cassandra এ তাদের সমাধান করেছি

5 Probleme bei der Suche nach harten Vektoren und wie wir sie in Apache Cassandra gelöst haben

5 problemas difíciles de búsqueda de vectores y cómo los resolvimos en Apache Cassandra

5 problèmes de recherche de vecteurs difficiles et comment nous les avons résolus dans Apache Cassandra

5 हार्ड वेक्टर खोज समस्याएं और हमने अपाचे कैसेंड्रा में उन्हें कैसे हल किया

5 つのハード ベクトル検索の問題と Apache Cassandra でのそれらの解決方法

5가지 하드 벡터 검색 문제와 Apache Cassandra에서 이를 해결한 방법

5 problemas difíceis de pesquisa de vetores e como os resolvemos no Apache Cassandra

5 сложных проблем векторного поиска и как мы их решили в Apache Cassandra

5 Zor Vektör Arama Problemi ve Bunları Apache Cassandra'da Nasıl Çözdük?

5 vấn đề về tìm kiếm vectơ cứng và cách chúng tôi giải quyết chúng trong Apache Cassandra

5 个硬向量搜索问题以及我们如何在 Apache Cassandra 中解决它们

DATASTAX

datastax.com

-- employees

Since 2010

COMPANY RANKING

DATASTAX

EVERGREEN INDEX #2984

Representing this company? Click here to claim and customize this page!

DataStax WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

DATASTAX

datastax.com

-- employees

Since 2010

COMPANY RANKING

DATASTAX

EVERGREEN INDEX #2984

RELATED COMPANIESRANK

STORIES ABOUT DATASTAX

Representing this company? Click here to claim and customize this page!

DataStax WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!