“The potential of today’s new era of AI can only be fully realized, in my opinion, if organizations have a strategy to unify data from multiple sources in near real-time without creating numerous copies of data and going through constant iterations of ...

IBM Unleashes the Potential of Data and AI with its Next-Gen IBM Storage Scale System 6000

Its fall 2023 platform release jump-starts GenAI innovation by transforming Domino’s AI Project Hub into an AI ecosystem hub with contributions from cutting-edge AI companies, reduces time to value with expanded data ... to build machine learning models ...

Domino Data Lab Unveils New Generative AI Capabilities in Latest Platform Release

Generative AI applications built with RAG stacks require a vector-enabled database and an orchestration framework like LangChain, to provide memory or context to LLMs for accurate and relevant answers. Developers use LangChain as the leading AI-first toolkit to connect their applications to different data sources.

DataStax LangChain integration builds new link to gen-AI for developers

The need to accelerate AI initiatives is real and widespread across ... models can reduce development time with scalable secure solutions that would revolutionize how easily you can capture, store, analyze, and use data to be more competitive.

Kickstart Your Business to the Next Level with AI Inferencing

DataGPT, a leading provider of conversational AI data analytics software, recently announced the launch of the DataGPT AI

DataGPT Announces DataGPT AI Analyst, Bridging the Gap Between LLMs and Advanced Analytics

Joe McKendrick, research analyst at Unisphere, and Robert Stanley, senior director special projects at Melissa Informatics, joined DBTA's webinar, The State of Data Quality: The Elephant in the AI Room,

The Clash Between Data Quality and AI: Unisphere’s Latest Findings

Aditi Gupta discusses the design choices made early on during service development that were crucial to scaling operations later on at Netflix.

Scaling Defenses Amidst Evolving Threat Landscape

The Horseracing Integrity and Safety Authority (HISA) is collaborating with Amazon Web Services (AWS) to apply data analytics, machine learning and artificial intelligence to ... Under HISA, for the first time in the sport's history, uniform reporting ...

HISA, Amazon Web Services Partner To Use Data Analytics, Artificial Intelligence To Reduce Equine Injuries

In this sponsored post, Devika Garg, PhD, Senior Solutions Marketing Manager for Analytics at Pure Storage, believes that in the current era of data-driven transformation, IT leaders must embrace complexity by simplifying their analytics and data footprint.

Achieving Faster Time To Insights with Modern Data Pipelines

DataStax today announced a new integration with LangChain, the most popular orchestration framework for developing

DataStax Launches New Integration with LangChain, Enables Developers to Build Production-ready Generative AI Applications

DataStax

datastax

A look at two databases that have made claims to the Kubernetes native label: TiDB and DataStax Astra DB. The cloud computing revolution has inspired and benefitted from multiple interrelated trends. The availability of self-service, public cloud infrastructure has helped to drive the adoption of microservice architectures and DevOps practices, including automation and observability. The drive toward containerization and container orchestration has led to the widespread adoption of Kubernetes as an environment for managing cloud-native applications. But one of the lagging areas in this revolution has been data and data infrastructure. For too long, data has been something that has lived outside of Kubernetes, leading to a lot of extra effort and complexity for developers in deploying cloud-native applications. One oft-repeated axiom in the early years of Kubernetes was that it was not yet ready for stateful workloads. Thankfully, a major shift has been quietly underway and has reached a point of maturity. The transformation happened slowly initially, beginning with efforts to containerize existing databases. This worked relatively well in small databases that ran on a single compute node, or databases that had been designed in a cloud-native world, like Apache Cassandra and DynamoDB, but challenges remained. Over the past two to three years, a new generation of databases has emerged. These “Kubernetes native” databases have been designed from the ground up to run on this open-source orchestration system. Here, we’ll define the qualities that make a database Kubernetes native and the benefits of adopting a Kubernetes native database. To do that, we’ll look at two databases claiming the Kubernetes native label: TiDB and DataStax Astra DB. Kubernetes Native MySQL with TiDB First, let’s examine a database with a relational emphasis: (short for Titanium Database). TiDB is an open-source system built by PingCAP that provides a and a columnar database to support hybrid transactional and analytic processing (known as HTAP, for short). TiDB MySQL-compatible database As shown in Figure 1 below, TiDB has a microservice design. The TiDB query layer, TiKV MySQL databases, TiFlash columnar databases, Spark nodes, and metadata management are each deployed as scalable microservices in their clusters. This design separates compute-intensive work from storage-intensive work, as the query and database layers are independently scalable. One critical commitment the TiDB creators made was that the database only runs on Kubernetes. Is that enough to make it Kubernetes native? Let’s dig a bit deeper. First, TiDB is deployed and managed by a Kubernetes operator using (CRDs). The TiDB CRDs include the TiDBCluster, which enables you to specify the scaling and configuration of each microservice and how the database layer components use storage through Kubernetes Persistent Volumes. Additional CRDs are used to deploy monitoring tools and manage operational tasks like backup and restore. custom resources that interfaces with the default K8s scheduler to make more application-aware scheduling decisions. This emphasis on using existing Kubernetes capabilities where available is the mark of a Kubernetes native database. TiDB also has an optional scheduler extension Kubernetes Native Cassandra with DataStax Astra DB Now, look at another Kubernetes native database and note some similarities and differences. that was one of the first to claim to be cloud native, but what does it look like to deploy Cassandra in Kubernetes? is a highly scalable NoSQL database Cassandra that has been factored into microservices, as shown in Figure 2. is a version of Cassandra DataStax Astra DB Like TiDB, the database includes microservices concerned with query processing and data storage, as well as services for identity and access control, data repair, and backup/restore. The data services are particularly interesting in their use of , with Kubernetes Persistent Volumes used only for caching and object storage used for longer-term persistence. Separating compaction into its service enables this compute-intensive processing to happen in the background without affecting the performance of data services serving read and write traffic. storage Astra DB is offered as a managed service available in multiple cloud regions. Each region contains a data plane consisting of the services mentioned above, managed by a Kubernetes operator, as well as infrastructure services, including the stack for observability and for metadata management. Kube-Promethus etcd The data planes are managed by a control plane that can run in one or more clouds to manage customer accounts and databases and provision Kubernetes clusters in new regions. in which multiple user databases can share the same microservices and supporting infrastructure, lowering unit economics for smaller-scale users. One novel aspect of Astra DB is its multi-tenant architecture As users grow their applications, they can move to dedicated resources to achieve optimal performance at scale, all on a “ ” basis. pay-as-you-go Kubernetes Native Database Principles Based on our observations of TiDB and Astra DB, we can derive some ideas of what makes a database Kubernetes native. Many of these correspond to a list of principles for cloud-native data, which I described in an : earlier article First, a database broken into constituent microservices enables each service to be scaled independently. Some types of compute-intensive processing may even be scaled to zero for a true serverless solution, especially when combined with a multitenant design. Composable microservice architecture: Microservices composed of a Kubernetes native database should make maximum usage of Kubernetes APIs for managing the fundamental resources of cloud-native applications: compute resources such as , Kubernetes ingress and services for exposing network access to data and more. This includes leveraging capabilities already present in Kubernetes, such as etcd for metadata management, instead of bringing along components with duplicative functionality. Treat compute, network, and storage as commodities: StatefulSets and deployments for managing workloads, the Persistent Volume subsystem for storage Following common patterns for Kubernetes applications will yield multiple operational benefits, for example, . By default, Kubernetes itself sets a great example that databases should follow for how to be secure: using Kubernetes Secrets to distribute security credentials, only exposing ports as needed, and so on. Leverage Kubernetes best practices: exposing liveness and readiness checks on each microservice to help availability and exposing metrics via the Prometheus PromQL API for observability A Kubernetes native database should embody the Kubernetes principles of declarative management via operators and custom resources, rather than relying on legacy database management UIs and CLIs. When necessary, . The goal is a clean separation of data plane functionality (managing data) from control plane functionality (managing the database). Declarative management via operators: Kubernetes extension points, such as scheduler extensions, can be used to add application-specific behavior Databases and other data infrastructures that faithfully adopt these principles will yield benefits, including a , lower operational complexity resulting in faster time to market, and standards-compliant solutions meeting today’s high availability and security demands. high performance for optimal cost at all scales The Future of Kubernetes Native Data Infrastructure Much progress is still to be made, and it’s not limited to databases alone. Kubernetes native principles can be applied to other types of data infrastructure, including streaming, analytics, and machine learning. Kubernetes native solutions will continue to make strides in multicluster and multi-cloud deployments to scale globally and will adopt multitenancy and serverless principles for better cost optimization. Kubernetes itself has room for improvement in adding more flexibility to StatefulSets and support for multicluster federation. The key to continued progress is open collaboration. The is a highly active group of data geeks bringing together builders of data-intensive applications and the infrastructure that supports them. Data on Kubernetes Community Join us to talk about ideas like developing reusable operators that can manage multiple databases or defining a common set of CRDs for concepts like backup/restore and data loading. Together we’ll continue to push the horizon of cloud computing for the benefit of all. Learn more about Kassandra native databases and more at the digital summit on March 14, 2023. Cassandra Forward This article is based on Chapter 7, “The Kubernetes Native Database,” from the O’Reilly book “ Managing Cloud Native Data on Kubernetes ” by Jeff Carpenter and Patrick McFadin. [ By Jeff Carpenter, DataStax Jeff Carpenter has worked as a software engineer and architect in multiple industries and as a developer advocate at DataStax, helping engineers succeed with Apache Cassandra. He's involved in multiple open source projects in the Cassandra and Kubernetes ecosystems, including Stargate and K8ssandra. He is co-author of the O’Reilly books "Cassandra: The Definitive Guide" and "Managing Cloud Native Data on Kubernetes."

Kubernetes-Native Database: TiDB Vs. DataStax Astra DB

Base de datos nativa de Kubernetes: TiDB vs. Base de datos DataStax Astra

Base de données native Kubernetes : TiDB Vs. DataStax Astra DB

कुबेरनेट्स-नेटिव डेटाबेस: TiDB बनाम। डेटास्टैक्स एस्ट्रा डीबी

Kubernetes ネイティブ データベース: TiDB Vs. DataStax アストラ DB

Banco de dados nativo do Kubernetes: TiDB vs. Data Stax Astra DB

Cơ sở dữ liệu gốc Kubernetes: TiDB Vs. DataStax Astra DB

Kubernetes 原生数据库：TiDB 对比。 DataStax Astra 数据库

Using knowledge graphs to improve the results of retrieval-augmented generation (RAG) applications has become a hot topic. Most examples demonstrate how to build a knowledge graph using a relatively small number of documents. This might be because the typical approach – extracting fine-grained, entity-centric information – just doesn’t scale. Running each document through a model to extract the entities (nodes) and relationships (edges) takes too long (and costs too much) to run on large datasets. We’ve argued that content-centric knowledge graphs – a vector-store allowing links between chunks – are an easier to use and more efficient approach. Here, we put that to the test. We load a subset of the Wikipedia articles from the 2wikimultihop dataset using both techniques and discuss what this means for loading the entire dataset. We demonstrate the results of some questions over the loaded data. We’ll also load the entire dataset – nearly 6 million documents – into a content-centric GraphVectorStore. Entity-centric: LLMGraphTransformer Loading documents into an entity-centric graph store like Neo4j was done using LangChain’s LLMGraphTransformer. The code is based on LangChain's "How to construct knowledge graphs." from langchain_core.documents import Document
from langchain_experimental.graph_transformers import LLMGraphTransformer
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(temperature=0, model_name="gpt-4-turbo")
llm_transformer = LLMGraphTransformer(llm=llm)

from time import perf_counter
start = perf_counter()

documents_to_load = [Document(page_content=line) for line in lines_to_load]
graph_documents = llm_transformer.convert_to_graph_documents(documents_to_load)

end = perf_counter()
print(f"Loaded (but NOT written) {NUM_LINES_TO_LOAD} in {end - start:0.2f}s") Content-centric: GraphVectorStore Loading the data into GraphVectorStore is roughly the same as loading it into a vector store. The only addition is that we compute metadata indicating how each page links to other pages. import json
from langchain_core.graph_vectorstores.links import METADATA_LINKS_KEY, Link

def parse_document(line: str) -> Document:
    para = json.loads(line)

    id = para["id"]
    links = {
        Link.outgoing(kind="href", tag=id)
        for m in para["mentions"]
        if m["ref_ids"] is not None
        for id in m["ref_ids"]
    }
    links.add(Link.incoming(kind="href", tag=id))
    return Document(
        id = id,
        page_content = " ".join(para["sentences"]),
        metadata = {
            "content_id": para["id"],
            METADATA_LINKS_KEY: list(links)
        },
    ) This is also a good example of how you can add your own links between nodes. from langchain_openai import OpenAIEmbeddings
from langchain_community.graph_vectorstores.cassandra import CassandraGraphVectorStore
import cassio

cassio.init(auto=True)
TABLE_NAME = "wiki_load"
store = CassandraGraphVectorStore(
embedding = OpenAIEmbeddings(),
node_table=TABLE_NAME,
insert_timeout = 1000.0,
)

from time import perf_counter
start = perf_counter()

from datasets.wikimultihop.load import parse_document
kg_documents = [parse_document(line) for line in lines_to_load]
store.add_documents(kg_documents)

end = perf_counter()
print(f"Loaded (and written) {NUM_LINES_TO_LOAD} in {end - start:0.2f}s") Loading benchmarks Running at 100 rows, the entity-centric approach using GPT-4o took 405.93s to extract the GraphDocuments and 10.99s to write them to Neo4j, while the content-centric approach took 1.43s. Extrapolating, it would take 41 weeks to load all 5,989,847 pages using the entity-centric approach and about 24 hours using the content-centric approach. But thanks to parallelism, the content-centric approach runs in only 2.5 hours! Assuming the same parallelism benefits, it would still take over four weeks to load everything using the entity-centric approach. I didn’t try it since the estimated cost would be $58,700 — assuming everything worked the first time! Bottom-line: the entity-centric approach of extracting knowledge graphs from content using an LLM was both time and cost prohibitive at scale. On the other hand, using GraphVectorStore was fast and inexpensive. Example answers In this section, a few questions, drawn from the subset of loaded documents, are asked to address the quality of answers. Entity-centric used 7324 prompt tokens and cost $0.03 to produce basically useless answers, while content-centric used 450 prompt tokens and cost $0.002 to produce concise answers directly answering the questions. It may be surprising that the fine-grained Neo4j graph returns useless answers. Looking at the logging from the chain, we see some of why this happens: > Entering new GraphCypherQAChain chain...

Generated Cypher:

cypher

MATCH (a:Album {id: 'The Circle'})-[:RELEASED_BY]->(r:Record_label)

RETURN a.id, r.id


Full Context:

[{'a.id': 'The Circle', 'r.id': 'Restless'}]


> Finished chain.

{'query': "When was 'The Circle' released?", 'result': "I don't know the answer."} So, the fine-grained schema only returned information about the record label. It makes sense that the LLM wasn’t able to answer the question based on the retrieved information. Conclusion Extracting fine-grained, entity-specific knowledge graphs is time- and cost-prohibitive at scale. When asked questions about the subset of data that was loaded, the additional granularity (and extra cost of loading the fine-grained graph) returned more tokens to include the prompt but generated useless answers! GraphVectorStore takes a coarse-grained, content-centric approach that makes it fast and easy to build a knowledge graph. You can start with your existing code for populating a VectorStore using LangChain and add links (edges) between chunks to improve the retrieval process. Graph RAG is a useful tool for enabling generative AI RAG applications to retrieve more deeply relevant contexts. But using a fine-grained, entity-centric approach does not scale to production needs. If you're looking to add knowledge graph capabilities to your RAG application, try GraphVectorStore. By Ben Chambers, DataStax Using knowledge graphs to improve the results of retrieval-augmented generation (RAG) applications has become a hot topic. Most examples demonstrate how to build a knowledge graph using a relatively small number of documents. This might be because the typical approach – extracting fine-grained, entity-centric information – just doesn’t scale. Running each document through a model to extract the entities (nodes) and relationships (edges) takes too long (and costs too much) to run on large datasets. We’ve argued that content-centric knowledge graphs – a vector-store allowing links between chunks – are an easier to use and more efficient approach. Here, we put that to the test. We load a subset of the Wikipedia articles from the 2wikimultihop dataset using both techniques and discuss what this means for loading the entire dataset. We demonstrate the results of some questions over the loaded data. We’ll also load the entire dataset – nearly 6 million documents – into a content-centric GraphVectorStore . content-centric knowledge graphs content-centric knowledge graphs 2wikimultihop 2wikimultihop GraphVectorStore GraphVectorStore Entity-centric: LLMGraphTransformer Loading documents into an entity-centric graph store like Neo4j was done using LangChain’s LLMGraphTransformer. The code is based on LangChain's "How to construct knowledge graphs." "How to construct knowledge graphs." "How to construct knowledge graphs." from langchain_core.documents import Document
from langchain_experimental.graph_transformers import LLMGraphTransformer
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(temperature=0, model_name="gpt-4-turbo")
llm_transformer = LLMGraphTransformer(llm=llm)

from time import perf_counter
start = perf_counter()

documents_to_load = [Document(page_content=line) for line in lines_to_load]
graph_documents = llm_transformer.convert_to_graph_documents(documents_to_load)

end = perf_counter()
print(f"Loaded (but NOT written) {NUM_LINES_TO_LOAD} in {end - start:0.2f}s") from langchain_core.documents import Document
from langchain_experimental.graph_transformers import LLMGraphTransformer
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(temperature=0, model_name="gpt-4-turbo")
llm_transformer = LLMGraphTransformer(llm=llm)

from time import perf_counter
start = perf_counter()

documents_to_load = [Document(page_content=line) for line in lines_to_load]
graph_documents = llm_transformer.convert_to_graph_documents(documents_to_load)

end = perf_counter()
print(f"Loaded (but NOT written) {NUM_LINES_TO_LOAD} in {end - start:0.2f}s") Content-centric: GraphVectorStore Loading the data into GraphVectorStore is roughly the same as loading it into a vector store. The only addition is that we compute metadata indicating how each page links to other pages. import json
from langchain_core.graph_vectorstores.links import METADATA_LINKS_KEY, Link

def parse_document(line: str) -> Document:
    para = json.loads(line)

    id = para["id"]
    links = {
        Link.outgoing(kind="href", tag=id)
        for m in para["mentions"]
        if m["ref_ids"] is not None
        for id in m["ref_ids"]
    }
    links.add(Link.incoming(kind="href", tag=id))
    return Document(
        id = id,
        page_content = " ".join(para["sentences"]),
        metadata = {
            "content_id": para["id"],
            METADATA_LINKS_KEY: list(links)
        },
    ) import json
from langchain_core.graph_vectorstores.links import METADATA_LINKS_KEY, Link

def parse_document(line: str) -> Document:
    para = json.loads(line)

    id = para["id"]
    links = {
        Link.outgoing(kind="href", tag=id)
        for m in para["mentions"]
        if m["ref_ids"] is not None
        for id in m["ref_ids"]
    }
    links.add(Link.incoming(kind="href", tag=id))
    return Document(
        id = id,
        page_content = " ".join(para["sentences"]),
        metadata = {
            "content_id": para["id"],
            METADATA_LINKS_KEY: list(links)
        },
    ) This is also a good example of how you can add your own links between nodes. from langchain_openai import OpenAIEmbeddings
from langchain_community.graph_vectorstores.cassandra import CassandraGraphVectorStore
import cassio

cassio.init(auto=True)
TABLE_NAME = "wiki_load"
store = CassandraGraphVectorStore(
embedding = OpenAIEmbeddings(),
node_table=TABLE_NAME,
insert_timeout = 1000.0,
)

from time import perf_counter
start = perf_counter()

from datasets.wikimultihop.load import parse_document
kg_documents = [parse_document(line) for line in lines_to_load]
store.add_documents(kg_documents)

end = perf_counter()
print(f"Loaded (and written) {NUM_LINES_TO_LOAD} in {end - start:0.2f}s") from langchain_openai import OpenAIEmbeddings
from langchain_community.graph_vectorstores.cassandra import CassandraGraphVectorStore
import cassio

cassio.init(auto=True)
TABLE_NAME = "wiki_load"
store = CassandraGraphVectorStore(
embedding = OpenAIEmbeddings(),
node_table=TABLE_NAME,
insert_timeout = 1000.0,
)

from time import perf_counter
start = perf_counter()

from datasets.wikimultihop.load import parse_document
kg_documents = [parse_document(line) for line in lines_to_load]
store.add_documents(kg_documents)

end = perf_counter()
print(f"Loaded (and written) {NUM_LINES_TO_LOAD} in {end - start:0.2f}s") Loading benchmarks Running at 100 rows, the entity-centric approach using GPT-4o took 405.93s to extract the GraphDocuments and 10.99s to write them to Neo4j, while the content-centric approach took 1.43s. Extrapolating, it would take 41 weeks to load all 5,989,847 pages using the entity-centric approach and about 24 hours using the content-centric approach. But thanks to parallelism, the content-centric approach runs in only 2.5 hours! Assuming the same parallelism benefits, it would still take over four weeks to load everything using the entity-centric approach. I didn’t try it since the estimated cost would be $58,700 — assuming everything worked the first time! Bottom-line: the entity-centric approach of extracting knowledge graphs from content using an LLM was both time and cost prohibitive at scale. On the other hand, using GraphVectorStore was fast and inexpensive. Example answers Example answers In this section, a few questions, drawn from the subset of loaded documents, are asked to address the quality of answers. Entity-centric used 7324 prompt tokens and cost $0.03 to produce basically useless answers, while content-centric used 450 prompt tokens and cost $0.002 to produce concise answers directly answering the questions. It may be surprising that the fine-grained Neo4j graph returns useless answers. Looking at the logging from the chain, we see some of why this happens: > Entering new GraphCypherQAChain chain...

Generated Cypher:

cypher

MATCH (a:Album {id: 'The Circle'})-[:RELEASED_BY]->(r:Record_label)

RETURN a.id, r.id


Full Context:

[{'a.id': 'The Circle', 'r.id': 'Restless'}]


> Finished chain.

{'query': "When was 'The Circle' released?", 'result': "I don't know the answer."} > Entering new GraphCypherQAChain chain...

Generated Cypher:

cypher

MATCH (a:Album {id: 'The Circle'})-[:RELEASED_BY]->(r:Record_label)

RETURN a.id, r.id


Full Context:

[{'a.id': 'The Circle', 'r.id': 'Restless'}]


> Finished chain.

{'query': "When was 'The Circle' released?", 'result': "I don't know the answer."} So, the fine-grained schema only returned information about the record label. It makes sense that the LLM wasn’t able to answer the question based on the retrieved information. Conclusion Extracting fine-grained, entity-specific knowledge graphs is time- and cost-prohibitive at scale. When asked questions about the subset of data that was loaded, the additional granularity (and extra cost of loading the fine-grained graph) returned more tokens to include the prompt but generated useless answers! GraphVectorStore takes a coarse-grained, content-centric approach that makes it fast and easy to build a knowledge graph. You can start with your existing code for populating a VectorStore using LangChain and add links (edges) between chunks to improve the retrieval process. Graph RAG is a useful tool for enabling generative AI RAG applications to retrieve more deeply relevant contexts. But using a fine-grained, entity-centric approach does not scale to production needs. If you're looking to add knowledge graph capabilities to your RAG application, try GraphVectorStore . GraphVectorStore GraphVectorStore By Ben Chambers , DataStax Ben Chambers

How to Save $70K Building a Knowledge Graph for RAG on 6M Wikipedia Pages

Hoe om $70 000 te bespaar Die bou van 'n kennisgrafiek vir JOOL op 6M Wikipedia-bladsye

كيفية توفير 70 ألف دولار من خلال إنشاء رسم بياني للمعرفة لـ RAG على 6 ملايين صفحة ويكيبيديا

Com estalviar 70.000 dòlars Construint un gràfic de coneixement per a RAG a 6 milions de pàgines de Viquipèdia

Cómo ahorrar $70 000 creando un gráfico de conocimiento para RAG en 6 millones de páginas de Wikipedia

600 万の Wikipedia ページで RAG のナレッジ グラフを構築して 7 万ドルを節約する方法

របៀបសន្សំ $70K បង្កើត Knowledge Graph សម្រាប់ RAG នៅលើទំព័រ Wikipedia 6M

6 сая Wikipedia хуудсан дээр RAG-д зориулсан мэдлэгийн график бүтээхэд 70 мянган долларыг хэрхэн хэмнэх вэ

6M विकिपिडिया पृष्ठहरूमा RAG को लागि ज्ञान ग्राफ निर्माण गरेर $ 70K कसरी बचत गर्ने

په 6M ويکيپېډيا پاڼو کې د RAG لپاره د پوهې ګراف جوړول $70K څنګه خوندي کول

Cum să economisiți 70.000 USD Construirea unui grafic de cunoștințe pentru RAG pe 6 milioane de pagini Wikipedia

ከመይ ጌርና $70K ንቑጥብ ንRAG ዝኸውን ናይ ፍልጠት ግራፍ ኣብ 6M ዊኪፒዲያ ገጻት ንምህናጽ

Indlela yokugcina i-$ 70K yokwakha iGrafu yoLwazi lwe-RAG kwi-6M yamaphepha e-Wikipedia

How to Build a Production-Grade Text2SQL Engine

কিভাবে একটি প্রোডাকশন-গ্রেড Text2SQL ইঞ্জিন তৈরি করবেন

So erstellen Sie eine produktionsreife Text2SQL-Engine

Cómo crear un motor Text2SQL de nivel de producción

Comment créer un moteur Text2SQL de niveau production

प्रोडक्शन-ग्रेड टेक्स्ट2एसक्यूएल इंजन कैसे बनाएं

本番環境レベルの Text2SQL エンジンを構築する方法

프로덕션 등급 Text2SQL 엔진을 구축하는 방법

Como construir um mecanismo Text2SQL de nível de produção

Как создать Text2SQL-движок промышленного уровня

Üretim Seviyesinde Bir Text2SQL Motoru Nasıl Oluşturulur

Cách xây dựng một công cụ Text2SQL cấp sản xuất

如何构建生产级 Text2SQL 引擎

Building Advanced Video Search: Frame Search Versus Multi-Modal Embeddings

উন্নত ভিডিও অনুসন্ধান তৈরি করা: ফ্রেম অনুসন্ধান বনাম মাল্টি-মোডাল এম্বেডিং

Erstellen einer erweiterten Videosuche: Frame-Suche im Vergleich zu multimodalen Einbettungen

Creación de búsqueda de vídeo avanzada: búsqueda de fotogramas frente a incrustaciones multimodales

Création d'une recherche vidéo avancée : recherche de trames et intégrations multimodales

उन्नत वीडियो खोज का निर्माण: फ़्रेम खोज बनाम मल्टी-मोडल एम्बेडिंग

高度なビデオ検索の構築: フレーム検索とマルチモーダル埋め込み

고급 비디오 검색 구축: 프레임 검색과 다중 모드 임베딩 비교

Construindo pesquisa avançada de vídeo: pesquisa de quadros versus incorporações multimodais

Создание расширенного поиска видео: поиск кадров в сравнении с мультимодальными встраиваниями

Gelişmiş Video Arama Oluşturma: Çerçeve Aramaya Karşı Çok Modlu Yerleştirmeler

Xây dựng Tìm kiếm video nâng cao: Tìm kiếm khung so với nhúng đa phương thức

构建高级视频搜索：帧搜索与多模态嵌入

A lot of questions are swirling about the state of generative AI right now. How far along are companies with their bespoke GenAI efforts? Are organizations actually building AI applications using their own proprietary data in ways that move the needle? What kind of architecture is required? These questions aren’t surprising — there’s a huge range of opinions about AI out there, from unabashed optimism to jaded cynicism, mixed with a lot of hype. I’ve found that it helps to clarify the state of affairs in the GenAI marketplace across three areas: what the market wants to see, fears about what might happen, and what will happen (and how) in 2024. What Does Everyone Want — and What Are They Afraid of? GenAI exploded on the scene in late 2022 when OpenAI released ChatGPT and showed how powerful and accessible this kind of technology could be. The excitement about the potential upside of AI was everywhere. In short order, GenAI was going to be infused into every application at every enterprise. Investors envisioned a hockey-stick-like growth curve for companies that provide the infrastructure to support GenAI. The naysayers, on the other hand, envisioned a dystopian AI future that’s a cross between “Westworld” and “Black Mirror.” Others warn of an AI bubble. From an investment perspective, some say it’s like crypto all over again — lots of excitement and hype, and then a smoking crater. I think both of these fears are unfounded. Sure, with every new technology wave, there’ll be bad actors using GenAI for the wrong reasons. And the excitement about the possibilities of GenAI is everywhere; it does have a “bubbly” feel to it, and might even be more strident than the crypto buzz. But the big difference between GenAI and crypto is the fact that there are many, many real use cases for the former, across organizations and across industries. In crypto, there was one strong use case: financial transactions between untrusted parties (aka money laundering). That’s something that the mainstream isn’t quite as interested in. Right now, the state of affairs for GenAI applications reminds me of e-commerce in the late 1990s, when companies were trying to figure out how to make it safe to use credit cards over the Internet. It took a little while for organizations to figure out how to do it securely, but once they did, suddenly everyone had an e-commerce site. The parallel I see in GenAI right now: how to ensure that language models don’t return inaccurate responses by hallucinating. The good news? That’s been figured out (thanks to retrieval-augmented generation, or RAG; more on that below). Where Are We Now, and Where Are We Going? A lot of what we saw last year were proof-of-concept GenAI projects: apps to demonstrate to a company’s leadership what’s possible. But very few companies have moved beyond that to build applications that are in full production. By “production,” I mean that an organization has an AI application that is being used by customers or employees in a non-prototype way. In other words, it’s available as a routine part of activities within some segments of business operations. It might be the front office, it might be what’s behind a call to customer service, but it’s somewhere near the mainstream part of business. Walmart is a good early example of this. The retailer announced in January that it has added GenAI-powered search to its shopping app. Apple is reportedly testing a GenAI tool to help its employees provide speedier technical support. Until we start seeing more examples like this, GenAI is going to linger a little longer in the early stage of what Gartner calls the “hype cycle.” Volkswagen just announced an in-house lab to develop GenAI apps for navigation and infotainment applications for its automobiles. That said, we’re not as far from reaching the “plateau of productivity” as some might think. As I mentioned earlier, trusting models’ output has been a hurdle for organizations that are still grappling with how to produce relevant and accurate large language model (LLM) responses by reducing hallucinations. RAG, which provides models with additional data or context in real-time from other sources — most often, a database that can store vectors — is being employed now to help solve this problem. This technology advancement is a key to developing domain-specific, bespoke GenAI applications built on organizations’ most valuable asset: their own data. While RAG has emerged as the de facto method for getting enterprise context into GenAI applications, fine-tuning — when a pre-trained model is trained further on a subset of data — is often mentioned as well. There are times when this method can be useful, but RAG is the right choice if there’s any concern for privacy, security, or speed. Regardless of how context is added to the application, the big question I get often from investors I’ve been speaking with is when will companies start to make money from GenAI apps? My response: Most of the enterprises you track are consumption-based businesses. Many are now supporting the experiments, proofs of concept (POCs), and niche apps that their customers have built; those don’t do much in the way of consumption. But this is starting to change as major AI applications start to go from POC into true production. I predict this is going to happen in a significant way by the end of 2024. It will come to fruition in the second half of 2024 starting in two places. First, it’s taking hold in retail (see the Walmart example mentioned earlier). You’ll also see widespread adoption in what I call the “AI intranet” area: Chat with PDF, knowledge bases, and internal call centers. With the consumption that these kinds of apps drive, companies like Microsoft, Google, and even Oracle are starting to report results from AI. Outside of the realm of hyperscalers, other AI infrastructure companies will likely start to highlight lifts in the earnings reports they release in January, February, and March of next year. The Path to Production for GenAI Applications The groundwork has already been laid for consumption-based AI infrastructure companies. We’ve already seen strong, commercial proof points that show what’s possible for a large base of domain-specific, bespoke applications. From the creative AI apps — Midjourney, Adobe Firefly, and other image generators, for example — to knowledge apps like GitHub Copilot (over 1 million developers use it), Glean, and others, these applications have enjoyed great adoption and have driven significant productivity gains. Progress on bespoke apps is most advanced in industries and use cases that need to facilitate delivering knowledge to the point of interaction. The knowledge will come from their own data, using off-the-shelf models (either open source or proprietary), RAG, and the cloud provider of their choice. Three elements are required for enterprises to build bespoke GenAI apps that are ready for the rigors of functioning at the production scale: smart context, relevance, and scalability. Smart Context Let’s take a quick look at how proprietary data is used to generate useful, relevant, and accurate responses in GenAI applications. Applications take user input in the shape of all kinds of data and feed it all into an embedding engine, which essentially derives meaning from the data, retrieves information from a vector database using RAG, and builds the “smart context” that the LLM can use to generate a contextualized, hallucination-free response that’s presented to the user in real-time. Relevance This isn’t a topic you hear much about at operational database companies. But in the field of AI and vector databases, relevance is a mix of recall and precision that’s critical to producing useful, accurate, non-hallucinatory responses. Unlike traditional database operations, vector databases enable semantic or similarity search, which is non-deterministic in nature. Because of this, the results returned for the same query can be different depending on the context and how the search process is executed. This is where accuracy and relevance play a key role in how the vector database operates in real-world applications. Natural interaction requires that the results returned on a similarity search are accurate AND relevant to the requested query. Scalability GenAI apps that go beyond POCs and into production require high throughput. Throughput essentially is the amount of data that can be stored, accessed, or retrieved in a given amount of time. High throughput is critical to delivering real-time, interactive, data-intensive features at scale; writes often involve billions of vectors from multiple sources, and GenAI applications can generate massive amounts of requests per second. Wrapping Up As with earlier waves of technology innovation, GenAI is following an established pattern, and all signs point to it moving even faster than previous tech revolutions. If you cut through all the negative and positive hype about it, it’s clear that promising progress is being made by companies working to move their POC GenAI apps to production. And companies like my employer DataStax that provide the scalable, easy-to-build-on foundations for these apps will start seeing the benefits of their customers’ consumption sooner than some might think. By Ed Anuff, DataStax Learn more about how DataStax enables customers to get their GenAI apps to production. A lot of questions are swirling about the state of generative AI right now. How far along are companies with their bespoke GenAI efforts? Are organizations actually building AI applications using their own proprietary data in ways that move the needle? What kind of architecture is required? These questions aren’t surprising — there’s a huge range of opinions about AI out there, from unabashed optimism to jaded cynicism, mixed with a lot of hype. I’ve found that it helps to clarify the state of affairs in the GenAI marketplace across three areas: what the market wants to see, fears about what might happen, and what will happen (and how) in 2024. What Does Everyone Want — and What Are They Afraid of? GenAI exploded on the scene in late 2022 when OpenAI released ChatGPT and showed how powerful and accessible this kind of technology could be. The excitement about the potential upside of AI was everywhere. In short order, GenAI was going to be infused into every application at every enterprise. Investors envisioned a hockey-stick-like growth curve for companies that provide the infrastructure to support GenAI. The naysayers, on the other hand, envisioned a dystopian AI future that’s a cross between “Westworld” and “Black Mirror.” Others warn of an AI bubble. From an investment perspective, some say it’s like crypto all over again — lots of excitement and hype, and then a smoking crater. warn I think both of these fears are unfounded. Sure, with every new technology wave, there’ll be bad actors using GenAI for the wrong reasons. And the excitement about the possibilities of GenAI is everywhere; it does have a “bubbly” feel to it, and might even be more strident than the crypto buzz. But the big difference between GenAI and crypto is the fact that there are many, many real use cases for the former, across organizations and across industries. In crypto, there was one strong use case: financial transactions between untrusted parties (aka money laundering). That’s something that the mainstream isn’t quite as interested in. Right now, the state of affairs for GenAI applications reminds me of e-commerce in the late 1990s , when companies were trying to figure out how to make it safe to use credit cards over the Internet. It took a little while for organizations to figure out how to do it securely, but once they did, suddenly everyone had an e-commerce site. GenAI applications reminds me of e-commerce in the late 1990s The parallel I see in GenAI right now: how to ensure that language models don’t return inaccurate responses by hallucinating . The good news? That’s been figured out (thanks to retrieval-augmented generation , or RAG; more on that below). hallucinating retrieval-augmented generation Where Are We Now, and Where Are We Going? A lot of what we saw last year were proof-of-concept GenAI projects: apps to demonstrate to a company’s leadership what’s possible. But very few companies have moved beyond that to build applications that are in full production. By “production,” I mean that an organization has an AI application that is being used by customers or employees in a non-prototype way. In other words, it’s available as a routine part of activities within some segments of business operations. It might be the front office, it might be what’s behind a call to customer service, but it’s somewhere near the mainstream part of business. Walmart is a good early example of this. The retailer announced in January that it has added GenAI-powered search to its shopping app. Apple is reportedly testing a GenAI tool to help its employees provide speedier technical support. Until we start seeing more examples like this, GenAI is going to linger a little longer in the early stage of what Gartner calls the “ hype cycle .” announced in January reportedly hype cycle Volkswagen just announced an in-house lab to develop GenAI apps for navigation and infotainment applications for its automobiles. That said, we’re not as far from reaching the “plateau of productivity” as some might think. As I mentioned earlier, trusting models’ output has been a hurdle for organizations that are still grappling with how to produce relevant and accurate large language model (LLM) responses by reducing hallucinations. large language model RAG, which provides models with additional data or context in real-time from other sources — most often, a database that can store vectors — is being employed now to help solve this problem. This technology advancement is a key to developing domain-specific, bespoke GenAI applications built on organizations’ most valuable asset: their own data. While RAG has emerged as the de facto method for getting enterprise context into GenAI applications, fine-tuning — when a pre-trained model is trained further on a subset of data — is often mentioned as well. There are times when this method can be useful , but RAG is the right choice if there’s any concern for privacy, security, or speed. There are times when this method can be useful Regardless of how context is added to the application, the big question I get often from investors I’ve been speaking with is when will companies start to make money from GenAI apps? My response: Most of the enterprises you track are consumption-based businesses. Many are now supporting the experiments, proofs of concept (POCs), and niche apps that their customers have built; those don’t do much in the way of consumption. But this is starting to change as major AI applications start to go from POC into true production. I predict this is going to happen in a significant way by the end of 2024. It will come to fruition in the second half of 2024 starting in two places. First, it’s taking hold in retail (see the Walmart example mentioned earlier). You’ll also see widespread adoption in what I call the “AI intranet” area: Chat with PDF, knowledge bases, and internal call centers. With the consumption that these kinds of apps drive, companies like Microsoft , Google , and even Oracle are starting to report results from AI. Outside of the realm of hyperscalers, other AI infrastructure companies will likely start to highlight lifts in the earnings reports they release in January, February, and March of next year. Microsoft Google Oracle report results The Path to Production for GenAI Applications The groundwork has already been laid for consumption-based AI infrastructure companies. We’ve already seen strong, commercial proof points that show what’s possible for a large base of domain-specific, bespoke applications. From the creative AI apps — Midjourney, Adobe Firefly, and other image generators, for example — to knowledge apps like GitHub Copilot ( over 1 million developers use it ), Glean, and others, these applications have enjoyed great adoption and have driven significant productivity gains. over 1 million developers use it Progress on bespoke apps is most advanced in industries and use cases that need to facilitate delivering knowledge to the point of interaction. The knowledge will come from their own data, using off-the-shelf models (either open source or proprietary), RAG, and the cloud provider of their choice. Three elements are required for enterprises to build bespoke GenAI apps that are ready for the rigors of functioning at the production scale: smart context, relevance, and scalability. Smart Context Let’s take a quick look at how proprietary data is used to generate useful, relevant, and accurate responses in GenAI applications. Applications take user input in the shape of all kinds of data and feed it all into an embedding engine, which essentially derives meaning from the data, retrieves information from a vector database using RAG, and builds the “smart context” that the LLM can use to generate a contextualized, hallucination-free response that’s presented to the user in real-time. Relevance This isn’t a topic you hear much about at operational database companies. But in the field of AI and vector databases, relevance is a mix of recall and precision that’s critical to producing useful, accurate, non-hallucinatory responses. Unlike traditional database operations, vector databases enable semantic or similarity search, which is non-deterministic in nature. Because of this, the results returned for the same query can be different depending on the context and how the search process is executed. This is where accuracy and relevance play a key role in how the vector database operates in real-world applications. Natural interaction requires that the results returned on a similarity search are accurate AND relevant to the requested query. Scalability GenAI apps that go beyond POCs and into production require high throughput. Throughput essentially is the amount of data that can be stored, accessed, or retrieved in a given amount of time. High throughput is critical to delivering real-time, interactive, data-intensive features at scale; writes often involve billions of vectors from multiple sources, and GenAI applications can generate massive amounts of requests per second. Wrapping Up As with earlier waves of technology innovation, GenAI is following an established pattern, and all signs point to it moving even faster than previous tech revolutions. If you cut through all the negative and positive hype about it, it’s clear that promising progress is being made by companies working to move their POC GenAI apps to production. And companies like my employer DataStax that provide the scalable, easy-to-build-on foundations for these apps will start seeing the benefits of their customers’ consumption sooner than some might think. By Ed Anuff, DataStax By Ed Anuff, DataStax Learn more about how DataStax enables customers to get their GenAI apps to production. Learn more about how DataStax enables customers to get their GenAI apps to production . DataStax enables customers to get their GenAI apps to production

When Will Infrastructure Companies See Gains from Generative AI?

কখন অবকাঠামো কোম্পানিগুলি জেনারেটিভ এআই থেকে লাভ দেখতে পাবে?

Wann werden Infrastrukturunternehmen von generativer KI profitieren?

¿Cuándo verán las empresas de infraestructura los beneficios de la IA generativa?

Quand les entreprises d’infrastructure bénéficieront-elles de l’IA générative ?

बुनियादी ढांचा कम्पनियों को जनरेटिव एआई से लाभ कब मिलेगा?

インフラ企業はいつジェネレーティブ AI から利益を得られるのでしょうか?

인프라 회사는 언제 Generative AI로부터 이익을 얻을 수 있습니까?

Quando as empresas de infraestrutura verão os ganhos da IA generativa?

Когда инфраструктурные компании увидят выгоду от генеративного искусственного интеллекта?

Altyapı Şirketleri Üretken Yapay Zekadan Kazançları Ne Zaman Görecek?

Khi nào các công ty cơ sở hạ tầng sẽ thu được lợi nhuận từ AI sáng tạo?

基础设施公司何时能从生成式人工智能中获益？

How to Use Knowledge Graphs for Retrieval-Augmented Generation—Without a Graph DB

পুনরুদ্ধার-অগমেন্টেড জেনারেশনের জন্য জ্ঞান গ্রাফগুলি কীভাবে ব্যবহার করবেন—একটি গ্রাফ ডিবি ছাড়াই

So nutzen Sie Wissensgraphen zur abruferweiterten Generierung – ohne Graph-DB

Cómo utilizar gráficos de conocimiento para la generación de recuperación aumentada, sin una base de datos de gráficos

Comment utiliser les Knowledge Graphs pour la génération augmentée par récupération – sans base de données graphique

रिट्रीवल-ऑगमेंटेड जेनरेशन के लिए नॉलेज ग्राफ़ का उपयोग कैसे करें - बिना ग्राफ़ DB के

グラフ DB を使用せずにナレッジ グラフを検索拡張生成に使用する方法

검색 증강 생성을 위해 지식 그래프를 사용하는 방법 - 그래프 DB 없이

Como usar gráficos de conhecimento para geração aumentada de recuperação - sem um banco de dados gráfico

Как использовать графы знаний для генерации с расширенным поиском — без базы данных графов

Almayla Artırılmış Oluşturma için Bilgi Grafikleri Nasıl Kullanılır - Grafik Veritabanı Olmadan

Cách sử dụng Sơ đồ tri thức cho thế hệ tăng cường truy xuất—Không cần cơ sở dữ liệu đồ thị

如何使用知识图谱进行检索增强生成（无需图形数据库）

Retrieval augmented generation (RAG) is by now of generative artificial intelligence (AI) applications. Supplementing your application prompt with relevant context retrieved from a vector database can dramatically increase accuracy and reduce hallucinations. This means that increasing relevance in vector search results has a direct correlation to the quality of your RAG application. a standard part There are two reasons RAG remains popular and increasingly relevant even : as large language models (LLMs) increase their context window LLM response time and price both increase linearly with context length. LLMs still struggle with both and across massive contexts. retrieval reasoning But isn’t a magic wand. In particular, the most common design, dense passage retrieval (DPR), represents both queries and passages as a single embedding vector and uses straightforward to score relevance. This means DPR relies heavily on the embeddings model having the breadth of training to recognize all the relevant search terms. RAG cosine similarity Unfortunately, off-the-shelf models struggle with unusual terms, including names, that are not commonly in their training data. DPR also tends to be hypersensitive to chunking strategy, which can cause a relevant passage to be missed if it’s surrounded by a lot of irrelevant information. All of this creates a burden on the application developer to “get it right the first time,” because a mistake usually results in the need to rebuild the index from scratch. Solving DPR’s challenges with ColBERT is a new way of scoring passage relevance using a language model that substantially solves the problems with DPR. This diagram from shows why it’s so exciting: ColBERT BERT the first ColBERT paper This compares the performance of ColBERT with other state-of-the-art solutions for the MS-MARCO dataset. (MS-MARCO is a set of Bing queries for which scored the most relevant passages by hand. It’s one of the better retrieval benchmarks.) Lower and to the right is better. Microsoft In short, ColBERT handily outperforms the field of mostly significantly more complex solutions at the cost of a small increase in latency. To test this, I created and indexed over 1,000 Wikipedia articles with both ada002 DPR and ColBERT. I found that ColBERT delivers significantly better results on unusual search terms. a demo The following screenshot shows that DPR fails to recognize the unusual name of William H. Herndon, an associate of Abraham Lincoln, while ColBERT finds the reference in the Springfield article. Also note that ColBERT’s No. 2 result is for a different William, while none of DPR’s results are relevant. ColBERT is often described in dense machine learning jargon, but it’s actually very straightforward. I’ll show how to implement ColBERT retrieval and scoring on with only a few lines of Python and (CQL). DataStax Astra DB Cassandra Query Language The big idea Instead of traditional, single-vector-based DPR that turns passages into a single “embedding” vector, ColBERT generates a contextually influenced for each token in the passages. ColBERT similarly generates vectors for each token in the query. vector (Tokenization refers to breaking up input into fractions of words before processing by an LLM. .) Andrej Karpathy, a founding member of the OpenAI team, just released an outstanding video on how this works Then, the score of each document is the sum of the maximum similarity of each query embedding to any of the document embeddings: def maxsim(qv, document_embeddings):
    return max(qv @ dv for dv in document_embeddings)

def score(query_embeddings, document_embeddings):
    return sum(maxsim(qv, document_embeddings) for qv in query_embeddings) (@ is the PyTorch operator for dot product and is the most common .) measure of vector similarity That’s it — you can implement ColBERT scoring in four lines of Python! Now you understand ColBERT better than 99% of the people posting about it on X (formerly known as Twitter). The rest of the ColBERT papers deal with: How do you fine-tune the BERT model to generate the best embeddings for a given data set? How do you limit the set of documents for which you compute the (relatively expensive) score shown here? The first question is optional and out of scope for this writeup. I’ll use the pretrained ColBERT checkpoint. But the second is straightforward to do with a vector database like DataStax Astra DB. ColBERT on Astra DB There is a popular Python all-in-one library for ColBERT called ; however, it assumes a static dataset. One of the powerful features of RAG applications is responding to dynamically . So instead, I’m going to use Astra’s vector index to narrow the set of documents I need to score down to the best candidates for each subvector. RAGatouille changing data in real time There are two steps when adding ColBERT to a RAG application: ingestion and retrieval. Ingestion Because each document chunk will have multiple embeddings associated with it, I’ll need two tables: CREATE TABLE chunks (
    title text,
    part int,
    body text,
    PRIMARY KEY (title, part)
);

CREATE TABLE colbert_embeddings (
    title text,
    part int,
    embedding_id int,
    bert_embedding vector<float, 128>,
    PRIMARY KEY (title, part, embedding_id)
);

CREATE INDEX colbert_ann ON colbert_embeddings(bert_embedding)
  WITH OPTIONS = { 'similarity_function': 'DOT_PRODUCT' }; After installing the ColBERT library ( ) and downloading , I can load documents into these tables: pip install colbert-ai the pretrained BERT checkpoint from colbert.infra.config import ColBERTConfig
from colbert.modeling.checkpoint import Checkpoint
from colbert.indexing.collection_encoder import CollectionEncoder

from cassandra.concurrent import execute_concurrent_with_args
from db import DB


def encode_and_save(title, passages):
    db = DB()
    cf = ColBERTConfig(checkpoint='checkpoints/colbertv2.0')
    cp = Checkpoint(cf.checkpoint, colbert_config=cf)
    encoder = CollectionEncoder(cf, cp)

    # encode_passages returns a flat list of embeddings and a list of how many correspond to each passage
    embeddings_flat, counts = encoder.encode_passages(passages)

    # split up embeddings_flat into a nested list
    start_indices = [0] + list(itertools.accumulate(counts[:-1]))
    embeddings_by_part = [embeddings_flat[start:start+count] for start, count in zip(start_indices, counts)]

    # insert into the database
    for part, embeddings in enumerate(embeddings_by_part):
        execute_concurrent_with_args(db.session,
                                     db.insert_colbert_stmt,
                                     [(title, part, i, e) for i, e in enumerate(embeddings)]) (I like to encapsulate my DB logic in a dedicated module; you can access the full source in my .) GitHub repository Retrieval Then retrieval looks like this: def retrieve_colbert(query):
    db = DB()
    cf = ColBERTConfig(checkpoint='checkpoints/colbertv2.0')
    cp = Checkpoint(cf.checkpoint, colbert_config=cf)
    encode = lambda q: cp.queryFromText([q])[0]

    query_encodings = encode(query)
    # find the most relevant documents for each query embedding. using a set
    # handles duplicates so we don't retrieve the same one more than once
    docparts = set()
    for qv in query_encodings:
        rows = db.session.execute(db.query_colbert_ann_stmt, [list(qv)])
        docparts.update((row.title, row.part) for row in rows)
    # retrieve these relevant documents and score each one
    scores = {}
    for title, part in docparts:
        rows = db.session.execute(db.query_colbert_parts_stmt, [title, part])
        embeddings_for_part = [tensor(row.bert_embedding) for row in rows]
        scores[(title, part)] = score(query_encodings, embeddings_for_part)
    # return the source chunk for the top 5
    return sorted(scores, key=scores.get, reverse=True)[:5] Here is the query being executed for the most-relevant-documents part ( ): db.query_colbert_ann_stmt SELECT title, part
FROM colbert_embeddings
ORDER BY bert_embedding ANN OF ?
LIMIT 5 Beyond the basics: RAGStack This article and the linked repository briefly introduce how ColBERT works. You can implement this today with your own data and see immediate results. As with everything in AI, best practices are changing daily, and new techniques are constantly emerging. To make keeping up with the state of the art easier, DataStax is rolling this and other enhancements into , our production-ready RAG library leveraging LangChain and LlamaIndex. Our goal is to provide developers with a consistent library for RAG applications that puts them in control of the step-up to new functionality. Instead of having to keep up with the myriad changes in techniques and libraries, you have a single stream, so you can focus on building your application. You can use RAGStack today to incorporate best practices for LangChain and LlamaIndex out of the box; advances like ColBERT will come to RAGstack in upcoming releases. RAGStack By Jonathan Ellis, DataStax Also appears . here

How ColBERT Helps Developers Overcome the Limits of RAG

কিভাবে ColBERT ডেভেলপারদের RAG এর সীমা অতিক্রম করতে সাহায্য করে

Wie ColBERT Entwicklern hilft, die Grenzen von RAG zu überwinden

Cómo ColBERT ayuda a los desarrolladores a superar los límites de RAG

Comment ColBERT aide les développeurs à surmonter les limites de RAG

कैसे ColBERT डेवलपर्स को RAG की सीमाओं को पार करने में मदद करता है

ColBERT が開発者による RAG の限界の克服にどのように役立つか

ColBERT가 개발자가 RAG의 한계를 극복하도록 돕는 방법

Como ColBERT ajuda os desenvolvedores a superar os limites do RAG

Как ColBERT помогает разработчикам преодолеть ограничения RAG

ColBERT, Geliştiricilerin RAG Sınırlarını Aşmasına Nasıl Yardımcı Olur?

ColBERT giúp nhà phát triển vượt qua giới hạn của RAG như thế nào

ColBERT 如何帮助开发人员克服 RAG 的限制

When building a generative AI application that needs to call a large language model (LLM) multiple times to complete a task, a common problem is that repeated queries to the LLM can be both expensive and unpredictable. Large models such as GPT-3.5/4 are incredibly resource-intensive to train and run inference; this is reflected in the API charges as well as occasional disruptions to service. ChatGPT was originally released as a research preview and wasn’t intended to be used for production applications. However, its usefulness across a vast array of applications is indisputable, so has exploded. interest in LLMs Since the inception of ChatGPT, users have been looking for ways around the lack of privacy and the inability to control uptime or inference settings when using GPT. This has led to the popularity of free, public models such as Meta’s and later the creation of quantized and lower-parameter versions of Llama that can run on consumer hardware. These public models are capable of providing much of the same functionality as GPT for much less computing power, though at the cost of fewer parameters and fewer verbose outputs. Llama 2 If your application doesn’t necessarily depend on processing excessively large contexts or producing verbose outputs, then hosting your own inference on instances you control can be a more cost-effective option. And when it comes to real-world applications of (RAG), the cost differences may become even more significant. retrieval augmented generation I’ll demonstrate a simple method for combining vector stores, lexical search, and prompt engineering to perform accurate RAG on commodity hardware. Using this method, you can both reduce the complexity of large volumes of information and make running generative AI applications more accurate, efficient, and cost-effective at scale. By using RAG on specific stores of information, you can gain the ability to eliminate hallucinations and create effective and knowledgeable agents from any source material without needing to pay for third-party APIs. To get started, you will need either a DataStax Enterprise 7 instance or to store the vectors and text data, as well as an LLM and a sentence transformer model to generate responses and encode your data with vectors. Depending on the complexity of your data or the user prompts, you may also consider combining this with a DataStax Enterprise 6.8 database that can perform searches to match wider ranges of data, which is what I used in this example. DataStax is continually working on improvements to enable all of these operations with a single database, but for now, I use two databases. DataStax Astra DB Solr Solving for Hallucinations Regardless of which LLM you choose, they all still suffer from . For now, that limitation needs to be resolved by feeding truthful information into the context of prompts to the LLM, otherwise known as RAG. The method by which you locate your information and transform it for the prompts completely depends on your data model, but you can find more pertinent information in a more efficient way by using vector databases. hallucinations Say, for instance, that you have a collection of e-books on a subject you’d like to explore, such as how to play . Under normal circumstances, it would take years to read through the supporting literature and gain enough gameplay experience to reach an expert level. Warhammer 40,000 A targeted question such as, “What can you tell me about Morvenn Vahl of Adepta Sororitas?” would be best answered by a veteran player or any employee at a Warhammer store. While ChatGPT can answer many questions about the game, it unfortunately has no training data that covers this particular character: Compare this to a Llama 2 13B parameter LLM hosted on a consumer workstation with an Nvidia RTX A4000 graphics card. Similarly, the model can demonstrate basic knowledge of the Warhammer universe, but because of the tuning, the model doesn’t care that the character isn’t found and provides instead a best-effort hallucination: If you want to build a chatbot that can help both newcomers and veterans play Warhammer 40,000, then these outputs are unacceptable. To be an effective game guide, the chatbot needs to know the rules of the game, the rules for each unit, some bits of the lore, and some strategy and commentary. Luckily, all of that information on the 10th-edition rules is available for free from Games Workshop and fan websites, and all you need to do is make it searchable to your chatbot app. Compare this to the same 13B Llama model, where with RAG, it is asked to compare a couple of sources on Morvenn Vahl and devise a relevant answer based on the user prompt. This time, the chatbot has access to a database full of all the public information on how to play Warhammer 40,000, 10th Edition: search database and a vector What a difference! Not only does it find pertinent information on this niche character, but it also keeps its outputs in line with the context of how to play the game with the 10th-edition rules. The hardest part in all of this is performing an effective search to find the relevant pages to feed into the LLM. This is where vector databases can be particularly useful. Applying vectors In this example, we’ll use DSE 7 and DSE 6.8 running in Docker instances to satisfy the database requirements of the chatbot application, which needs to be able to compare vectors and perform lexical searches. DSE 7 and Astra DB have introduced the ability to store vectors and perform vector searches, as well as filtering by text matches. We only need to search a few dozen books for this example, so running DSE instances in Docker will be sufficient for most consumer hardware. Using vectors in your databases will help to find documents that are similar to a given query, or they can be used to compare results retrieved from another search. This can help you to overcome the limitations of lexical search and to improve the effectiveness of data models. For instance, something like e-book PDFs can benefit from being encoded with sentence transformers like , and the vectors can be used to run a similarity comparison between a query and a given source. In this case, a sentence transformer model is used to create embeddings of a page’s text in an e-book, and this can enable you to compare to the user’s prompt to figure out if a result is relevant to the query. Relevant pages should contain one or more instances of terms that are similar to the user’s query and result in better similarity scores from the model’s standpoint. miniLM That said, the vectors are best applied as a supplement to an existing lexical search model. If you search by vectors only, then you might end up unexpectedly retrieving unrelated documents and providing them as context where they don’t apply. In this example, the query “What can you tell me about Morvenn Vahl of Adepta Sororitas?” can be transformed by an LLM to a set of simple search terms: Morvenn, Vahl, Adepta, Sororitas The first step in finding relevant documents would be to search for documents that contain those basic terms. This can be done by first filtering for text matches in the database to find keywords in the page text matching such a query. The reason for using an LLM to generate keywords is to provide a wider range of possible keywords to search, as it often attempts to add more keywords that are related but are not in the text of the original prompt. Be careful with this, however, as LLMs can also generate special characters and odd sequences that you will need to sanitize. Once you have at least one result, you can vectorize the user’s query and compare it to the vectors of the lexical search, creating scores of how relevant each result is. This allows you to check the search results for accuracy to the query and set a threshold for rejecting unrelated results when it comes to finally presenting your results to the LLM. In this case, the first step should match pages that specifically show Morvenn Vahl’s index card or gameplay mechanics because those describe the character’s unit in terms of how it plays in the game. If the page meets a certain relevance threshold to the user query, determined by the application, then it gets summarized and placed in a list of results. Finally, the search results can be compiled into a list and fed back to the LLM, where it is asked to use the most relevant contexts to answer the original query. Here is a visualization of the flow: As you can see, the LLM gets called quite frequently for this flow. The LLM is responsible for transforming the user prompt into keywords, summarizing applicable results, and choosing which context best answers a query. Each source to check adds another LLM invocation, which can be quite expensive when making queries to GPT. But if you already have the information you need and just want to summarize it or transform it, then you may not need to use such a large model. In fact, switching to smaller models can provide a number of benefits. By using a smaller LLM, you can reduce the computational cost of each query, which can lead to significant savings over time. This can also result in faster response times for your users, which can improve their overall experience. In this example, where RAG is performed using a small LLM and small databases, all hosted on the same GPU instance, it takes about 30 seconds to retrieve 15 sources, analyze them for relevance, and provide a final answer. And the shorter the prompts (sources), the faster the outputs can be returned. Additionally, this method allows for increased security and scalability. With prompt engineering and a pipeline of calls to the LLM, you’re in is accessed and what the users will get in their responses. In terms of resource usage, the example 13B parameter model only consumes a little over 8GB of VRAM and still provides relevant answers. Depending on needs, this shows potential for even running RAG on myriad other platforms, such as user workstations and mobile devices. full control of how the data Controlling the output Prompt engineering is key to making RAG do exactly what you want. You are in control of how the chatbot interprets the data and the context under which it should be thinking. In this example, we want to ensure that the chatbot knows we are specifically seeking Warhammer information, so we can first ask it to help provide supporting context to the user’s query: Query: “<user query>” Give me a minimal, comma-separated list of Warhammer 40K keywords for a search engine. Respond with only the query. Do not use emojis or special characters. Answer: Warhammer 40,000 is full of terms and names that might appear in other, unrelated popular culture, so it is important to set the context of the RAG in the very first query. This context should be something available to select or modify if your application covers multiple contexts, such as if you need to cover multiple editions of the Warhammer game rules or combine them with the official lore books. Note that the user’s query is always encapsulated with quotes for this experiment. This helps the LLM to distinguish between the query that it’s attempting to directly answer and the separate prompt-engineering instructions, which it must not directly answer. The question/answer portion of the prompt can be adjusted to fit a particular context, but mostly, all you need to be able to do is inform the LLM what it should and should not directly respond to and how to respond. In this case, it’s safe to assume that the LLM has a general knowledge of the game universe since the series is reasonably popular and general information is available for free. The output of this first query helps generate some keywords to use in lexical search without us having to build a cheat sheet into our application. The lexical and vector comparisons can then be performed in the background, and a list of results is compiled for review by the LLM. Because the user’s original prompt is never directly answered with inference at the first step, the LLM only transforms what is found in a search and can easily be stopped from answering queries outside of its guard rails or knowledge base. If there are relevant results from the search: Query: “<user query>” Review these search results and use them to answer the query. Result 1 Result 2 etc. Answer: If there are no relevant results from the search: Query: “<user query>” Politely tell me you searched but could not find an answer for the query. Answer to the best of your knowledge instead. Answer: For added security, you can completely reject or redirect the request when it cannot be served. Query: “<user query>” Politely tell me you searched but could not find an answer to the query. Instruct me to reach out to the Customer Support team for assistance instead. Answer: You can even make the outputs lengthier by asking for more details. As long as you can fit your source material within the context window, the LLM can transform it for you. Query: “<user query>” Review these search results and use them to answer the query. Be as detailed as possible and cite the sources. Result 1 Result 2 etc. Answer: Limitations The LLM has a limited context window and will fail to process exceptionally large pages of text. Consider placing limits on row size so your and easier for the LLM to process. For example, cutting pages into chunks of about 1,000 characters seems to work well, and try to avoid feeding more than four or five detailed answers into the prompt. data is more manageable The LLM has no memory of a conversation aside from what you can fit in the context window. It is possible to build a permanent store of conversation data, but it is not possible for an LLM to fit excessively large conversations or detailed context into a prompt; there is an upper limit to what it can transform. This means that no matter what, at a certain point, you will notice that the LLM seems to “forget” certain details even when they are provided as context; this is just an inherent limitation of the tool. It is best to rely on it for short conversations only and focus on transforming small amounts of text at a time to minimize hallucinations. Randomness in the LLM can be a problem. Testing and tuning will be necessary to determine which prompts work best for your data set and to find which model works best for your use case. In my testing with a 13B parameter model, there was a lot of unpredictability regarding which search keywords were generated from the first prompt, especially as the prompt length increased. For best results, stick to shorter prompts. Conclusion In summary, leveraging RAG by combining models allows for more effective finding and sorting of relevant results and generating agent outputs that are far less prone to hallucinations. The smaller the searchable context, the more precise and accurate the responses. Constructing your own custom pipeline of LLM calls offers far more flexibility in tuning responses toward your desired level of accuracy and guard rails. vector and lexical search While it cannot process excessively large amounts of data within the limited context window, it does offer the ability to create effective assistants on limited knowledge bases, as well as run more concurrent agents on the same or lesser hardware than before. This could open up more possibilities for virtual assistants for applications such as tabletop gaming or even cover more complex topics for use by government, legal and accounting firms, scientific research, energy, and more. If you’re ready to get started building, you can . Create your database and start loading your RAG sources today, with no cloud or database Ops experience required. try Astra DB for free By Mario Charnell-Delgado, DataStax Also published . here

Want to Learn Warhammer 40K? Build This Chatbot with Vectors and RAG on Commodity Hardware

Warhammer 40K শিখতে চান? কমোডিটি হার্ডওয়্যারে ভেক্টর এবং RAG দিয়ে এই চ্যাটবট তৈরি করুন

Möchten Sie Warhammer 40K lernen? Erstellen Sie diesen Chatbot mit Vektoren und RAG auf Commodity-Hardware

¿Quieres aprender Warhammer 40K? Cree este chatbot con vectores y RAG en hardware básico

Vous voulez apprendre Warhammer 40K ? Construisez ce chatbot avec des vecteurs et RAG sur du matériel de base

क्या आप वॉरहैमर 40K सीखना चाहते हैं? कमोडिटी हार्डवेयर पर वेक्टर और आरएजी के साथ इस चैटबॉट का निर्माण करें

ウォーハンマー 40K を学びたいですか? Vectors と RAG を使用してコモディティ ハードウェア上でこのチャットボットを構築する

워해머 40K를 배우고 싶으십니까? 상용 하드웨어에서 벡터와 RAG를 사용하여 챗봇 구축

Quer aprender Warhammer 40K? Construa este chatbot com vetores e RAG em hardware de commodities

Хотите изучить Warhammer 40K? Создайте этого чат-бота с помощью Vector и RAG на стандартном оборудовании

Warhammer 40K'yı Öğrenmek İster misiniz? Ticari Donanımda Vektörler ve RAG ile Bu Chatbot'u Oluşturun

Bạn muốn học Warhammer 40K? Xây dựng Chatbot này với Vector và RAG trên phần cứng hàng hóa

想学习战锤40K吗？在商品硬件上使用向量和 RAG 构建此聊天机器人

Developing generative AI applications with (RAG) and vector databases can be challenging. You’ll often need to troubleshoot RAG implementations that rely on vector databases to ensure that relevant context is retrieved; this is critical, because that context is then included in a prompt to a large language model to provide more accurate results. retrieval augmented generation In , we looked at the embedding pipeline, which populates the with embeddings, and considered three areas that can lead to poor results: suboptimal embedding models, inefficient chunking strategies and lack of metadata filtering. the previous article in this series vector database Here, we’ll look at the actual interaction with the LLM and examine a couple common problems that can lead to poor results: imprecise prompts and insufficient generation approaches. Prompt refinement Even the best embedding and chunking strategies can’t replace the need for quality . This involves making the prompts more explicit, specific and aligned with the intended output. Varied prompt formats, lengths and lexical choices should be tested to fine-tune the RAG process. prompt engineering Specifically, there are several things you should consider when building your prompts for RAG applications. These include: When interacting with LLM agents like ChatGPT, they will default to acting like a helpful chatbot. However, you can alter the nature of responses that will be generated by instructing the LLM to act in a particular way. Examples could be things like “you are an attorney evaluating if either party is in breach of an agreement” or “you are a customer service agent for an internet service provider; your job is to help people troubleshoot their internet issues” or whatever makes sense for your particular situation. Tell the LLM its role: : Make it clear to the LLM that you are providing context, and you want the generated response to reflect that context. You can do this by saying something like “your response should take into account the following context” followed by the context. Use examples: In the just-mentioned scenario of asking the LLM to act as an attorney evaluating contracts, you may want to include several examples in the prompt. For instance, you could provide an example of a contract that says payment is due within 30 days, but the buyer didn’t send their payment until 40 days after the contract was signed and therefore violated the agreement. You could provide additional examples of rights to remedy and possible ways to resolve contract violations. Explicitly tell the LLM to use the provided context If your use case requires specific output, you can specify the format that the generated output should adhere to. You can combine this technique along with the above tip to provide examples to make it clear to the LLM both how you want it to respond, as well as the key points of information you’re expecting in the generated response. Specify the output format: : For use cases where reasoning is involved to determine the appropriate response, you can consider using a technique called to help explain the steps that you want the LLM to follow to arrive at a generated response. For example, in the legal contracts case, you may want to walk the LLM through the logical steps that a person would follow to determine if a contract term has been violated. For instance, when dealing with legal contracts you might tell the LLM to first look for clauses that specify payment terms, then identify the amount of time the buyer had to submit payment, then calculate the number of days between the time payment was received and the date the contract was signed. Then, if the payment took longer than the agreed upon time frame, the buyer violated the agreement. Use chain of thought chain of thought Leveraging these techniques to improve your prompt engineering can have a significant impact on the quality of the generated results you’re able to produce in your RAG applications. However, sometimes you will need to use techniques that involve multiple exchanges with the LLM to achieve an acceptable response. Advanced Patterns FLARE Forward-looking active retrieval, or , is an example of a multi-query RAG technique that involves iteratively calling the LLM with custom instructions in your prompt that ask the LLM to provide additional questions about key phrases that would help it generate a better answer. Once the LLM has context with no gaps, it terminates with the final response. The process adds a loop between the LLM and the AI agent (the AI chatbot in the diagram) to facilitate these iterations: FLARE You can see an example of how FLARE works in the . LangChain cookbook example of FLARE RAG Fusion By generating similar queries to the user’s query and retrieving relevant context for both the original query as well as the generated similar queries, we are able to increase the likelihood that we have selected the most useful context to generate accurate results. The process, called “RAG fusion,” looks like this: The key step here is to use a to further refine the ANN results to identify the most relevant context to use to generate a response. reciprocal rank function Conclusion RAG is an approach that’s simple to get started with, but often leaves developers frustrated with unsatisfactory results for unclear reasons. The relevance of outputs in RAG-enabled generative AI applications can be affected by several factors. By meticulously selecting embedding models, formulating chunking strategies and designing prompts, you can significantly elevate the quality and precision of generated responses in systems powered by LLMs like ChatGPT. Hopefully these tips will help you create more useful RAG applications that deliver the experience and value you’re aiming for. . , the only vector database for building production-level AI applications on real-time data Try DataStax Astra DB By Chris Latimer, DataStax Also appears . here

Achieving Relevant LLM Responses By Addressing Common Retrieval Augmented Generation Challenges

সাধারণ পুনরুদ্ধার অগমেন্টেড জেনারেশন চ্যালেঞ্জ মোকাবেলা করে প্রাসঙ্গিক এলএলএম প্রতিক্রিয়া অর্জন করা

Erreichen relevanter LLM-Antworten durch Bewältigung häufiger Herausforderungen im Bereich Retrieval Augmented Generation

Lograr respuestas relevantes de LLM abordando los desafíos comunes de la generación aumentada de recuperación

Obtenir des réponses LLM pertinentes en relevant les défis courants de génération augmentée de récupération

सामान्य पुनर्प्राप्ति संवर्धित पीढ़ी चुनौतियों का समाधान करके प्रासंगिक एलएलएम प्रतिक्रियाएं प्राप्त करना

一般的な検索拡張生成の課題に対処することで、関連する LLM 応答を実現する

일반적인 검색 증강 생성 문제를 해결하여 관련 LLM 응답 달성

Alcançando respostas LLM relevantes abordando desafios comuns de geração aumentada de recuperação

Достижение соответствующих ответов LLM путем решения общих проблем поиска дополненной генерации

Ortak Erişim Artırılmış Üretim Zorluklarını Ele Alarak İlgili Yüksek Lisans Yanıtlarına Ulaşmak

Đạt được các phản hồi LLM có liên quan bằng cách giải quyết các thách thức chung về thế hệ tăng cường truy xuất

通过解决常见的检索增强生成挑战来获得相关的法学硕士学位

In the first installment of this two-part series, we look at how suboptimal embedding models, inefficient chunking strategies, and a lack of metadata filtering can make it hard to get relevant responses from your LLM. Here’s how to surmount these challenges. Building generative AI applications that use (RAG) can pose a host of challenges. Let’s look at troubleshooting RAG implementations that rely on vector databases to retrieve relevant context that’s then included in a prompt to a large language model to provide more relevant results. retrieval augmented generation We will break this process down into two main parts. The first, which we’ll address in this first article in the series, is the embedding pipeline, which populates the with embeddings: vector database Here, we’ll consider three main areas that can lead to poor results: suboptimal embedding models, inefficient chunking strategies, and a lack of metadata filtering. (In the upcoming article, we’ll look at the actual interaction with the LLM and examine some common problems that crop up there and can lead to poor results.) Selecting an Appropriate Embedding Model Your choice of an embedding model will have a significant impact on the overall relevance and usability of your RAG application. As such, it requires a nuanced understanding of each model’s capabilities and an analysis of how those capabilities align with your application’s requirements. If you’re relatively new to RAG and embeddings in general, one of the best resources you should be aware of is the . We focus on retrieval use cases in this post, but embeddings can, of course, be used for many other applications, including classification, clustering, and summarization. MTEB (Massive Text Embedding Benchmark) embedding leaderboard The leaderboard can help you identify the models that will perform best for your specific use case. One of the most common reasons for poor RAG performance is that developers new to this space do a Google search to find examples of embedding generation. They often find samples that use embedding models such as Word2Vec, sBERT, and RoBERTa that are poor choices for retrieval use cases. If you found this article because you’re debugging poor relevance results and you used something like sBERT to generate your embeddings, then we’ve likely identified the cause of your relevance problems. If so, the next question you will likely have is which embedding models you can use to improve your similarity search results. Without knowing the particulars of your use case, the three we would recommend are: text-embedding-ada-002 (Ada v2) is probably the most common starting point for most RAG applications simply because so many developers start off with Open AI’s APIs. Ada v2 performs admirably in retrieval use cases and was built to handle different types of content, including text and code. Ada v2 from OpenAI With a maximum input sequence length of up to 8,192 tokens, it also allows you to create embeddings for much longer pieces of text than alternative models. This is both a blessing and a curse. Having a large sequence size simplifies the process of creating embeddings for more of your text content, and it allows the embedding model to identify relationships across words and sentences in a bigger body of text. However, this also results in similarity searches that can become more fuzzy when comparing the similarity of two long documents when what you’re looking for is relevant chunks of context to facilitate the generation process. There are two big drawbacks of Ada v2. The first is that it can’t be run locally. You must use OpenAI’s API to create the embedding. This can not only introduce bottlenecks for cases where you want to create embeddings for many pieces of content, but it also adds a cost of $0.0001 per 1,000  tokens. The second is that the embeddings created from the Open AI model are 1,536 dimensions each. If you are using a cloud vector database, this can considerably add to your vector storage costs. You want a simple solution that only requires an API call, you potentially need to vectorize large documents, and cost is not an issue. When to choose: jina-embeddings-v2 (Jina v2) Jina v2 is a new open-source embedding model that gives you the same 8,000 input sequence support as Ada v2 and actually scores slightly better in retrieval use cases. Jina v2 provides an antidote to the problems of Ada v2. It’s under Apache License 2.0 and can be run locally, which, of course, is also a drawback if you aren’t looking to run your own code to do this. It also produces an embedding vector with half the dimensions of Ada v2. open source So not only do you get slightly better retrieval performance on benchmark use cases, but you also get those improved results with lower storage and compute requirements from a vector database perspective. You want to use an open-source solution and potentially need to vectorize large documents and are comfortable running embedding pipelines locally. You want to reduce vector database costs with lower-dimension embeddings. When to choose: bge-large-en-v1.5 is open-sourced under the MIT license and is currently the top-ranked embedding model on the MTEB leaderboard for retrieval use cases. With a smaller input sequence, it will require you to give more thought to your chunking strategy but ultimately provides the best all-around performance for retrieval use cases. bge-large-en-v1.5 When to choose: You want to use an open-source solution and are willing to spend more time on chunking strategies to stay within the input size limitations. You are comfortable running embedding pipelines locally. You want the best-performing embedding model for retrieval use cases. While outside the scope of this article, you might want to dig deeper into the 15 benchmarks in the MTEB leaderboard to identify the one that most closely resembles your specific situation. While there are definitely patterns in terms of how well various embedding models perform across the different benchmarks, there are often specific models that stand out in each. If you need to further refine your embedding selection, this is a possible area of further investigation. Optimizing Your Chunking Strategy The segmentation or, “chunking” of input text, is a pivotal factor that significantly influences the relevance and accuracy of the generated output. Various chunking strategies offer unique advantages and are suited for specific types of tasks. Here, we delve into these methodologies and provide guidelines for their application, incorporating some key considerations: Fixed-length chunking - Unless your content itself is highly structured and of fixed length, you usually want to rely on a more useful chunking strategy like the ones that follow. When to use - While very simple to implement, this chunking strategy is generally going to lead to poor results in RAG applications. Technical consideration If you are using a fixed-length strategy with your RAG application and having trouble retrieving relevant context, you should consider switching to a different chunking approach. Additional insight Sentence-level chunking - This strategy is effective when each sentence in the input text is rich in meaning and context. It allows the model to concentrate on the intricacies within each sentence, thereby generating more coherent and contextually relevant responses. You’ll rarely rely on sentence-level chunking for RAG use cases. When to use Sentence-level chunking often involves tokenization based on sentence boundaries, which can be achieved using natural language processing (NLP) libraries. Technical consideration - Sentence-level chunking can be particularly useful when you’re searching for specific statements, such as in a transcript of a meeting where you’re trying to find semantically similar statements to a given piece of text. Additional insight - Paragraph-level chunking - Employ this strategy when the input text is organized into distinct sections or paragraphs, each encapsulating a separate idea or topic. This enables the model to focus on the relevant information within each paragraph. When to use Identifying paragraph boundaries usually involves detecting newline characters or other delimiters that signify the end of a paragraph. Technical consideration - Paragraph-level chunking can be useful when you have documents that cover many different aspects of the same topic. For example, a page of product documentation might introduce a product feature, explain when to use it, talk about how to configure it, and give examples of different configurations. Additional insight - Using paragraph-level chunking can help you identify the most relevant part of the document to provide to the LLM as context. Content-aware chunking Opt for this strategy when the relevance of specific sections within the text is paramount. For instance, in legal documents, segmenting the text based on clauses or sections can yield more context-specific responses. When to use - This approach may require advanced NLP techniques to understand the semantic boundaries within the text. Technical consideration - Content-aware chunking is especially useful when dealing with structured or semi-structured data, as specific chunks can be combined with metadata filtering for more precise retrieval. Additional insight - For example, in a legal document, you might want to extract all warranty or indemnification clauses, and when you store embeddings for chunks in a vector database, you can use metadata to make it easier to search for content of a given type when building a RAG use case. Recursive chunking Recursive chunking divides data into smaller and smaller pieces, using a hierarchical approach. For example, when chunking a text document, you might divide the text into paragraphs first, then into sentences, and finally, words. When to use - Once the data has been divided into the first set of chunks, you can then recursively apply the chunking process to each of the smaller chunks, repeating until you reach the smallest chunk size you’re interested in. Implementing recursive chunking might involve a multilevel parsing strategy where chunks are further divided into sub-chunks based on additional criteria. If you’re using , its recursive implementation is a bit simpler than what is described here. Technical consideration - LangChain This approach enables the model to understand the context at multiple levels, from high-level themes to detailed nuances, making it particularly useful for complex documents like academic papers, technical manuals, or legal contracts. This brings flexibility benefits since similarity searches can identify similar text for both broader and shorter queries. Additional insight - However, this also means that there is a possibility that similar chunks from the same source document could end up being overrepresented in similarity searches as well, especially if you opt for a longer overlap between chunks in your text splitter configuration. As a general approach, before you try chunking up a big corpus and vectorizing it, you should consider doing some ad-hoc experimentation with your data. Manually inspect the documents you would want to retrieve for a given query, identify the chunks that represent the ideal context you would want to provide the LLM, and then experiment with chunking strategies to see which one gives you the chunks you feel would be most relevant for the LLM to have. Context Window Consideration The available context window of an LLM is an important factor in selecting a chunking strategy. If the context window is small, you’ll need to be more selective in the chunks you feed into the model to ensure that the most relevant information is included. Conversely, a larger context window allows for more flexibility, enabling the inclusion of additional context that may enhance the model’s output, even if not all of it is strictly necessary. By experimenting with these chunking strategies and taking these considerations into account, you can evaluate their impact on the relevance of the generated outputs. The key is to align the chosen strategy with the specific requirements of your RAG application, preserve the semantic integrity of the input, and offer a comprehensive understanding of the context. This will enable you to find the right chunking process for optimal performance. Metadata Filtering As the number of embeddings in your search index grows, approximate nearest neighbors (ANN) become less helpful when looking for relevant context to include in your prompts. Let’s say you have indexed embeddings for 200 articles in your knowledge base. If you can identify the top nearest neighbor with an accuracy of 1%, you are likely to find pretty relevant results because 1% represents the top two articles out of those 200, and you’re going to get one of those two. Now, consider a search index containing every article on Wikipedia. That would amount to approximately 6.7 million articles. If your nearest neighbor is in the top 1% of most similar articles, that means that you’re getting one of the 67,000 most similar articles. With a corpus like Wikipedia, this means that you could still end up being very far off the mark. Metadata filtering gives you a way to narrow down the pieces of content by first filtering the documents and then applying the nearest neighbor algorithm. In cases where you're dealing with a large number of possible matches, this initial pre-filtering can help you narrow the possible options before retrieving the nearest neighbors. Coming up next, we’ll dive into the interaction with the LLM and examine some common problems that can lead to poor results. . Try , the only vector database for building production-level AI applications on real-time data DataStax Astra DB By Chris Latimer, DataStax Also published here

Need More Relevant LLM Responses? Address These Retrieval Augmented Generation Challenges

আরো প্রাসঙ্গিক LLM প্রতিক্রিয়া প্রয়োজন? এই পুনরুদ্ধার অগমেন্টেড জেনারেশন চ্যালেঞ্জের সমাধান করুন (পর্ব 1)

Benötigen Sie relevantere LLM-Antworten? Bewältigen Sie diese Herausforderungen im Bereich Retrieval Augmented Generation (Teil 1)

¿Necesita respuestas de LLM más relevantes? Aborde estos desafíos de la generación aumentada de recuperación (Parte 1)

Besoin de réponses LLM plus pertinentes ? Relever ces défis de génération augmentée de récupération (partie 1)

अधिक प्रासंगिक एलएलएम प्रतिक्रियाओं की आवश्यकता है? इन पुनर्प्राप्ति संवर्धित पीढ़ी चुनौतियों का समाधान करें (भाग 1)

さらに関連性のある LLM 応答が必要ですか?これらの検索拡張生成の課題に対処する (パート 1)

더 많은 관련 LLM 응답이 필요하십니까? 이러한 검색 증강 세대 문제 해결(1부)

Precisa de respostas LLM mais relevantes? Aborde esses desafios de geração aumentada de recuperação (Parte 1)

Нужны более релевантные ответы LLM? Решите эти проблемы поиска дополненной генерации (часть 1)

Daha İlgili LLM Yanıtlarına mı ihtiyacınız var? Bu Erişim Artırılmış Nesil Zorluklarının Ele Alınması (Bölüm 1)

Cần thêm phản hồi LLM có liên quan? Giải quyết những thách thức về thế hệ tăng cường truy xuất này (Phần 1)

需要更多相关的法学硕士回复？应对这些检索增强生成挑战（第 1 部分）

Stories, News & Mentions about DATASTAX

Datastax’s Stories on HackerNoon

Datastax's News & Mentions