Hello, readers; I would like to extend a warm welcome to all of you for reading my blog post. This particular post is my second one, focusing on vector databases. In my previous , I covered the concepts of vectors, vector databases, and vector search. I also provided a practical demonstration of the FAISS vector store using Python. Similarly, in this article, we will delve into another popular vector store known as ChromaDB. blog post Introduction to ChromaDB In this part of the article, we will explore what ChromaDB is, its features, and how it works in real-time. What is ChromaDB? is a vector store that is open-source and is utilized for the storage and retrieval of vector embeddings. Its primary purpose is to store embeddings and associated metadata for future use by extensive language models. Furthermore, it can also be employed for semantic search engines that operate on text data. Chroma DB What are the features of ChromaDB? ChromaDB offers various storage options, such as DuckDB for standalone use or ClickHouse for scalability. It provides Python and JavaScript/TypeScript SDKs and emphasizes simplicity, speed, and analysis capabilities. ChromaDB can store vectors with additional metadata and allows for filtering during the query search on the vector database. Chroma DB provides various options for storing vector embeddings. The options include storing the vector database , where it is flushed when the RAM is refreshed. in-memory Storing it on the and loading it into memory when needed. local file system Another option is to host the database on a machine, allowing to make requests to the server for each query. server clients How does ChromaDB work? The initial step in vector databases is to create a collection, which is similar to a table in a relational database. By default, ChromaDB utilizes the model for generating embeddings. The embedding model can be customized according to the user's requirements. all-MiniLM-L6-V2 Documents can be added to the collection, and if they are in text format, ChromaDB will automatically convert them into embeddings based on the specified embedding model. It is optional to include meta information when adding a document, but a unique document ID must be provided for identification purposes. After creating and populating the collection with documents, we can begin querying the vector store to retrieve similar documents. The process of filtering the documents while querying is referred to as , and it is also available as an option in ChromaDB. meta-filtering In the next part, we will use Chroma and all-MiniLM-L6-V2 to create our own vector DB. Getting started with ChromaDB In this section, we will create a vector store, add collections, add text to the collection, and perform a query search with and without meta-filtering using in-memory ChromaDB. As the first step, we will try installing the ChromaDB package. !pip3 install chromadb Once the package is installed, try importing the required packages into your working notebook. import chromadb
import pandas as pd
import numpy as np After importing the packages, we will now have a look into our dataset, which we feed to our vector store for training. passage_data = pd.read_csv("/Users/Python_Programs/Text_Similarity/Stanford_Edited.csv")
passage_data For this tutorial, I have selected the Stanford dataset. The dataset has 37,792 rows of data and 3 columns named: Id: Unique id to represent each document in the data store. Content: The content of each document. Language: The language in which the content is written (for filtering purpose). For meta-filtering, we will have the language column in the dataset with values en_US as English and ru_RU as Russian. The data count in each of these labels is as follows: Having explored the dataset, now it is time to prepare the data in the format that the ChromaDB accepts. To add a set of documents to the Chroma index, documents - list of documents(text) metadatas - list of dictionary that has unique ids to represent the documents (ids should be in String format) documents = passage_data['Content'].tolist()
metadatas = []
ids = []

for idx in range(len(passage_data)):
    metadatas.append({"Language": passage_data['Language'][idx]})
    ids.append(str(passage_data['Id'][idx])) The dataset and input documents with metadata and IDs are prepared. We will create a client connection to ChromaDB. client = chromadb.Client() Once the client is initialized, we create a new collection to hold the vectors(data) as an index. collection = client.create_collection("Stanford_Collection") It is time to add the documents to the index. collection.add(
    documents=documents,
    metadatas=metadatas,
    ids=ids
) There are two approaches to querying the index: querying without filters and querying with filters (known as meta-filtering). We will now attempt both methods. Attempting a query without any filters. Attempting a query with filters (meta-filtering). Now, we will test both approaches. # 1. query without any filters
results = collection.query(
    query_texts=["University of Architecture"],
    n_results=3,
)
print(results) The output displays three documents that have similarities to the query document ("University of Architecture"). It includes the document IDs, distances, and metadata of these documents. Now, we will try a query with a language filter. results = collection.query(
    query_texts=["University of Architecture"],
    n_results=3,
    where={"Language": "en_US"}, # optional filter
)
print(results) The output displays three documents that are in the English language for the given query. This is how meta-filtering works. Few other functionalities Having seen the main functionalities of ChromaDB, we will now explore a few other functionalities in short. To update a document in the Chroma Index: collection.update(
    ids=["2"],
    documents=["This document contains information about medical college"],
    metadatas=[{"Language": "en_US"}],
) There is slight difference between update() and upsert() in ChromaDB. In short, if you are trying to update a document which does not exist in the Index: update() raises a exception, whereas upsert() tries to insert the document as new vector into the index. To delete a document from the Chroma Index: collection.delete(ids = ['1']) To get the count of documents present in an index, we use the following: print(collection.count()) To get the documents present in the collection, we use: print(collection.get()) To modify the name of the existing collection: collection.modify(name="new_collection_name") To delete or drop an Index in ChromeDB, we use: client.delete_collection(name="new_collection_name") Conclusion This article has presented an introduction to Chroma, a Vector Database that is open source. We have discussed the process of creating collections in Chroma and adding documents to them. Furthermore, we have examined how Chroma DB generates vector embeddings from the data. Lastly, we have gained knowledge on how to retrieve relevant information from a specific collection in the Vector Store based on a given query. I hope this blog post has aided you in exploring another vector database and understanding the concepts clearly. I look forward to seeing you on my next blog. Until then, happy learning 👍🏻.

The code in this story is for educational purposes. The readers are solely responsible for whatever they build with it.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

大規模言語モデル: トランスフォーマーの探究 - パート 2

ベクトル データベース - Python のベクトル検索とラングチェーン パッケージの基礎

Read My Stories

Connect on LinkedIn

このオーディオは、ストーリーの元の言語で制作されています。

ベクターデータベース: ChromaDB の概要など

About Author

コメント

ラベル

この記事は

Related Stories

大規模言語モデル: トランスフォーマーの探究 - パート 2

AWS S3 から MinIO への復帰に必要なすべての情報

Elasticsearch の更新、挿入、削除: 仕組みと制限を理解する

LLM の長所、短所、リスクの分析

Aptible による DevOps ライティングコンテスト: 第 2 ラウンドの勝者が発表されました!

SQL ウィンドウ関数とその機能を理解するための初心者ガイド

大規模言語モデル: トランスフォーマーの探究 - パート 2

AWS S3 から MinIO への復帰に必要なすべての情報

Elasticsearch の更新、挿入、削除: 仕組みと制限を理解する

LLM の長所、短所、リスクの分析

Aptible による DevOps ライティングコンテスト: 第 2 ラウンドの勝者が発表されました!

SQL ウィンドウ関数とその機能を理解するための初心者ガイド

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps

ベクター データベース: ChromaDB の概要など

About Author

コメント

ラベル

この記事は

Related Stories

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps

ベクターデータベース: ChromaDB の概要など