Hello, readers; I would like to extend a warm welcome to all of you for reading my blog post. This particular post is my second one, focusing on vector databases. In my previous , I covered the concepts of vectors, vector databases, and vector search. I also provided a practical demonstration of the FAISS vector store using Python. Similarly, in this article, we will delve into another popular vector store known as ChromaDB. blog post Introduction to ChromaDB In this part of the article, we will explore what ChromaDB is, its features, and how it works in real-time. What is ChromaDB? is a vector store that is open-source and is utilized for the storage and retrieval of vector embeddings. Its primary purpose is to store embeddings and associated metadata for future use by extensive language models. Furthermore, it can also be employed for semantic search engines that operate on text data. Chroma DB What are the features of ChromaDB? ChromaDB offers various storage options, such as DuckDB for standalone use or ClickHouse for scalability. It provides Python and JavaScript/TypeScript SDKs and emphasizes simplicity, speed, and analysis capabilities. ChromaDB can store vectors with additional metadata and allows for filtering during the query search on the vector database. Chroma DB provides various options for storing vector embeddings. The options include storing the vector database , where it is flushed when the RAM is refreshed. in-memory Storing it on the and loading it into memory when needed. local file system Another option is to host the database on a machine, allowing to make requests to the server for each query. server clients How does ChromaDB work? The initial step in vector databases is to create a collection, which is similar to a table in a relational database. By default, ChromaDB utilizes the model for generating embeddings. The embedding model can be customized according to the user's requirements. all-MiniLM-L6-V2 Documents can be added to the collection, and if they are in text format, ChromaDB will automatically convert them into embeddings based on the specified embedding model. It is optional to include meta information when adding a document, but a unique document ID must be provided for identification purposes. After creating and populating the collection with documents, we can begin querying the vector store to retrieve similar documents. The process of filtering the documents while querying is referred to as , and it is also available as an option in ChromaDB. meta-filtering In the next part, we will use Chroma and all-MiniLM-L6-V2 to create our own vector DB. Getting started with ChromaDB In this section, we will create a vector store, add collections, add text to the collection, and perform a query search with and without meta-filtering using in-memory ChromaDB. As the first step, we will try installing the ChromaDB package. !pip3 install chromadb Once the package is installed, try importing the required packages into your working notebook. import chromadb
import pandas as pd
import numpy as np After importing the packages, we will now have a look into our dataset, which we feed to our vector store for training. passage_data = pd.read_csv("/Users/Python_Programs/Text_Similarity/Stanford_Edited.csv")
passage_data For this tutorial, I have selected the Stanford dataset. The dataset has 37,792 rows of data and 3 columns named: Id: Unique id to represent each document in the data store. Content: The content of each document. Language: The language in which the content is written (for filtering purpose). For meta-filtering, we will have the language column in the dataset with values en_US as English and ru_RU as Russian. The data count in each of these labels is as follows: Having explored the dataset, now it is time to prepare the data in the format that the ChromaDB accepts. To add a set of documents to the Chroma index, documents - list of documents(text) metadatas - list of dictionary that has unique ids to represent the documents (ids should be in String format) documents = passage_data['Content'].tolist()
metadatas = []
ids = []

for idx in range(len(passage_data)):
    metadatas.append({"Language": passage_data['Language'][idx]})
    ids.append(str(passage_data['Id'][idx])) The dataset and input documents with metadata and IDs are prepared. We will create a client connection to ChromaDB. client = chromadb.Client() Once the client is initialized, we create a new collection to hold the vectors(data) as an index. collection = client.create_collection("Stanford_Collection") It is time to add the documents to the index. collection.add(
    documents=documents,
    metadatas=metadatas,
    ids=ids
) There are two approaches to querying the index: querying without filters and querying with filters (known as meta-filtering). We will now attempt both methods. Attempting a query without any filters. Attempting a query with filters (meta-filtering). Now, we will test both approaches. # 1. query without any filters
results = collection.query(
    query_texts=["University of Architecture"],
    n_results=3,
)
print(results) The output displays three documents that have similarities to the query document ("University of Architecture"). It includes the document IDs, distances, and metadata of these documents. Now, we will try a query with a language filter. results = collection.query(
    query_texts=["University of Architecture"],
    n_results=3,
    where={"Language": "en_US"}, # optional filter
)
print(results) The output displays three documents that are in the English language for the given query. This is how meta-filtering works. Few other functionalities Having seen the main functionalities of ChromaDB, we will now explore a few other functionalities in short. To update a document in the Chroma Index: collection.update(
    ids=["2"],
    documents=["This document contains information about medical college"],
    metadatas=[{"Language": "en_US"}],
) There is slight difference between update() and upsert() in ChromaDB. In short, if you are trying to update a document which does not exist in the Index: update() raises a exception, whereas upsert() tries to insert the document as new vector into the index. To delete a document from the Chroma Index: collection.delete(ids = ['1']) To get the count of documents present in an index, we use the following: print(collection.count()) To get the documents present in the collection, we use: print(collection.get()) To modify the name of the existing collection: collection.modify(name="new_collection_name") To delete or drop an Index in ChromeDB, we use: client.delete_collection(name="new_collection_name") Conclusion This article has presented an introduction to Chroma, a Vector Database that is open source. We have discussed the process of creating collections in Chroma and adding documents to them. Furthermore, we have examined how Chroma DB generates vector embeddings from the data. Lastly, we have gained knowledge on how to retrieve relevant information from a specific collection in the Vector Store based on a given query. I hope this blog post has aided you in exploring another vector database and understanding the concepts clearly. I look forward to seeing you on my next blog. Until then, happy learning 👍🏻.

The code in this story is for educational purposes. The readers are solely responsible for whatever they build with it.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Mô hình ngôn ngữ lớn: Khám phá Transformers - Phần 2

Cơ sở dữ liệu Vector - Khái niệm cơ bản về Tìm kiếm Vector và Gói Langchain trong Python

Read My Stories

Connect on LinkedIn

Nghe bài viết này bằng Tiếng Anh, đọc bởi robot thông minh của HackerNoon

Cơ sở dữ liệu vectơ: Bắt đầu với ChromaDB và hơn thế nữa

About Author

BÌNH LUẬN

chuyên mục

BÀI VIẾT NÀY CŨNG CÓ MẶT TẠI

Related Stories

Mô hình ngôn ngữ lớn: Khám phá Transformers - Phần 2

Tất cả những điều bạn cần biết để hồi hương từ AWS S3 về MinIO

Cập nhật, chèn, xóa Elaticsearch: Tìm hiểu cách chúng hoạt động và những hạn chế của chúng

Phân tích ưu, nhược điểm và rủi ro của LLM

Cuộc thi viết DevOps của Aptible: Công bố người chiến thắng vòng 2!

Hướng dẫn cho người mới bắt đầu để hiểu các chức năng của cửa sổ SQL và khả năng của chúng

Mô hình ngôn ngữ lớn: Khám phá Transformers - Phần 2

Tất cả những điều bạn cần biết để hồi hương từ AWS S3 về MinIO

Cập nhật, chèn, xóa Elaticsearch: Tìm hiểu cách chúng hoạt động và những hạn chế của chúng

Phân tích ưu, nhược điểm và rủi ro của LLM

Cuộc thi viết DevOps của Aptible: Công bố người chiến thắng vòng 2!

Hướng dẫn cho người mới bắt đầu để hiểu các chức năng của cửa sổ SQL và khả năng của chúng

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps