Vector databases lie at the heart of modern data-driven applications, enabling efficient and accurate similarity search across vast datasets. These databases handle high-dimensional vectors, capturing complex relationships and patterns within the data. In this article, we elucidate the inner workings of prominent vector databases such as Chroma, FAISS, and Pinecone.
Handling large input files for answer retrieval often incurs high costs in terms of search and querying. Additionally, at times Large Language Models(LLMs) have tendency to hallucinate when the context is unknown and a question is asked by the user without much background.
Did you know: “Set” word has the highest number of definitions in English dictionary. If a user asks question based on word set, the answer can be anything under the hood. But if there is a context associated, the answer retrieved is relevant to the user’s question. This is the exact scenario where vector databases can be helpful.
What is the main function of a vector Database?
-Efficient Storage and Querying: Vector databases are efficient in storage and have query capabilities for the vector embeddings.
-Handling High-Dimensional Data: Vector databases provide the ability to store and retrieve vectors as high-dimensional points.
-Operationalizing Embedding Models: Vectors generated by embeddings can be indexed into a vector database, a neighboring vector can be queried from the database.
-Semantic or Contextual Relevance: The primary benefit of a vector database is to precisely locate and retrieve data according to their vector proximity or similarity. This allows for the searches through context rather than just exact match.
-Useful Generative AI Models: Vector databases can provide an external knowledge base for generative AI chatbots.
Let’s look into couple of key differentiating factors of some of the most frequently used vector databases.
FAISS (Facebook AI Similarity Search)
FAISS a high performance library created and optimized for dense vector similarity search and grouping. FAISS is optimized for memory & accuracy trade off and has sophisticated techniques like quantization and partitioning.
There are several indexes such as IVF, HNSW, Locality Sensitive Hashing(LSH) etc. in FAISS which makes the search fast & efficient. Let’s look at some of them below.
-Inverted File Index (IVF):
The inverted file index is a data structure that enables efficient search in large datasets.
Inverted File divides the entire dataset into partitions. Each partition is associated with a centroid, and every vector in the dataset is assigned to a partition corresponding to its nearest centroid.
The elements within inverted lists are encoded vectors. While searching, FAISS scans the elements sequentially and returns the top-k smallest distances seen so far.
-Locality Sensitive Hashing (LSH):
When dataset of vectors are fed, LSH hashes these vectors into buckets based on their similarity. Similar vectors end up in the same bucket, allowing for efficient search and retrieval. LSH aims at faster and efficient search with comparatively lesser computational load.
CHROMA DB:
Chroma DB offers self-hosted server option where vector embeddings can be stored and semantic searches are enabled over text data. It provides SDKs for Python, JavaScript etc.
Like other vector DBs, Chroma DB also converts the query into embeddings using embedding model, performs similarity search & retrieves relevant data based on similarity (Cosine similarity or Euclidean distance etc).
After installation, setting up the Chroma client and starting to create and manage embedding, document, and metadata collections is pretty straight forward.
Croma DB also provides an option to import pre-generated embeddings straight into your collection. This minimizes time & effort to create embeddings all over again.
PINECONE:
Pinecone Vector DB is the most popular vector database which very fast & can create index in very quick period of time. Pinecone uses Approximate Nearest-Neighbor(ANN) with various metrics such cosine distance for internal search and retrieval.
Pinecone offers enterprise-grade security and compliance with SOC 2 and HIPAA regulations which suits well for banking and other financial data demanding heavy security to store the data and embeddings.
Pinecone offers both a paid licensing option and a free-tier plan. But in free tier, users are restricted to creating one index and one project in single starter pod.
However, in the paid plan, there can be number of indexes and the billing works on number of indexes and total time they are running in a month. Pinecone also has limitations such as maximum of 20,000 vector dimensions and a maximum of 2MB for upsert requests and 100 vectors per request. Additionally, vectors might not be available immediately after upsert.
Pinecone is available via AWS & Azure marketplace which enhances its availability for cloud native application. Below is the screenshot of pricing for RAG on AWS platform as given on Pinecone official website. The Pinecone pricing for Azure & GCP platform is coming soon, as mentioned on the website.
Finally, when to use which vector database is left to the Data Scientist developing GenAI applications. However, each vector database is optimized for specific use case even though they perform fairly well on all use cases in general.
FAISS: It enables efficient similarity search in large-scale applications involving high-dimensional vectors.
Chroma DB: Optimized for local development and prototyping.
Pinecone: For enterprise grade security & managed vector database, Pinecone is the go to vector database.