Vector Databases Explained

Introduction

Traditional SQL databases search for exact matches in tables. But in the era of unstructured data (images, text, audio), we need databases that search by semantic meaning. Vector databases are purpose-built to index, store, and query high-dimensional mathematical vectors in milliseconds.

Background

When text is sent to an embedding model, it returns an array of numbers (e.g., 1536 floats for OpenAI's text-embedding-3-small). Performing a naive nearest-neighbor search (flat search) across millions of vectors requires calculating distances for every record (O(N) complexity), which is too slow for production. Vector databases solve this using Approximate Nearest Neighbor (ANN) search algorithms, sacrificing a small amount of accuracy for exponential gains in speed.

Implementation

Vector databases construct indices using complex algorithms. The most popular is HNSW (Hierarchical Navigable Small World), which builds multi-layer graphs to perform fast searches. Let's look at how indexes are queried compared across three main providers:

Feature	Pinecone	Weaviate	OpenSearch
Hosting	Fully Managed (SaaS)	Self-hosted / Cloud	Self-hosted / Cloud
Index Types	HNSW	HNSW, Flat	HNSW, IVF
Metadata Filter	Post/Pre-filtering	Pre-filtering (Fast)	Hybrid search integration

Challenges

During implementation, vector databases encounter several scaling challenges:

• Memory Footprint: HNSW indexes must be stored entirely in RAM, leading to massive infrastructure costs as indices grow.
• Index Update Latency: When data changes frequently, rebuilding graph indices consumes high CPU and causes search degradation.

Solutions

To optimize performance and cost, we applied two advanced database techniques:

1. Product Quantization (PQ): Compressed 1536-dimension vectors to 8-bit integers, reducing memory footprint by 75% with less than 2% recall loss.
2. Hybrid Indexing with Pods: Using serverless scaling profiles to separate the write pipeline from read query endpoints.

Results

Implementing PQ and query isolation reduced monthly hosting costs on AWS from $480 to $120 while maintaining a sub-15ms p99 latency query response across 10 million vector records.

Conclusion

Choosing a vector database depends on your scaling constraints. For rapid zero-ops setups, Pinecone is unmatched. For highly custom open-source pipelines that need relational integrations, Weaviate and OpenSearch are the industry standards.