. They resolve an actual drawback, and in lots of circumstances, they’re the best alternative for RAG techniques. However right here’s the factor: simply since you’re utilizing embeddings doesn’t imply you want a vector database.
We’ve seen a rising pattern the place each RAG implementation begins by plugging in a vector DB. Which may make sense for large-scale, persistent data bases, but it surely’s not at all times probably the most environment friendly path, particularly when your use case is extra dynamic or time-sensitive.
At Planck, we make the most of embeddings to boost LLM-based techniques. Nevertheless, in considered one of our real-world purposes, we opted to keep away from a vector database and as a substitute used a easy key-value retailer, which turned out to be a significantly better match.
Earlier than I dive into that, let’s discover a easy, generalized model of our state of affairs to elucidate why.
Foo Instance
Let’s think about a easy RAG-style system. A consumer uploads just a few textual content information, perhaps some reviews or assembly notes. We break up these information into chunks, generate embeddings for every chunk, and use these embeddings to reply questions. The consumer asks a handful of questions over the subsequent couple of minutes, then leaves. At that time, each the information and their embeddings are ineffective and may be safely discarded.
In different phrases, the information is ephemeral, the consumer will ask solely a few questions, and we need to reply them as quick as potential.
Now pause for a second and ask your self:
The place ought to I retailer these embeddings?
Most individuals’s intuition is: “I’ve embeddings, so I want a vector database”, however pause for a second and take into consideration what’s really occurring behind that abstraction. While you ship embeddings to a vector DB, it doesn’t simply “retailer” them. It builds an index that accelerates similarity searches. That indexing work is the place a whole lot of the magic comes from, and likewise the place a whole lot of the fee lives.
In a long-lived, large-scale data base, this trade-off makes good sense: you pay an indexing value as soon as (or incrementally as information modifications), after which unfold that value over hundreds of thousands of queries. In our Foo instance, that’s not what’s occurring. We’re doing the alternative: continuously including small, one-off batches of embeddings, answering a tiny variety of queries per batch, after which throwing all the pieces away.
So the true query will not be “ought to I take advantage of a vector database?” however “is the indexing work value it?” To reply that, we are able to have a look at a easy benchmark.
Benchmarking: No-Index Retrieval vs. Listed Retrieval
This part is extra technical. We’ll have a look at Python code and clarify the underlying algorithms. If the precise implementation particulars aren’t related to you, be at liberty to skip forward to the Outcomes part.
We need to examine two techniques:
- No indexing in any respect, simply retains embeddings in reminiscence and scans them instantly.
- A vector database, the place we pay an indexing value upfront to make every question sooner.
First, take into account the “no vector DB” method. When a question is available in, we compute similarities between the question embedding and all saved embeddings, then choose the top-k. That’s simply K-Nearest Neighbors with none index.
import numpy as np
def run_knn(embeddings: np.ndarray, query_embedding: np.ndarray, top_k: int) -> np.ndarray:
sims = embeddings @ query_embedding
return sims.argsort()[-top_k:][::-1]The code makes use of the dot product as a proxy for cosine similarity (assuming normalized vectors) and types the scores to seek out the perfect matches. It actually simply scans all vectors and picks the closest ones.
Now, let’s have a look at what a vector DB sometimes does. Underneath the hood, most vector databases depend on an approximate nearest neighbor (ANN) index. ANN strategies commerce a little bit of accuracy for a big enhance in search velocity, and some of the broadly used algorithms for that is HNSW. We’ll use the hnswlib library to simulate the index habits.
import numpy as np
import hnswlib
def create_hnsw_index(embeddings: np.ndarray, num_dims: int) -> hnswlib.Index:
index = hnswlib.Index(house='cosine', dim=num_dims)
index.init_index(max_elements=embeddings.form[0])
index.add_items(embeddings)
return index
def query_hnsw(index: hnswlib.Index, query_embedding: np.ndarray, top_k: int) -> np.ndarray:
labels, distances = index.knn_query(query_embedding, ok=top_k)
return labels[0]To see the place the trade-off lands, we are able to generate some random embeddings, normalize them, and measure how lengthy every step takes:
import time
import numpy as np
import hnswlib
from tqdm import tqdm
def run_benchmark(num_embeddings: int, num_dims: int, top_k: int, num_iterations: int) -> None:
print(f"Benchmarking with {num_embeddings} embeddings of dimension {num_dims}, retrieving top-{top_k} nearest neighbors.")
knn_times: checklist[float] = []
index_times: checklist[float] = []
hnsw_query_times: checklist[float] = []
for _ in tqdm(vary(num_iterations), desc="Working benchmark"):
embeddings = np.random.rand(num_embeddings, num_dims).astype('float32')
embeddings = embeddings / np.linalg.norm(embeddings, axis=1, keepdims=True)
query_embedding = np.random.rand(num_dims).astype('float32')
query_embedding = query_embedding / np.linalg.norm(query_embedding)
start_time = time.time()
run_knn(embeddings, query_embedding, top_k)
knn_times.append((time.time() - start_time) * 1e3)
start_time = time.time()
vector_db_index = create_hnsw_index(embeddings, num_dims)
index_times.append((time.time() - start_time) * 1e3)
start_time = time.time()
query_hnsw(vector_db_index, query_embedding, top_k)
hnsw_query_times.append((time.time() - start_time) * 1e3)
print(f"BENCHMARK RESULTS (averaged over {num_iterations} iterations)")
print(f"[Naive KNN] Common search time with out indexing: {np.imply(knn_times):.2f} ms")
print(f"[HNSW Index] Common index development time: {np.imply(index_times):.2f} ms")
print(f"[HNSW Index] Common question time with indexing: {np.imply(hnsw_query_times):.2f} ms")
run_benchmark(num_embeddings=50000, num_dims=1536, top_k=5, num_iterations=20)Outcomes
On this instance, we use 50,000 embeddings with 1,536 dimensions (matching OpenAI’s text-embedding-3-small) and retrieve the top-5 neighbors. The precise outcomes will range with totally different configs, however the sample we care about is similar.
I encourage you to run the benchmark with your individual numbers, it’s one of the simplest ways to see how the trade-offs play out in your particular use case.
On common, the naive KNN search takes 24.54 milliseconds per question. Constructing the HNSW index for a similar embeddings takes round 277 seconds. As soon as the index is constructed, every question takes about 0.47 milliseconds.
From this, we are able to estimate the break-even level. The distinction between naive KNN and listed queries is 24.07 ms per question. That means you want 11,510 queries earlier than the time saved on every question compensates for the time spent constructing the index.
Moreover, even with totally different values for the variety of embeddings and top-k, the break-even level stays within the hundreds of queries and stays inside a reasonably slender vary. You don’t get a state of affairs the place indexing begins to repay after just some dozen queries.
Now examine that to the Foo instance. A consumer uploads a small set of information and asks just a few questions, not hundreds. The system by no means reaches the purpose the place the index pays off. As a substitute, the indexing step merely delays the second when the system can reply the primary query and provides operational complexity.
For this type of short-lived, per-user context, the easy in-memory KNN method will not be solely simpler to implement and function, however it’s also sooner end-to-end.
If in-memory storage will not be an choice, both as a result of the system is distributed or as a result of we have to protect the consumer’s state for a couple of minutes, we are able to use a key-value retailer like Redis. We will retailer a novel identifier for the consumer’s request as the important thing and retailer all of the embeddings as the worth.
This provides us a light-weight, low-complexity resolution that’s well-suited to our use case of short-lived, low-query contexts.
Actual-World Instance: Why We Selected a Key-Worth Retailer
At Planck, we reply insurance-related questions on companies. A typical request begins with a enterprise title and tackle, after which we retrieve real-time information about that particular enterprise, together with its on-line presence, registrations, and different public information. This information turns into our context, and we use LLMs and algorithms to reply questions based mostly on it.
The vital bit is that each time we get a request, we generate a recent context. We’re not reusing present information, it’s fetched on demand and stays related for a couple of minutes at most.
When you assume again to the sooner benchmark, this sample ought to already be triggering your “this isn’t a vector DB use case” sensor.
Each time we obtain a request, we generate recent embeddings for short-lived information that we’ll possible question only some hundred occasions. Indexing these embeddings in a vector DB provides pointless latency. In distinction, with Redis, we are able to instantly retailer the embeddings and run a fast similarity search within the utility code with nearly no indexing delay.
That’s why we selected Redis as a substitute of a vector database. Whereas vector DBs are glorious at dealing with giant volumes of embeddings and supporting quick nearest-neighbor queries, they introduce indexing overhead, and in our case, that overhead will not be value it.
In Conclusion
If you might want to retailer hundreds of thousands of embeddings and assist high-query workloads throughout a shared corpus, a vector DB could be a greater match. And sure, there are undoubtedly use circumstances on the market that actually want and profit from a vector DB.
However simply since you’re utilizing embeddings or constructing a RAG system doesn’t imply you need to default to a vector DB.
Every database expertise has its strengths and trade-offs. Your best option begins with a deep understanding of your information and use case, somewhat than mindlessly following the pattern.
So, the subsequent time you might want to select a database, pause for a second and ask: am I selecting the best one based mostly on goal trade-offs, or am I simply going with the trendiest, shiniest alternative?
