Vector Database 101: Similarity Thresholds and Indexing

For the past two years I've been working with a team building AI powered RFP tools for a few clients we have. A big part of that product is retrieval: the user asks a question, we search their document corpus, and whatever comes back gets handed to a language model to write from. It works well right up until someone asks a question the corpus can't answer.

Because then it answers anyway. Ten paragraphs come back, ranked, formatted exactly like the ten that come back when the answer does exist. They're just the ten least irrelevant things we happen to have lying around. Feed those to an LLM and you get a fluent, confident, fully cited, completely wrong response — which is worse than no response, because it takes so much longer to catch.

So we set out to give our search a floor. And while we were in there, we figured we'd index it for speed too. Both of those are one-line changes. Putting them together cost us 48 benchmark configurations and taught us something I still can't find written down anywhere. In this article I'll be sharing what we found.

Every number below came from a run we did. Where a number is weak, I say so.

What We Actually Wanted

Two things, and they seemed unrelated.

The first was a similarity threshold. Every result has to clear a bar, and if nothing clears it, the search returns nothing and admits it. In LanceDB that's one call:

table.search(query_vector).distance_range(upper_bound=1 - min_similarity)

The second was speed as the corpus grows. Comparing a query against every stored vector is fine at a thousand documents and miserable at a million. That means building an index. Also routine, also one call.

Both are well documented on their own. Put them together and there's a problem: an index can return exactly the right documents and attach a similarity score to each one that's too wrong to make a decision with.

It's easy to miss, and the reason is in how we all judge index quality. The standard question is whether the right rows came back — a question about ordering and membership. For ranked search that's the right question. A threshold doesn't care about ordering at all. It cares whether 0.71 actually means 0.71.

The Cost of Indexing, and the Cost Nobody Mentions

Without an index, every search reads every vector. That cost is linear and it shows up fast:

Indexed search barely moves as the corpus grows, which is the whole point. In exchange you pay in storage, in a background job you now have to run forever, and in results. An index is a shortcut past most of your data, so it misses things a full scan would have found. Across our grid, every index type lost somewhere between 6% and 13% of results, and lost more of them at larger scales.

That trade is the one every tutorial covers, and honestly it's a reasonable trade. It's also not the one that nearly broke us.

A quick aside, because it surprised us too: why doesn't scanning 50,000 documents take seconds?
At 3072 dimensions, 50,000 vectors is 614 MB of float32. Exhaustive search reads all of it, and on our machine that took 282 ms.
The instinct that this should be slow comes from picturing 50,000 separate lookups. It isn't that. The vectors live in one contiguous column, so the whole thing is a single sequential pass with one multiply-add per dimension, compiled and SIMD-vectorized. We measured the arithmetic alone at about 40 ms — the rest is just moving bytes. For contrast, the same comparison written as a plain Python loop extrapolates to roughly 28 seconds. Two orders of magnitude, same algorithm.
This is also the entire trick of an IVF index. It partitions that column and reads only the partitions nearest the query, by default around 9% of them. Nine percent of 282 ms is 25 ms. We measured 26.

The Popular Index Is the Wrong One

ivf_pq is the one you'll reach for. It's fast, it's small, and it's in every tutorial. Product quantization works by storing compressed vectors and computing distances against the compressed copies, which is exactly why it's small and exactly why its distances are approximations.

Similarity runs from 0 to 1. ivf_pq reported it wrong by up to 0.52. Consistently, in all twelve configurations we ran it in — every corpus size, every vector width. Your threshold still runs. It's just being applied to a number that has stopped describing anything.

And nothing tells you. No error, no warning, every row comes back with a plausible-looking score between 0 and 1, and every test still passes. You'd find out from a user asking why search returned garbage, months later, if at all.

To be fair to PQ, it's built for a case that is probably not yours. LanceDB's own docs put it at index sizes of 2–20 GB, and NVIDIA measures it beating ivf_flat at 100 million rows because the smaller index transfers faster. Our index at 100,000 rows was 1.2 GB. If you're not memory-bound at serious scale, you're not the customer.

Learning #1: Recall and distance fidelity are two different failure modes, and only one of them gets measured by default. Verify your index against exhaustive search and expect agreement down to float noise.

The Failure Behind the Failure

Look at that ivf_pq recall column again: 0.62.

That's an unremarkable number. It's the kind of figure you shrug at in exchange for a 4× memory saving — the sort of thing that gets written up as "we lose a few results and gain a lot of headroom." Every ANN benchmark reports it. Nothing about it says stop.

The 0.52 says stop, and it appears in no standard benchmark we could find.

Then We Added the Bound, and Recall Fell Off a Cliff

Here's where it gets worse, and where we had to correct ourselves before we could even read our own results.

Recall in that table was measured the normal way: run the query, compare the rows against exhaustive search. But a threshold never runs that query. It runs a bounded one. So the grid measured that separately — search the index with the bound applied, search exhaustively with the same bound, compare.

That second measurement sat in our results file for three days before anyone read it. And when we did read it, the numbers looked survivable, which turned out to be our own bug. Our recall calculation credits a query with perfect recall when the exhaustive side comes back empty. A bound that most of the corpus can't answer produces a lot of empty exhaustive results, and every one of them was quietly scoring 1.0.

So we re-ran the 1536-dimension column, this time recording the share of queries that had any bounded ground truth at all. At a bound of 0.45 that share runs from 13% at 10,000 rows to 33% at 50,000. The rest of every cell was padding.

Back the padding out and this stops being a trade and starts being a wall:

On the queries where a bounded exhaustive search actually returns something, both product quantization variants return almost none of it. Not a degraded fraction. Close to nothing, at every scale we measured. A bound handed to the database gets applied during the coarse scan, so the rows it throws away are chosen using the same compressed distances that were already wrong by 0.52 — and a row discarded there is not a row any later stage can get back.

ivf_pq_refine is the part we did not see coming. It matched raw ivf_pq to within 0.002 in every single cell, while its unbounded recall sat 0.31 higher. Refinement re-ranks candidates that the coarse scan has already chosen, so it fixes the score attached to a row and has no say whatsoever in which rows survived the bound. The index that looks like the careful compromise behaves exactly like the one it was supposed to fix.

Two limits on this, because I'd rather say them than have you find them. The re-run covers 10,000 to 50,000 rows and stopped before the 100,000 cell, so the largest scale here is inferred from the trend rather than measured. And it ran against a newer LanceDB than the grid above, so read that corrected column as its own measurement rather than a continuation of the first table.

Learning #2: Measure recall with your bound applied, not without it. An index that loses 8% of results unbounded can lose effectively all of them once a threshold is in play, and re-ranking does not save you.

Filtering In Your Own Code Doesn't Save You

The obvious escape route is to let the index be approximate, pull the top k, and apply the threshold yourself on whatever scores come back.

It fails twice over. The scores you'd be filtering on are the wrong scores, which was the original problem and is completely unchanged. And you can only filter what you actually received — the index already truncated to k, so a genuinely relevant document that fell outside that window is gone before your code ever runs.

The bound has to go into the database, where it gets applied before truncation instead of after.

So Which Index? ivf_flat

It stores full vectors and computes real distances, so its error is float noise. It costs roughly 2× storage. For almost everyone reading this, that's the answer.

If you genuinely need PQ's compression, ivf_pq_refine re-ranks its candidates against the original uncompressed vectors, which is why it measured exactly 0.0 error where raw PQ measured 0.52. You pay about 7× the build time for it — and please read the bounded recall section above before you take it as a fix. The scores it returns are exact. The set of rows that clears a bound is still raw PQ's.

Staying exhaustive is also a real answer for a lot longer than people assume. At our production corpus size a full scan took 15 ms, and LanceDB declined to build a meaningful index at all because there wasn't enough data to partition. In our data the crossover sat around 10,000 rows.

An Index Is a Snapshot

It covers the rows that existed when it was built. Everything uploaded afterwards sits outside it and gets scanned one by one on every query, on top of the index lookup.

This bites faster than seems reasonable. On a 5,000-row indexed table, inserting a single row took query p95 from 9.8 ms to 31.5 ms. One row, three times slower. Leave it long enough and you converge on the worst of both worlds: exhaustive-search latency while still paying for an index's storage.

There's one piece of good news, and it's specific to thresholds. Those unindexed rows get scanned exhaustively, so their distances are exact. Staleness costs you speed here, not correctness.

But Rebuilding On Every Write Is Worse

The obvious correction is to rebuild whenever something changes. That's expensive, for a reason that isn't obvious until you measure it:

Two things are going on here. Cost scales with the size of the whole table rather than with how much actually changed — the same 1,000 new rows cost 0.9 s at 10k and 16.8 s at 100k, and that gets permanently worse as you grow. And most of the cost is fixed per call rather than per row, which the bottom two rows show starkly: 100× the data for 3.4× the time.

So batch it. Let changes accumulate and then rebuild — but don't let them accumulate too far, because by 10,000 unindexed rows our queries had degraded to 100 ms. Per tenant, when a few thousand rows are outstanding, or overnight, whichever comes first.

And budget for it as a scheduled job rather than a cron line. Seventeen seconds per tenant at 100,000 rows, multiplied across a lot of tenants, is a scheduling problem.

Learning #3: Index rebuild cost scales with total table size, not with how much changed, and most of it is fixed per call. Batch rebuilds on a schedule and cap how stale you let the index get.

Don't Borrow a Threshold — Including Ours

We measured where the bar has to sit on two different corpora and got two incompatible answers.

Both the position of the band and its width move. A threshold tuned on one corpus is meaningless on the other, and not off by a little. Whatever profile you store, key it on the embedding model, the dimension count and the task type together, and measure it on your own data.

One caveat on that public row, because it weakens the number: we used random unit vectors as the "no match" control, and in 1536 dimensions random vectors are near-orthogonal to everything, scoring around 0.1. A real but off-topic question would score higher than that, so 0.221 is an upper bound rather than a fair estimate.

0.5 Is Not a Moderate Setting

It looks like one. On our corpus the entire real-query distribution sat above 0.57, with a 5th percentile of 0.571 and a median of 0.663. A threshold of 0.5 admits everything. It's a no-op wearing the costume of a setting, and it's the value people reach for precisely because it sounds balanced.

Worth resetting a related expectation while we're here: a threshold separates on-topic from off-topic, not good from mediocre. Our specific and generic queries overlapped heavily — 0.71–0.76 against 0.68–0.75. A threshold will stop an unanswerable question from returning ten chunks. It will not make a vague question return only the best three.

Maximum Strictness Quietly Meant No Filter At All

This one's my favorite, in the way that only a bug you've already fixed can be a favorite.

Requesting the strictest possible bar converts to a distance bound of 0.0. LanceDB's hybrid query builder forwards that bound like this:

if self._lower_bound or self._upper_bound:

Zero is falsy in Python. The strictest request our API accepts silently became no filter at all — the exact inverse of what was asked for.

The part that stings is what it did to our testing. The vector-only builder forwards the bound unconditionally, so a test written against vector search passes happily while the hybrid path is broken. We clamp to a small epsilon now, and the test that covers it is a hybrid test.

Learning #4: Boundary values that are falsy in your language will slip through truthiness checks in someone else's query builder. Test the extremes of a range on every code path, not just the one path you happened to write the test against.

One Collection, One Embedding Model

Similarity is only comparable between vectors produced by the same model at the same width. Change models, re-ingest part of a corpus, and you're left with two generations of vector in one collection — and a threshold then compares numbers that mean different things from row to row.

This is cheap to check. A pushdown count of rows that don't match the current model is exact and takes milliseconds:

mismatched = table.count_rows(
    f"embedding_model != '{model}' OR embedding_dim != {dim}"
)

If it comes back non-zero, serve unfiltered results and tell the caller the threshold wasn't applied. Filtering on a number you already know is mixed is worse than not filtering at all.

Learning #5: Measure your threshold on your own corpus, with your embedder, at your vector width, and store it keyed by all three. Where there's no calibration, ship no value — return unfiltered and say so rather than quietly applying a plausible-looking constant.

We know what guessing costs because we did it. Our first set of constants came from the wrong measurement entirely — chunk-to-chunk similarity instead of query-to-chunk — and would have admitted the whole corpus at every setting while looking completely reasonable in code review.

Hybrid Search Breaks All of This Again

Hybrid search fuses semantic results with keyword (BM25) results, and it breaks the threshold story in a specific way. Keyword scores have no absolute scale, and most implementations normalize them within each result set, which destroys the relative scale too. There's no honest similarity number to compare a keyword hit against.

The tempting fix is to stamp keyword hits as a perfect match so everything is comparable again. It breaks three ways at once. The slider stops shrinking, because at maximum strictness every keyword match still comes back — and results failing to shrink as the user tightens is the one thing a strictness control cannot do. Ranking inverts, because on a corpus of precise terminology a query like "security requirements" keyword-matches every chunk containing the word "security", boilerplate included, and stamping those as perfect puts them above genuine semantic matches. And aggregate confidence climbs at exactly the moment retrieval is weakest, because those fabricated perfect scores get averaged into it.

Thresholding the fused score instead is the other tempting move, since fusion hands you one number per row. But a fusion score is computed from ranks, not from similarity. It answers "how highly did both methods rate this" and has no fixed relationship to "how similar is this", and its scale shifts with the size of the result set.

What works is to stop trying to put both legs on one number. Bound the vector leg only, keep every keyword hit, tag each row with which leg found it, and leave similarity absent rather than invented for keyword-only rows.

A caller who wants strictness filters on similarity; a caller who wants keyword recall keeps the rest. In LanceDB this is close to free, because distance_range is only forwarded to the vector sub-query anyway — so keyword recall survives a bound without you doing anything, and a null distance on a fused row is itself the signal that the row came from the keyword side.

Learning #6: Don't force two incompatible scoring systems onto one number. Bound the leg that has a real scale, leave the other one unscored, and tag every row with where it came from.

The Checklist

If you're about to put a threshold in front of an indexed vector search, this is what I'd check:

Does your index report true distances? Verify against exhaustive search and expect agreement to float noise. If they disagree, either your harness is wrong or your index is unusable for thresholds, and you need to know which.

Have you measured distance error, and not just recall?

Have you measured recall with the bound applied?

Is the bound pushed into the database rather than applied after top-k?

Is the rebuild scheduled and batched, rather than per-write or never?

Is the collection single-model? Checked, not assumed.

Is one score definition used on every search path, including the distance metric? Ours silently used cosine on one path and L2 on another. It was invisible because fused ranking came out identical, and it stays invisible right up until a cosine-derived bound is applied, since for unit vectors l2² = 2·cosine_distance.

Did you calibrate on your own corpus?

On hybrid, is only the vector leg bounded, with keyword rows left unscored?

What We Haven't Established

I'd rather list these than let you assume we covered them.

Calibrated values for named strictness levels need labeled relevance data we haven't collected yet.

Latency above 100,000 rows is extrapolated. A 1M × 1536 float32 array is 6.1 GB and ours ran out of memory on a 15 GB machine.

The bounded recall re-run stopped at 50,000 rows, so the 100,000 figure in that section is trend, not measurement.

We also retracted a finding partway through. An earlier draft claimed a related fix improved separation between on-corpus and off-corpus queries. It compared the best on-corpus score against the best off-corpus score, when the number that actually governs a threshold is the worst on-corpus against the best off-corpus — that's the window the bar has to fit inside. Measured properly, separation slightly narrowed, and the change now ships off by default. I've left it in here because a write-up that quietly drops its own corrections is marketing.

Retrospective

Why did this take 48 configurations and a self-correction to find?

Because we were asking the question the ecosystem taught us to ask. Every ANN benchmark, every index comparison, every "which vector index should I use" blog post measures recall. Recall is the right metric for ranked search, and ranked search is what vector databases were built for. The moment you add a threshold you've changed the question from "did the right things come back in the right order" to "is this number trustworthy enough to make a decision with" — and nothing in the tooling notices you've changed it.

The same blind spot showed up inside our own harness. We built a bounded recall measurement, ran it, and it reported numbers that looked fine, because an empty result set scored as perfect recall. We had the measurement and still couldn't read it for three days. The fix wasn't more data, it was recording how much of each number was actually a measurement.

That's the part I'd take to any team: when you change what a system is for, go back and check that your metrics changed with it. Ours hadn't, and every tool we reached for agreed with us that everything was fine.

The Short Version

Indexing a vector database is normally presented as a trade between speed and recall. Add a similarity threshold and there's a third axis nobody mentions: whether the number attached to each result still means what it says.

The failures are quiet, and that's the point. Search stays fast. It returns reasonable-looking documents with reasonable-looking scores. It just stops being able to tell you it doesn't know.

P.S.

If you want the raw numbers, the harness and all 48 configurations are reproducible — the benchmark holds queries out of a public dataset rather than embedding anything, so it costs nothing to re-run.

Check out my other work if you wanna pick my brain :)

Measured on LanceDB 0.30.2, with the bounded recall re-run on 0.39.0. Index grid: a 1M-row DBpedia subset, 48 configurations across 4 corpus sizes, 3 vector widths and 4 index types. Threshold bands: one public corpus and one private.