Semantic Search, Part 2: Building a Real System

Part 1 left us with a working prototype: a pre-trained embedding model, a product catalog, and a matrix multiply that finds semantically relevant items even when the query and the product share no words.

That system is intentionally simple. It works. But it's leaving signal on the table — in the data, in the index, in the scoring, and in signals we haven't touched yet (keyword quality, visual appearance).

This part builds the real thing. Same Vuori catalog. Same query. We improve it at every layer:

1. Enriched chunks — pull all the structured metadata Vuori provides and bake it into the embedding
2. A real index — move from a numpy matrix into Supabase pgvector, with HNSW and metadata filtering
3. BM25 — replace raw term frequency with a calibrated keyword scorer
4. RRF — fuse keyword and vector signals without the score-scale mismatch
5. Image embeddings — CLIP gives every product a visual fingerprint alongside its text vector
6. 3-way hybrid — BM25 + text + image, fused with RRF

Then a detour into index architecture: when does each layer of this stack break under scale, and what does it cost?

Section 1 — Enriching the chunks

In Part 1, each product became a single string:

"{name} — {description}"

That's what the embedding model saw. For most Vuori products, the description is 1–3 sentences of marketing copy: "Our most versatile short, crafted from recycled materials." It's sparse, SEO-flat, and omits most of what a shopper actually cares about — the fit, the fabric, the garment category, the style tags.

Vuori's product API returns all of that. We just didn't use it.

The key insight: the embedding model can only encode what you give it. A better chunk isn't a different model — it's better input to the same model.

What the API actually returns

The _next/data endpoint we used in Part 1 returns far more than name and description. Here's a raw product with all fields visible:

{
  "slug": "jeffreys-pullover-heather-grey",
  "name": "Jeffrey's Pullover",
  "color": "Heather Grey",
  "description": "An elevated everyday layer...",
  "category": "Hoodies & Sweatshirts",
  "material": "60% cotton, 40% recycled polyester fleece",
  "fit_notes": "Relaxed through chest and body. Slightly oversized.",
  "features": ["4-way stretch", "moisture-wicking", "quick-dry"],
  "tags": ["end-use::fitness-lounge", "fit::relaxed", "prod-usage::travel"]
}

The tags field is a dumping ground. Looking at real products, most are internal Shopify infrastructure — collection badges (badge::gg-her-bestsellers-1), A/B test flags (CHATGPT-9_11), personalization markers (Nosto-new). Only four namespaces consistently carry semantic signal:

The right filter is an allowlist by namespace prefix, not a blocklist by content. We keep those four namespaces and strip the prefix before embedding — "prod-usage::travel" becomes "travel", which is what a model trained on natural language actually understands.

Pulling the structured fields

Here's the updated fetch_product() that pulls all of it:

The chunk template

Now that we have all the fields, we need to concatenate them into a single string for the embedding model. The goal is one dense natural-language string per product that packs in every piece of signal a shopper might query on. A few principles:

• Lead with name and description — highest-density signal
• Follow with structured fields as short, natural phrases
• Tags last — low individual weight, but useful in aggregate
• Keep it under ~400 tokens — BGE's effective context is 512, and embedding quality degrades past it

def build_chunk(p: dict) -> str:
    parts = [f"{p['name']} — {p['description']}"]
 
    if p.get("category"):   parts.append(f"Category: {p['category']}.")
    if p.get("material"):   parts.append(f"Material: {p['material']}.")
    if p.get("fit_notes"):  parts.append(f"Fit: {p['fit_notes']}.")
    if p.get("features"):   parts.append("Features: " + ", ".join(p["features"]) + ".")
    if p.get("tags"):       parts.append("Tags: " + ", ".join(p["tags"]) + ".")
 
    return " ".join(parts)

A concrete example — the same product, plain vs. enriched:

Plain:
"Jeffrey's Pullover — An elevated everyday layer crafted from our signature..."

Enriched:
"Jeffrey's Pullover — An elevated everyday layer crafted from our signature...
 Category: Hoodies & Sweatshirts. Material: 60% cotton, 40% recycled polyester fleece.
 Fit: Relaxed through chest and body. Slightly oversized. Features: 4-way stretch,
 moisture-wicking, quick-dry. Tags: fitness lounge, relaxed, travel."

A note on BGE's asymmetric retrieval

Before comparing results: you may have noticed in Part 1 that the query gets a "query: " prefix but the product texts don't. That's not boilerplate — it's load-bearing.

BGE is an asymmetric retrieval model. During training, short queries were always prefixed with "query: " and documents were not. The model learned to embed them into the same vector space but from slightly different angles — the prefix shifts the query vector toward the region of space that represents "what a user is looking for" rather than "what a product says it is."

Removing the prefix from the query makes scores drop noticeably. Adding the prefix to documents also hurts — the model wasn't trained that way. The asymmetry is the feature.

For the products: no prefix, full chunk. For the query: "query: " prepended, always.

Does the geometry actually change?

We embed all products two ways — plain name + description and the full enriched chunk — and run t-SNE on both to see whether the structured metadata changed how the model organizes the catalog.

plain_embs    = model.encode(plain_texts, normalize_embeddings=True)
enriched_embs = model.encode(enriched_texts, normalize_embeddings=True)
# shape: (879, 768) each

The geometry changed — and not subtly.

Full catalog: plain is a blob, categories bleeding into each other, the query star buried in undifferentiated mass. Enriched has visible white space between clusters. The query star moved to the edge of a distinct cluster.

Zoomed neighbors: plain's 20 nearest neighbors are scattered — Ponto Pullover Hoodies mixed with Ponto Performance Crews mixed with a Baselayer Tight and a Venture Quarter Zip. Enriched's neighborhood is almost entirely Restore and Cypress line items, multiple colorways of the same products stacked nearly on top of each other. That's the right answer.

Let's see what that shift actually did to the ranked results.

Two things to notice.

Fit notes move the sweatshirt results. Items like Haven Hoodie and Cypress Vintage Crew entered the top 10 in the enriched version because their fit language matched the query's vibe — "oversized", "loose fit from chest through bottom", "fit is generous through the body." The model can't see the fit notes in the plain version; it only has marketing copy.

Tags move the outdoor results. In the hiking query (try it yourself), plain search puts an Alpine Vest at #3 — its description has "alpine" and "outdoor" language, but it's a vest. Enriched pushes it down and surfaces the Outdoor Trainer Shell, which has prod-usage::climbing, prod-usage::hiking, and end-use::outdoor in its tags. That's signal that doesn't exist in the product description at all — it only enters the embedding through the chunk.

The core point: the chunk is a design decision. The model encodes whatever you give it, faithfully and literally. Richer input → richer embedding → better retrieval. No model changes, no retraining.

Going granular: colorways as separate items

The catalog we've been working with collapses every colorway into one entry — the Restore Oversized Crew in Heather Grey and the Restore Oversized Crew in Forest Green are treated as the same product.

But the original query asked for "a lighter color, not white, not a weird color like green." Color is load-bearing signal. A system that collapses colorways can't surface the right one — it might find the right style and return the green version.

The fix is simple: drop the deduplication, add color to the chunk, re-embed. Each colorway becomes its own vector. Two consequences worth seeing:

1. The catalog grows from ~900 to ~2,600 items — closer to what a real production index looks like
2. Same-product colorways should cluster tightly in the embedding space, with the query star pulled toward lighter-colored ones

One implementation note: the color field in Vuori's API is unreliable — it always returns the first colorway's name regardless of which slug you fetched. The correct source is structuredData.hasVariant[0].color, which matches the slug.

Re-fetch without deduplication

Same scraper, same enriched fields — just no dedup step, correct color source, and color added to the chunk:

Catalog sizes:
  Deduped (Part 1 style):  879 items
  Full colorway catalog:  2,600 items

Restore Oversized Crew 2.0 colorways (8):
  Pale Grey Heather         chunk: Restore Oversized Crew 2.0 in Pale Grey Heather — ...
  Heather Grey              chunk: Restore Oversized Crew 2.0 in Heather Grey — ...
  Buttermilk                chunk: Restore Oversized Crew 2.0 in Buttermilk — ...
  Forest Green              chunk: Restore Oversized Crew 2.0 in Forest Green — ...
  ...

Does colorway-level indexing actually help?

Same-product colorways cluster tightly — they share almost all chunk content and differ only in one color word. The query star is pulled toward the lighter-colored ones.

Query: 'I need some kind of oversized sweatshirt...'

#1  [0.731]  Restore Oversized Crew 2.0  Pale Grey Heather
#2  [0.728]  Restore Oversized Crew 2.0  Pale Grey Heather   (second colorway variant)
#3  [0.724]  Restore Oversized Crew 2.0  Buttermilk
#4  [0.718]  Sunday Performance Hoodie   Dark Java
#5  [0.716]  Restore Oversized Crew 2.0  Heather Grey

Pale Grey Heather at #1 and #2, Buttermilk and Heather Grey filling the next slots. The light neutrals dominate the top half — color signal is real and it's working.

Dark Java at #4 is the honest failure. Haven Hoodie and Sunday Hoodie have strong fit and tag signal ("relaxed", "fitness lounge", "oversized") that outweighs the color mismatch. Color is a soft signal in the embedding — weighted against everything else in the chunk, not enforced as a constraint.

This is an important distinction. Embedding color into the chunk influences ranking. It doesn't filter. If a shopper says "not green," the model will down-rank green items — but a green item with overwhelming semantic alignment elsewhere can still surface. For hard exclusions you need a metadata predicate on top of vector search — a thread we pick up in Section 3 once we have a database to filter in.

The enriched, full-colorway embeddings are what we use for the rest of this post.

Rabbit hole: n-dimensional spaces and t-SNE (skippable)

In Part 1, and again above, we took the catalog's 768-dimensional embeddings and projected them down to a 2D scatter. We waved at "t-SNE collapses it to 2D while preserving neighborhood structure" and moved on. This section explains how that actually works — and picks up ideas (curse of dimensionality, PCA, gradient descent, KL divergence) that come up constantly in ML. It's the most math-y part of the post and is not load-bearing for the search system. Skip straight to Section 2 if you just want to keep building.

Why you can't just look at 768 dimensions

A 768-dimensional embedding is a point with 768 coordinates. We have no intuition for that — our intuition tops out at 3. And the honest answer to "what does 768-D look like?" is genuinely weird, because high-dimensional space behaves nothing like the 2-D and 3-D we picture.

The headline effect is the curse of dimensionality: as you add dimensions, everything drifts toward the same distance from everything else. Points that should be "near" and "far" become almost indistinguishable.

def dist_spread(d, n=500):
    """In d dimensions, how different are the nearest and farthest pairwise distances?"""
    X = np.random.randn(n, d)
    D = pairwise_distances(X)
    np.fill_diagonal(D, np.inf)
    return D.min(axis=1).mean() / D.replace(np.inf, 0).max(axis=1).mean()
 
for d in [1, 2, 10, 50, 100, 768]:
    print(f"  {d:>4}D  ratio = {dist_spread(d):.3f}")

   1D  ratio = 0.001   # farthest is 1000× farther than nearest
   2D  ratio = 0.020
  10D  ratio = 0.220
  50D  ratio = 0.680
 100D  ratio = 0.800
 768D  ratio = 0.978   # farthest is barely farther than nearest

In 1-D the farthest pair is ~1000× more distant than the nearest — distance is meaningful. By 768-D the ratio is nearly 1: the farthest point is barely farther than the nearest. Everything is roughly equidistant.

Two consequences ripple through the rest of this post:

1. You can't reduce 768-D to 2-D by preserving distances — the distances barely mean anything to begin with. You have to preserve something else: which points are each other's neighbors.
2. This is also why obvious index structures fail (foreshadowing Section 2). A k-d tree splits space with distance thresholds; when every distance is the same, the splits stop separating anything.

Dimensionality reduction: keep the neighborhoods, drop the axes

Dimensionality reduction is the family of techniques that squash high-D data into a few dimensions you can look at. The two you'll meet most:

• PCA (Principal Component Analysis) is linear: it rotates the space to the axes of greatest variance and keeps the top two. Fast, deterministic, great when the data lies near a flat plane — but a linear projection can't unfold a curved, clustered manifold. Distant blobs routinely land on top of each other.
• t-SNE (t-distributed Stochastic Neighbor Embedding) is nonlinear and built around one goal: if two points are neighbors in 768-D, keep them neighbors in 2-D. It doesn't care about preserving global distances — only local neighborhoods. That's exactly the tradeoff the curse of dimensionality forces on us.

# PCA: center, SVD, project onto top-2 variance directions
Xc = full_embs - full_embs.mean(axis=0)
U, S, Vt = np.linalg.svd(Xc, full_matrices=False)
pca_2d = Xc @ Vt[:2].T  # shape: (2600, 2)
var_explained = (S[:2]**2).sum() / (S**2).sum()
# → 0.031: PCA's top-2 directions explain only 3.1% of variance

How t-SNE works, in three steps

The whole algorithm is a matching problem between two probability distributions:

1. In high-D, turn "who is near whom" into a probability distribution — for each point, a spread of probability over its neighbors.
2. In low-D (the 2-D map we're solving for), define the same kind of distribution.
3. Nudge the 2-D points until the two distributions match, measured by KL divergence, minimized with gradient descent.

Step 1 — high-D similarities as a Gaussian:

For each point i, we place a Gaussian over its distance to every other point j and normalize so similarities to i sum to 1:

p(j|i) = exp(−||xᵢ − xⱼ||² / 2σᵢ²)  /  Σₖ exp(−||xᵢ − xₖ||² / 2σᵢ²)

Near points get high probability; far points get near-zero. The width σᵢ is set by a knob called perplexity — roughly "how many neighbors each point should pay attention to."

Step 2 — low-D similarities as a Student-t distribution:

In the 2-D map we use a Student-t distribution (Cauchy) instead of a Gaussian:

q(i,j) = (1 + ||yᵢ − yⱼ||²)⁻¹  /  Σₖ≠ₗ (1 + ||yₖ − yₗ||²)⁻¹

Why the switch? The crowding problem: there's vastly less room in 2-D than 768-D, so moderately-distant points get crushed together near the origin. The Student-t's heavy tails let those points sit far apart in the map without a big penalty — it's what makes t-SNE clusters visibly separate. (This heavy-tailed t is the "t" in t-SNE.)

Step 3 — match the two distributions with gradient descent:

We want Q (the 2-D layout) to look like P (the data). The mismatch is measured by KL divergence:

C = Σᵢ≠ⱼ p(i,j) · log( p(i,j) / q(i,j) )

This has a clean gradient: if two points are more similar in high-D than the map shows (p > q), pull them together; if less, push them apart. Gradient descent applies these forces repeatedly. The "Stochastic" in t-SNE's name comes from the random starting layout.

What you cannot read off a t-SNE plot

Because the layout is stochastic and only local neighborhoods are preserved:

• Axes are meaningless. There are no "x" and "y" features.
• Distances between clusters are not to scale. Only within-neighborhood proximity is trustworthy.
• Cluster sizes are not to scale. The heavy-tailed Student-t inflates dense clusters.
• Every run differs. Trust structure that's stable across runs, not any single layout.

t-SNE is a diagnostic lens, never part of the search path. We embed in 768-D, we search in 768-D; we only drop to 2-D to sanity-check with our eyes. The curse of dimensionality doesn't go away when we stop plotting — it's waiting for us in the search engine. Comparing a query against all 2,600 vectors works fine. Against 560 million, "just compute every distance" collapses. Time to build an index.

Section 2 — The index: buying sublinear search

Our enriched search works beautifully for Vuori's ~2,600 items. Every query embeds once, then we compare it against every product with a single matrix multiply — a few milliseconds, imperceptible.

But "compare against every product" is the load-bearing phrase. That's a flat scan: linear in the number of items, O(n). Fine at 2,600. What about 1 million products? 10 million? What if we were building this for Amazon — estimated 560M+ distinct items in the US alone?

Two costs grow with the catalog:

The one that bites first is latency: a flat scan re-reads the entire catalog on every query. At Amazon scale, a single search would take minutes. So we need sublinear search: finding the best matches without comparing against every vector. That means storing vectors in something smarter than a flat array — an index.

Every index trades against the same three axes:

• Speed — how fast is a query?
• Accuracy (recall) — of the true top-k nearest neighbors, how many did we return?
• Memory — how much RAM or disk does the index cost?

You can't max all three. Flat scan is perfect on accuracy and worst on speed. Every other index gives up a little recall to win big on speed, memory, or both. We'll build the real ones locally in FAISS — not because you'd ship FAISS in production, but because it lets us probe each one directly.

The memory hierarchy

Every index decision is ultimately a decision about where data lives:

FAISS lives in RAM. Postgres/pgvector lives on NVMe (with a RAM buffer pool for hot pages). S3-native indexes like turbopuffer push cold data all the way to object storage. The cost gap is stark: 1 billion 768-dim float32 vectors costs ~$8,400/month in pure RAM, ~$280/month on NVMe, ~$56/month on S3.

Why trees and hashes don't work here

k-d trees partition space with axis-aligned splits — they work great in 2D or 10D. In 768D they fail completely: with so many dimensions, every point is roughly equidistant from every other. The tree degenerates to a linear scan. The curse of dimensionality kills tree-based structures above ~20 dims.

Locality Sensitive Hashing (LSH) tries random projections to hash similar vectors into the same bucket. Clever in theory. But at high dimensions you need an enormous number of hash tables for acceptable recall — Pinecone benchmarks on Sift1M (1M 128-dim vectors, simpler than our 768-dim BGE embeddings) show LSH needs 8,192 hash bits to reach 90% recall and still only hits 0.85 recall with heavy resources. At 768 dims it's worse on all counts. Essentially dead for modern high-dimensional embeddings.

The two structures that actually work at scale are IVF (partition the space) and HNSW (build a graph).

IVF — partition the space

IVF stands for Inverted File Index. Use k-means to cluster all vectors into nlist groups (Voronoi cells) at build time. Each group has a centroid.

At query time: find the nprobe nearest centroids, then only search the vectors inside those cells. Skip everything else. If nlist = 100 and nprobe = 10, you're searching 10% of the data — but with a good partition, those 10% of cells contain most of the true nearest neighbors.

import faiss
import numpy as np
 
vecs = np.ascontiguousarray(full_embs.astype(np.float32))
dim, n = vecs.shape[1], vecs.shape[0]  # 768, ~2600
 
# Rule of thumb: nlist = 4*sqrt(n). Cap so each cell has ≥39 training points (FAISS requirement).
nlist = min(max(4, int(4 * n ** 0.5)), n // 39)
 
quantizer = faiss.IndexFlatIP(dim)
ivf = faiss.IndexIVFFlat(quantizer, dim, nlist, faiss.METRIC_INNER_PRODUCT)
 
ivf.train(vecs)   # runs k-means to learn cell centroids — required step
ivf.add(vecs)     # assigns each vector to its nearest centroid
ivf.nprobe = max(1, nlist // 10)   # search 10% of cells per query

The recall/latency tradeoff is fully controlled by nprobe:

IVF nprobe sweep  (nlist=66)

  nprobe  % cells   latency   recall
     1       2%      0.04ms    0.70
     4       6%      0.05ms    0.90
     8      12%      0.06ms    1.00
    16      24%      0.08ms    1.00
    33      50%      0.11ms    1.00
    66     100%      0.18ms    1.00   ← becomes exact at nprobe=nlist

At low nprobe, recall drops — the query's true neighbors might be in cells we didn't probe. At nprobe = nlist (every cell), IVF becomes an exact search identical to flat. The knob is continuous: tune nprobe until recall is acceptable for your use case.

HNSW — build a graph

HNSW (Hierarchical Navigable Small World) takes a completely different approach: instead of partitioning space, it builds a multi-layer graph where each node has edges to its nearest neighbors.

Think of a city map with multiple layers. The top layer is a sparse graph connecting only a few landmark nodes — long edges that let you travel far quickly. The bottom layer is the full graph with every node and short local connections. At query time you enter at the top, greedily walk toward the query (always taking the edge that moves you closer), then descend and repeat at each layer — zooming in with more nodes available at each level.

M = 16   # edges per node per layer — higher = better recall, more memory
hnsw = faiss.IndexHNSWFlat(dim, M, faiss.METRIC_INNER_PRODUCT)
hnsw.hnsw.efConstruction = 64   # build quality — higher = better index, slower build
 
hnsw.add(vecs)   # no train() step required — HNSW builds as you add
 
# efSearch controls the search beam width at query time
print(f"HNSW efSearch sweep  (M={M})\n")
for ef in [16, 32, 64, 128, 256]:
    hnsw.hnsw.efSearch = ef
    # time 20 queries, measure recall vs flat

HNSW efSearch sweep  (M=16)

  efSearch   latency   recall
      16      0.09ms    0.90
      32      0.13ms    0.90
      64      0.22ms    1.00
     128      0.42ms    1.00
     256      0.82ms    1.00

Flat (exact):  0.18ms   recall=1.00

HNSW hits perfect recall at efSearch=64 and is faster than flat scan even at our small catalog size. At 1M vectors, flat would take ~200ms and HNSW stays under 2ms.

One operational difference from IVF: HNSW doesn't support efficient deletes. Removing a node means rewiring its neighbors' edge lists — most implementations skip this, using soft-deletes and periodic rebuilds. For a product catalog that updates in bulk nightly, this is fine. For a catalog with frequent single-item removals, IVF is easier to maintain.

Quantization — shrinking the vectors

At 1B vectors, even HNSW hits a wall: 2.8 TB of raw float32 data doesn't fit on a single machine. Quantization compresses each vector into a smaller representation, trading a small amount of recall for large memory reduction.

Product Quantization (PQ) is the standard approach. Divide each 768-dim vector into M equal sub-vectors (e.g., 96 sub-vectors of 8 dims each). For each sub-space, run k-means with 256 centroids. Encode each sub-vector as a single byte — the index of its nearest centroid.

768 × 4 bytes = 3,072 bytes per vector → 96 × 1 byte = 96 bytes per vector. 32× compression.

M_pq = 96   # 768 / 96 = 8 dims per sub-vector
pq_index = faiss.IndexIVFPQ(quantizer, dim, nlist, M_pq, 8)
pq_index.train(vecs)
pq_index.add(vecs)
 
# Memory comparison:
print(f"float32: {n * dim * 4 / 1024:.0f} KB  →  PQ: {n * M_pq / 1024:.1f} KB")
print(f"At 1B vectors:  float32={1e9*dim*4/1e12:.1f} TB  →  PQ={1e9*M_pq/1e9:.0f} GB")
 
# IVF+PQ latency: 0.07ms   recall: 0.80
# (recall drop is the cost of 32× compression)

The recall drop is real but often acceptable — and tunable by increasing M_pq (more sub-quantizers = less compression = better recall). At 1B vectors the alternative is 2.8 TB of RAM; the compression is rarely optional.

Matryoshka embeddings are a newer alternative: train the model so the first N dimensions are themselves a valid smaller embedding. BGE-M3 and OpenAI's text-embedding-3 both support this. At query time you just truncate — vecs[:, :256] — and get ~90% of the quality at 1/3 the memory and compute. No compression step, no codebook, no recall tradeoff from quantization. The cleanest path if your embedding model supports it.

From FAISS to production

We now understand the design space: flat for tiny catalogs, HNSW when reads dominate and latency matters, IVF when writes are heavy, PQ when memory is the binding constraint. For Vuori — a read-heavy catalog that fits comfortably in memory — HNSW is the answer.

But we built all of this locally, in RAM, in a Python process that forgets everything when it exits. A real system needs persistence, filtering, and someone else to operate it.

Section 3 — pgvector: the production index

We just built HNSW, IVF, and PQ by hand in FAISS. That was to understand them — but FAISS is a library, not a database. Search it and you get back integer indices; you maintain a parallel Python list mapping index → product, nothing persists, and there's no filtering — you can't say "only search vectors where category = 'Jackets & Hoodies'."

In production you want all of that to be someone else's problem. pgvector is PostgreSQL with a vector column type added. You get a real database — rows, columns, SQL, persistence, joins — the <=> operator does cosine distance directly in a query, and the HNSW graph is one CREATE INDEX statement away. Filtering by category, color, or any field is just a WHERE clause.

The table shape:

We store chunk alongside the embedding so we can always read back exactly what signal the model had. When retrieval goes wrong, this is the first thing you check.

Connecting and inserting

from supabase import create_client
import os
from dotenv import load_dotenv
 
load_dotenv()
sb = create_client(os.environ['SUPABASE_URL'], os.environ['SUPABASE_SERVICE_KEY'])
 
# Insert in batches of 100 — ~25× faster than row-by-row
BATCH = 100
rows = [
    {
        'slug':      p['slug'],
        'name':      p['name'],
        'color':     p['color'],
        'category':  p['category'],
        'url':       p['url'],
        'chunk':     build_chunk_full(p),
        'embedding': emb.tolist(),   # numpy arrays aren't JSON-serializable
    }
    for p, emb in zip(products_full, full_embs)
]
for i in range(0, len(rows), BATCH):
    sb.table('products').upsert(rows[i:i+BATCH], on_conflict='slug').execute()

upsert with on_conflict='slug' skips duplicates silently — safe to re-run without double-inserting.

Querying the index

Instead of embeddings @ query_vec over a numpy matrix in memory, we send the query vector to Postgres and it does the similarity computation there — returning only the top-k rows over the wire.

The <=> operator is cosine distance (not similarity), so smaller is better and we ORDER BY ... ASC. Three operators pgvector exposes:

We use <=> because our embeddings are L2-normalized — cosine distance and inner product are equivalent on unit vectors, and cosine is the most interpretable.

PostgREST doesn't support raw ORDER BY expressions with operators, so we wrap the query in a Postgres function — define it once in the Supabase SQL editor, call it via sb.rpc():

CREATE OR REPLACE FUNCTION match_products(
    query_embedding vector(768),
    match_count      int DEFAULT 10
)
RETURNS TABLE (slug text, name text, color text, category text, url text, score float)
LANGUAGE sql STABLE AS $$
    SELECT slug, name, color, category, url,
           1 - (embedding <=> query_embedding) AS score
    FROM   products
    ORDER  BY embedding <=> query_embedding
    LIMIT  match_count;
$$;

def search_supabase(query: str, k: int = 10):
    qvec = model.encode(f'query: {query}', normalize_embeddings=True)
    return sb.rpc('match_products', {
        'query_embedding': qvec.tolist(),
        'match_count': k,
    }).execute().data

Results match exactly what we got from numpy matmul in Section 1. The only difference is where the computation happened: Python memory vs. inside Postgres.

The flat search is O(n)

Right now Postgres is doing a sequential scan: it reads every row in the table, computes cosine distance to the query vector, and returns the top-k. Run EXPLAIN ANALYZE in the Supabase SQL editor and you'll see Seq Scan on products. At 2,600 rows: ~2ms. At 100K: ~80ms. At 1M: ~800ms. Linear.

Add the HNSW index

One SQL statement:

CREATE INDEX ON products
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

• m = 16 — edges per node per layer. 16 is the standard default.
• ef_construction = 64 — build quality. 64 is fine for our scale.
• vector_cosine_ops — tells pgvector to optimize for the <=> operator. Must match what your queries use.

After it builds (a few seconds at our scale), run EXPLAIN ANALYZE again — you'll see Index Scan instead of Seq Scan. Supabase uses the index automatically. No code changes required.

Add metadata filtering

Now for the thing FAISS can't do. We can add a WHERE clause to the search function and Postgres filters before running the ANN search — only vectors matching the predicate are candidates. Let's solve the Dark Java problem from Section 1:

CREATE OR REPLACE FUNCTION match_products_filtered(
    query_embedding  vector(768),
    match_count      int     DEFAULT 10,
    filter_category  text    DEFAULT NULL,
    exclude_color    text    DEFAULT NULL
)
RETURNS TABLE (slug text, name text, color text, category text, url text, score float)
LANGUAGE sql STABLE AS $$
    SELECT slug, name, color, category, url,
           1 - (embedding <=> query_embedding) AS score
    FROM   products
    WHERE  (filter_category IS NULL OR category = filter_category)
    AND    (exclude_color    IS NULL OR color NOT ILIKE '%' || exclude_color || '%')
    ORDER  BY embedding <=> query_embedding
    LIMIT  match_count;
$$;

def search_filtered(query: str, k: int = 10, category: str = None, exclude_color: str = None):
    qvec = model.encode(f'query: {query}', normalize_embeddings=True)
    return sb.rpc('match_products_filtered', {
        'query_embedding': qvec.tolist(),
        'match_count': k,
        'filter_category': category,
        'exclude_color': exclude_color,
    }).execute().data
 
filtered = search_filtered(query, k=5, exclude_color='dark')

Unfiltered:                           Excluding 'dark' colors:
  #1  Restore Oversized Crew 2.0  Pale Grey Heather     #1  Restore Oversized Crew 2.0  Pale Grey Heather
  #2  Restore Oversized Crew 2.0  Pale Grey Heather     #2  Restore Oversized Crew 2.0  Pale Grey Heather
  #3  Restore Oversized Crew 2.0  Buttermilk            #3  Restore Oversized Crew 2.0  Buttermilk
  #4  Sunday Performance Hoodie   Dark Java    ←        #4  Haven Hoodie                Heather Grey
  #5  Restore Oversized Crew 2.0  Heather Grey          #5  Restore Oversized Crew 2.0  Heather Grey

Dark Java is gone. The WHERE clause ran first, removing all rows with "dark" in the color name before the ANN search even started. The semantic ranking within the light-color subset is unchanged.

This is the architectural gap that matters. Embeddings influence ranking. SQL filters enforce constraints. A well-designed system uses both: vector search to find the right vibe, metadata filters to enforce hard rules the shopper stated explicitly.

Our production index does two things well: semantic ranking and hard constraints. But there's a gap between them. What about an exact term the embedding blurs — a specific fabric name, a spec like "UPF 50+" — where the shopper means that literal thing, not something semantically nearby? And once we have a second way to search, how do we fuse a keyword ranking and a semantic ranking into a single result list?

That's keyword search and rank fusion — next.

Section 4 — Keyword search and fusion

Vector search is good at vibe. "Oversized cozy sweatshirt, lighter color, not white" — that query has no exact word overlap with any product description, and BGE handles it well because it learned what those words mean in context.

But vibe is the wrong tool for precision. If a shopper searches "DreamKnit" — Vuori's proprietary fabric — they want products that literally use that fabric, not items that are semantically adjacent to it. Vector search will find soft, stretchy items that feel like DreamKnit. Keyword search finds items that are DreamKnit.

The fix is a second signal: a keyword scorer that runs alongside the vector search, then we fuse the two rankings into one.

Why raw term frequency is broken

The simplest keyword scorer counts how many times each query word appears in a document. More matches → higher score. Three problems:

1. Repetition doesn't mean relevance. A description that says "soft" five times isn't five times more relevant than one that says it once.
2. Common words dominate. "The", "with", "a" appear everywhere. A document matching "the" ten times scores higher than one matching "DreamKnit" once.
3. Long documents win by default. A 200-word description has more chances to contain any given word than a 50-word one, even if the 50-word one is a tighter match.

BM25 fixes all three with one formula on top of the same token counts.

The three BM25 fixes

score(q, d) = Σ  IDF(term)  ×  (tf × (k1 + 1)) / (tf + k1 × (1 - b + b × |d| / avgdl))

Default constants: k1 = 1.5, b = 0.75. These work well for most corpora.

Build the index

from rank_bm25 import BM25Okapi
import re
 
def tokenize(text: str) -> list[str]:
    return re.findall(r"[a-z0-9]+", text.lower())
 
# Same enriched chunks used for vector search — tags, fit notes, features all in scope
chunks  = [build_chunk_full(p) for p in products_full]
corpus  = [tokenize(c) for c in chunks]
bm25    = BM25Okapi(corpus)
 
def bm25_search(query: str, k: int = 10) -> list[tuple[dict, float]]:
    tokens = tokenize(query)
    scores = bm25.get_scores(tokens)
    top_idx = scores.argsort()[::-1][:k]
    return [(products_full[i], float(scores[i])) for i in top_idx]

Where BM25 wins: exact brand and fabric terms

The oversized sweatshirt query has no exact overlap with product text — it's a pure vibe query. BM25 will struggle. Let's try the opposite: a query where the shopper knows exactly what they want and uses Vuori's own vocabulary.

Query: 'dreamknit fabric'

#   BM25                                          Vector
#1  Kore Short 7" in Heather Grey [14.7]          Kore Short 7" in Heather Grey [0.817]
#2  DreamKnit Pocket Tee [12.9]                   DreamKnit Pocket Tee [0.812]
#3  DreamKnit Quarter Zip [11.4]                  Kore Short 9" in Slate [0.809]
#4  Kore Short 9" in Slate [10.8]                 DreamKnit Essential Tee [0.801]
#5  DreamKnit Essential Tee [9.7]                 Venture Hoodie [0.795]   ← not DreamKnit

BM25's top results are all DreamKnit products — exact term match, IDF gives it high weight since it's rare across the corpus. Vector search found semantically related soft/comfortable items, but some won't say DreamKnit at all.

Now the vibe query — the original sweatshirt query from Part 1. No exact term overlap with any product description, pure semantic intent.

Query: 'oversized cozy sweatshirt lighter color faded style'

#   BM25                                          Vector
#1  Ponto Performance Crew [8.2]                  Restore Oversized Crew 2.0 [0.731]
#2  Vuori Ponto Pullover [7.9]                    Jeffrey's Pullover [0.724]
#3  Restore Oversized Crew 2.0 [7.1]              Cypress Vintage Crew [0.720]
#4  Haven Hoodie [6.8]                            Restore Oversized Hoodie [0.718]
#5  Coronado Mixed Media Crew [6.4]               Sunday Performance Hoodie [0.715]

Each signal wins where the other loses. BM25 nails exact terms; vector nails vibe. Notice also the score ranges: BM25 returns values in the 0–15 range; cosine similarity in 0.65–0.85. Adding them directly would let BM25 dominate every result by sheer magnitude. You can't normalize them either — their distributions are different shapes. This is the score incompatibility problem that RRF was designed to solve.

A precise spec the embedding blurs: "UPF"

DreamKnit is a brand term. Here's the more common — and more dangerous — version of the same problem: an exact spec.

A shopper who wants genuine sun protection cares about one thing: a stated UPF rating. They might phrase it in plain language — "lightweight top with sun protection for hiking" — but what they mean is the technical fact "this garment lists a UPF rating."

The embedding treats "sun protection" as a soft, fuzzy concept and happily returns anything summery, breezy, or outdoorsy. Only a handful of Vuori products actually state a UPF rating, and it's buried in a spec sentence — semantically indistinguishable from a hundred other "great for warm weather" items.

# Ground truth: which products actually state a UPF rating?
def has_upf(p):
    blob = " ".join(str(p.get(k, "")) for k in ["description", "features", "fit_notes"])
    return bool(re.search(r"\bupf\s*\d+", blob, re.I))
 
upf_slugs = {p["slug"] for p in products_full if has_upf(p)}
# → only 4 products genuinely list a UPF rating
 
q = "upf lightweight breathable sun protection"
print(f"BM25  top-8: {sum(1 for p,_ in bm25_search(q, k=8) if p['slug'] in upf_slugs)}/4 UPF items")
print(f"Vector top-8: {sum(1 for p,_ in vector_search_local(q, k=8) if p['slug'] in upf_slugs)}/4 UPF items")
# BM25:    3/4 UPF items in top-8
# Vector:  0/4 UPF items in top-8

The embedding understands sun protection but can't enforce the spec — it ranks a UPF shell right next to fifty vaguely-summery tops. BM25 doesn't understand anything; it just matches the token "upf" and pins exactly the items that carry the rating. Neither is wrong — they're answering different questions. Which is precisely why we want both, fused.

Fusing the two signals with RRF

We have two ranked lists. We can't combine their scores. What we can do is ignore the scores entirely and only look at position.

RRF — Reciprocal Rank Fusion — converts each result's rank into a score:

rrf_score = Σ  1 / (k + rank_i)

where k = 60 (a constant from the 2009 paper that dampens the advantage of the very top ranks) and the sum is over every list the item appears in.

A result at rank 1 in both lists scores 1/(61) + 1/(61) ≈ 0.033. A result at rank 1 in one list and absent from the other scores 1/(61) ≈ 0.016. The math enforces a simple invariant: agreement between signals beats dominance in one.

def rrf(ranked_lists: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    scores = {}
    for ranked in ranked_lists:
        for rank, slug in enumerate(ranked, start=1):
            scores[slug] = scores.get(slug, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)
 
def hybrid_search(query: str, k: int = 10, fetch: int = 50) -> list[dict]:
    bm25_ranked = [p["slug"] for p, _ in bm25_search(query, k=fetch)]
    vec_ranked  = [p["slug"] for p, _ in vector_search_local(query, k=fetch)]
    fused = rrf([bm25_ranked, vec_ranked], k=60)
    slug_to_product = {p["slug"]: p for p in products_full}
    return [slug_to_product[slug] for slug, _ in fused[:k] if slug in slug_to_product]

Crucially: the raw BM25 score of 14.7 and the cosine score of 0.82 never touch each other. Only ranks matter.

Three-column comparison

Both queries across all three retrieval modes:

Query: 'dreamknit fabric'

#   BM25                    Vector                  RRF Hybrid
#1  Kore Short 7" HG        Kore Short 7" HG        Kore Short 7" HG
#2  DreamKnit Pocket Tee    DreamKnit Pocket Tee    DreamKnit Pocket Tee
#3  DreamKnit Quarter Zip   Kore Short 9"           DreamKnit Quarter Zip
#4  Kore Short 9"           DreamKnit Essential     Kore Short 9"
#5  DreamKnit Essential     Venture Hoodie ←        DreamKnit Essential

Query: 'oversized cozy sweatshirt lighter color faded style'

#   BM25                    Vector                  RRF Hybrid
#1  Ponto Performance Crew  Restore Oversized Crew  Restore Oversized Crew
#2  Vuori Ponto Pullover    Jeffrey's Pullover      Jeffrey's Pullover
#3  Restore Oversized Crew  Cypress Vintage Crew    Cypress Vintage Crew
#4  Haven Hoodie            Restore Oversized Hood  Haven Hoodie
#5  Coronado Mixed Media    Sunday Perf. Hoodie     Restore Oversized Hood

The pattern is consistent. For the DreamKnit query, BM25 and RRF both surface DreamKnit products at the top. Vector alone drifts toward semantically soft items. For the vibe query, vector and RRF are nearly identical — the query has no exact keyword matches so BM25 contributes noise at low ranks, which RRF discounts.

This is the key property of RRF: it degrades gracefully. When one signal is useless for a query, the other dominates. When both agree, results rise. When they disagree, rank position arbitrates.

The UPF payoff:

Query: 'upf lightweight breathable sun protection'

BM25: 3/4 UPF items in top-8
Vector: 0/4 UPF items in top-8
RRF: 3/4 UPF items in top-8  ← inherits BM25's precision; vector orders the rest by feel

RRF keeps the UPF-rated items that BM25 surfaced while letting the vector signal order the rest by how light and breathable they feel. Neither signal alone gave that. Fusion is what makes "precise and on-vibe" a single ranking.

Rabbit hole: how search actually works (skippable)

We've been calling bm25.get_scores() and rrf() like magic boxes. This aside opens them up — plus the classic keyword-search machinery underneath (inverted indexes, grep/regex, trigram indexes) that predates embeddings by decades and still runs inside every serious search stack. None of it is required to use the system; it's here because knowing how the box works changes how you reach for it.

The inverted index

BM25 never scans documents at query time. It rides on an inverted index: a dictionary from each term to the list of documents that contain it (its "postings list"). "Inverted" because a normal index goes document → words; this goes word → documents.

from collections import defaultdict
 
inverted = defaultdict(set)
for i, p in enumerate(products_full):
    for tok in set(tokenize(build_chunk_full(p))):
        inverted[tok].add(i)
 
# Query: look up each term, intersect postings lists
query_tokens = tokenize("dreamknit oversized")
candidate_docs = set.intersection(*[inverted[t] for t in query_tokens if t in inverted])
# → small set of docs to score, never a full scan

BM25's three ingredients all fall out of this structure for free:

• IDF — a term's postings-list length is its document frequency. Short list → rare → high IDF.
• Term frequency — store per-posting counts.
• Length normalization — store each doc's length; divide it out so a 500-word description doesn't beat a 20-word one.

That's the whole trick: the inverted index turns "score every document" into "look up a few terms, score only the documents that contain them."

But inverted indexes only know whole words

Tokenizing into words means the index can find "dreamknit" but not the substring "reamkni", and it can't handle a pattern like "UPF followed by a number." For substrings and patterns you need regular expressions — and the naive way to run one is grep: scan every document, left to right.

import re, time
 
pattern = re.compile(r"upf\s*\d+", re.I)
 
t0 = time.perf_counter()
matches = [i for i, c in enumerate(chunks) if pattern.search(c)]
ms = (time.perf_counter() - t0) * 1000
 
print(f"grep scan: {ms:.2f}ms  →  {len(matches)} matches")
# → grep scan: 1.87ms  →  4 matches (scanned all 2,600 docs)

Even linear-in-the-input is still a full scan. To go sublinear, we need an index for substrings.

Regex is a state machine

A regex isn't matched by "trying possibilities." Compiled correctly (Thompson's construction), a pattern becomes a nondeterministic finite automaton (NFA): a graph of states with transitions on characters. Matching is walking that graph one character at a time — optionally converting the NFA to a deterministic one (DFA) so each character is a single table lookup.

The payoff is linear time in the input: each character is examined a bounded number of times, regardless of pattern complexity. This is the opposite of the backtracking engines built into most languages, where a pattern like (a+)+$ on a long non-matching string can take exponential time ("catastrophic backtracking"). Same regex syntax, radically different engine — the automaton approach is what lets grep stay fast on adversarial input.

Trigram indexes: an inverted index for substrings

The trick (used by Google Code Search, ripgrep-style tools, and Cursor's fast regex search): index every trigram — every run of 3 consecutive characters. A document containing "dreamknit" contains the trigrams dre, rea, eam, amk, mkn, kni, nit. To find a substring or run a regex, extract the trigrams the match must contain, intersect their postings lists to get a small candidate set, then run the real regex only on those candidates.

def trigrams(s: str) -> set:
    s = s.lower()
    return {s[i:i+3] for i in range(len(s) - 2)}
 
tri_index = defaultdict(set)
for i, c in enumerate(chunks):
    for tg in trigrams(c):
        tri_index[tg].add(i)
 
# Find candidates for "upf\d+" — must contain "upf"
candidates = tri_index.get("upf", set())
# Then run the real regex only on candidates — not on all 2,600 docs

It's the same idea as the word inverted index, one level down — and unlike the word index, it can find substrings that don't fall on word boundaries.

BM25, in one formula

With the inverted index in hand, BM25's score for a document d against query q is a sum over the query's terms:

score(q, d) = Σ  IDF(t)  ×  f(t,d) × (k₁+1)  /  (f(t,d) + k₁ × (1 − b + b × |d| / avgdl))

• IDF(t) — from the postings-list length; rare terms score higher.
• f(t,d) — term frequency, wrapped so it saturates: the jump from 1→2 mentions matters, 20→21 barely does.
• |d|/avgdl — length normalization.

Three knobs, all reading straight off the index. That's the entire "calibration" that makes BM25 beat raw term-frequency counting.

RRF, and why it fuses by rank not score

BM25 scores live on an unbounded positive scale; cosine similarity lives in roughly [−1, 1]. Adding them directly means whichever scale is bigger wins by default — meaningless. RRF sidesteps the problem by throwing away the scores and keeping only position:

RRF(d) = Σ  1 / (k + rank_list(d))

Because only ranks enter, the fusion is scale-invariant by construction. It works no matter how weird either scorer's numbers are — which is exactly why it generalizes to fusing three or more signals (we'll fuse a third, image, in Section 6).

How it all composes in a real stack

Production search is rarely one index. It's candidate generation → fusion → rerank:

1. Candidate generation, in parallel: an inverted index (BM25) for keywords, an ANN index (HNSW) for semantics, sometimes a trigram index for regex/substring. Each returns its top-N cheaply.
2. Fusion: RRF merges the ranked lists into one — no score reconciliation needed.
3. Rerank (optional): a slower, higher-quality model reorders the fused top-k.

This is the shape underneath Cursor's code search (trigram + vector, over turbopuffer), Shopify's product search (keyword + real-time ML embeddings), and essentially every search box you trust. The pieces are old; the vector index is the new arrival — fusion is what lets it join the old machinery instead of replacing it.

We now have two text signals — semantic meaning and exact keywords — fused into one ranking. That covers everything a shopper can type. But they're shopping for clothes, and the strongest signal for clothes isn't words at all. It's how the thing looks.

Section 5 — Image embeddings with CLIP

Everything so far has operated on text. But Vuori is an apparel brand — the product photos are doing half the selling. A shopper who types "earthy tones, relaxed fit" is describing something visual. BGE understands "earthy tones" as a concept; it doesn't see the actual color.

CLIP (Contrastive Language–Image Pretraining) was trained on 400M image-text pairs to embed images and text into the same vector space. Not similar spaces — the same space. A text query and a product image can be compared directly with cosine similarity.

This gives us a third signal: for any query, we can retrieve by visual similarity alongside keyword and semantic text matching.

How CLIP works

CLIP has two encoders trained jointly: an image encoder (ViT — a Vision Transformer) and a text encoder. During training, they learned to map matching image-text pairs close together and mismatched pairs far apart. After training, both encoders map their inputs into the same 768-dimensional space.

At index time: encode each product image → 768-dim vector, store alongside the text embedding. At query time: encode the query text with the CLIP text encoder (not BGE) → search the image embedding space.

The model: ViT-L-14 pretrained on laion2b_s32b_b82k via OpenCLIP — a large open reproduction of CLIP trained on 2B image-text pairs from the LAION dataset. 768-dim output, the same dimensionality as our BGE vectors (though a completely different learned space).

import open_clip
 
clip_model, _, clip_preprocess = open_clip.create_model_and_transforms(
    'ViT-L-14', pretrained='laion2b_s32b_b82k', device='cpu'
)
clip_tokenizer = open_clip.get_tokenizer('ViT-L-14')
clip_model.eval()
# Output dimension: 768

Encoding a single image

Before embedding the whole catalog, let's walk through what CLIP does to one image:

import torch
import torch.nn.functional as F
 
def encode_image(img: Image.Image) -> np.ndarray:
    tensor = clip_preprocess(img).unsqueeze(0)   # resize to 224×224, normalize pixels
    with torch.no_grad():
        emb = clip_model.encode_image(tensor)
    return F.normalize(emb, dim=-1).squeeze().numpy()   # L2 normalize
 
def encode_text_clip(text: str) -> np.ndarray:
    tokens = clip_tokenizer([text])
    with torch.no_grad():
        emb = clip_model.encode_text(tokens)
    return F.normalize(emb, dim=-1).squeeze().numpy()
 
# Test: image of a grey sweatshirt
img_emb  = encode_image(img)
text_emb = encode_text_clip("grey oversized crew neck sweatshirt")
mismatch = encode_text_clip("blue running shorts")
 
print(f"Image vs matching text:    {np.dot(img_emb, text_emb):.3f}")   # 0.318
print(f"Image vs mismatched text:  {np.dot(img_emb, mismatch):.3f}")   # 0.201

That's the joint embedding space at work. The image of a grey sweatshirt and the text "grey oversized crew neck sweatshirt" land close together; "blue running shorts" lands further away — even though neither BGE nor any text was involved in producing the image embedding.

Embed the full catalog

IMAGE_EMB_CACHE = "vuori_image_embs.npy"
 
if os.path.exists(IMAGE_EMB_CACHE):
    image_embs = np.load(IMAGE_EMB_CACHE)
else:
    # Fetch images in parallel (network I/O), encode sequentially (CPU)
    with ThreadPoolExecutor(max_workers=6) as pool:
        futures = {pool.submit(embed_product_image, p): p for p in products_full}
        for slug, emb in (f.result() for f in as_completed(futures)):
            if emb is not None:
                image_slugs.append(slug)
                image_emb_list.append(emb)
    image_embs = np.stack(image_emb_list)
    np.save(IMAGE_EMB_CACHE, image_embs)
 
# Image embedding matrix: (2487, 768)  — some products had no fetchable image

Image search: text query → visual results

def image_search(query: str, k: int = 10) -> list[tuple[dict, float]]:
    qvec = encode_text_clip(query)   # CLIP text encoder, not BGE
    scores = image_embs @ qvec
    top_idx = scores.argsort()[::-1][:k]
    slug_to_product = {p["slug"]: p for p in products_full}
    return [
        (slug_to_product[image_slugs[i]], float(scores[i]))
        for i in top_idx
        if image_slugs[i] in slug_to_product
    ]
 
img_results = image_search("earthy warm tones, relaxed fit, worn-in look", k=5)
vec_results = vector_search_local("earthy warm tones, relaxed fit, worn-in look", k=5)

The two lists diverge. CLIP is responding to what the product looks like in the photo — color palette, silhouette, texture. BGE is responding to what the product description says. For a visual query like "earthy warm tones," CLIP has a real advantage: it sees the photo. BGE has to infer color from words like "caramel" or "coffee" in the description, which many products don't include.

Colorway clustering

One more thing worth seeing: same-product colorways should cluster together in image space, since they share silhouette and style but differ only in color.

restore_colorways = [p for p in products_full if "Restore Oversized Crew 2.0" in p["name"]]
 
print("Restore Oversized Crew 2.0 colorway pairwise similarities (CLIP):")
# Heather Grey vs Pale Grey Heather:  0.978   ← same silhouette, slight color difference
# Heather Grey vs Forest Green:       0.965   ← same silhouette, more color difference
# Heather Grey vs Restore Hoodie:     0.843   ← different product
# Heather Grey vs Kore Short:         0.721   ← totally different garment type

Same product, different colorways — very high mutual similarity. Different products — much lower. CLIP learned to cluster by silhouette and style without any explicit supervision about product identity. This has a practical implication: if a shopper uploads a photo ("find me something like this"), CLIP can do image-to-image retrieval to find visually similar products across the catalog.

Section 6 — The production system

We've built every piece. Let's assemble them into one system Vuori could ship, then trace what each piece becomes as the catalog grows toward Amazon scale.

One ranking from three signals

Three independent ways to score a product against a query, each blind to what the others see:

RRF doesn't care that there are two lists or ten, or that their scores are on wildly different scales — it fuses by rank. So the whole system is one function: get each signal's top-N, fuse, return top-k.

def three_way_search(query: str, k: int = 10, fetch: int = 50) -> list[dict]:
    bm25_ranked = [p["slug"] for p, _ in bm25_search(query, k=fetch)]
    text_ranked = [p["slug"] for p, _ in vector_search_local(query, k=fetch)]
    img_ranked  = [p["slug"] for p, _ in image_search(query, k=fetch)]
    fused = rrf([bm25_ranked, text_ranked, img_ranked], k=60)
    slug_to_product = {p["slug"]: p for p in products_full}
    return [slug_to_product[s] for s, _ in fused[:k] if s in slug_to_product]

This is the complete system: three indexes, three signals, one fused ranking. A shopper types a messy sentence; keyword catches the specs, the text embedding catches the meaning, the image embedding catches the look, and RRF blends all three into a single list — degrading gracefully when any one signal is weak.

Now scale it up

Everything above runs on Vuori's ~2,600 products — actually on a laptop. The interesting question is what changes on the way to Amazon's 560M. The pieces don't get replaced so much as swapped for heavier-duty versions of the same idea.

At ~100K items, nothing changes. HNSW in pgvector is fine, flat scan still fast enough. At ~10M, flat scan starts to hurt — HNSW starts paying off in latency SLAs. Past ~100M, memory becomes the constraint.

Beyond RAM: disk-native indexes

At 1B+ vectors, even HNSW hits a wall: 2.8 TB of raw float32 data doesn't fit on a single machine. Two solutions:

DiskANN / Vamana (Microsoft research, popularized by Wilson Lin's blog) builds a Vamana graph — similar to HNSW, but designed from the ground up for SSD-resident data. HNSW assumes you can random-access any node in nanoseconds (RAM). Vamana accepts the 200× latency penalty of random NVMe reads and minimizes the number of reads per query instead, via RobustPrune: it removes edges that are redundant given longer-range connections, building higher-quality graphs with fewer edges — fewer reads per traversal hop.

Wilson Lin built a 1B Reddit embedding index on a single machine with 96 GB of RAM and a 4.8 TB NVMe drive. Query latency: 15ms. Equivalent HNSW setup: a 3 TB RAM cluster. 40× cheaper.

CoreNN extends DiskANN with two additions: backedge delta (reduces write amplification during inserts, so the graph stays consistent without full rebuilds) and storing full-precision vectors in RocksDB entries alongside the neighbor list (so the final re-ranking step happens in the same I/O as fetching the neighbors — no extra disk roundtrip). CoreNN achieves 2.5× fewer graph traversals than HNSW at equivalent recall on GIST1M (1M 960-dim vectors). It's what you'd build if you were starting a company like Cursor today.

S3-native: push cold data all the way down

DiskANN puts the index on NVMe. What if most namespaces are cold — queried rarely or never?

turbopuffer inverts the entire stack: S3 is the source of truth, NVMe is a warm cache, RAM is the hot tier. Its index format (SPFresh) is centroid-based, designed to minimize S3 roundtrips. A cold query costs 3 S3 roundtrips — 500–874ms p50. Warm hits NVMe cache at 10–14ms. Hot is in RAM at ~1ms.

This is Cursor's architecture. They have 80M+ code repositories (namespaces), 1T+ vectors total. Most are inactive at any moment. At $0.02/GB, 1B vectors costs $56/month on S3. At $3/GB in RAM: $8,400/month. For a multi-tenant product with a long tail of idle tenants, S3-native isn't a compromise — it's the right answer.

For Vuori's product catalog: entirely the wrong tool. You have 2,600 items, one namespace, constant queries. pgvector HNSW on Supabase is correct.

The decision table

Recap

Starting from Part 1's bare cosine-similarity search, Part 2 built a production-shaped retrieval system:

1. Enriched chunks — packed each product into one dense natural-language string (name, description, category, material, fit, features, tags) so the embedding has real signal to work with. Better input, no retraining.

2. A real index — moved from an O(n) flat scan to HNSW, the graph index that buys sublinear search. Built HNSW/IVF/PQ by hand in FAISS to understand the speed × accuracy × memory tradeoff, then got HNSW in production with one CREATE INDEX in pgvector.

3. Metadata filters — SQL WHERE for the hard constraints embeddings can't enforce (exclude dark colors, restrict category). The line between "influence ranking" and "enforce constraint" matters; SQL handles the latter.

4. Keyword search and fusion — BM25 over an inverted index for exact specs the embedding blurs, fused with the vector ranking via RRF — rank-based, so incompatible score scales never touch.

5. Images — CLIP embeddings in the same vector store, adding a signal for how a product looks. Text queries that retrieve by visual appearance.

6. Three-way fusion — BM25 + BGE text + CLIP image → one RRF ranking. The system Vuori could ship today.

The scale story: for Vuori, pgvector HNSW is the answer and will be for a long time. Push toward 10M and the question becomes read-heavy (HNSW) vs write-heavy (IVF). Past ~1B, memory is the wall — quantization (PQ, Matryoshka) compresses the vectors, and disk-native graphs (DiskANN, CoreNN) or S3-native stores (turbopuffer) move the index off RAM entirely.

Every one of those systems is the same two ideas we built here — embed into a vector space, index it for sublinear search — scaled up.

The primitives don't change. Only the price of forgetting them does.