Semantic Search: From Zero to Production

Let's say you're trying to get your girlfriend something through an online store like Vuori. Instead of scrolling endlessly for what you think she wants, you try their new chatbot. You type in what you remember her describing:

"I need some kind of oversized sweatshirt for my girlfriend, she likes it kind of cropped, a lighter color probably, cozy but also nice enough to wear out like to class or work"

You get back some generic grey sweatshirts.

In your head you can picture almost exactly what she wants. The results have nothing to do with it.

This is the kind of thing ecommerce stores are going to have to get very good at if they want their "agentic shopping experience" to actually be dialed. Luckily, semantic search can get us most of the way there. But before we get to it — what else could we try?

First approaches

Dump the entire catalog into the prompt and let the model figure it out. Behind the scenes that looks something like this:

prompt = """You are an ecommerce agent. Look at the users query and match it to what we have in the catalog:
{user_query}
{product_catalog}
"""

Problem is, a real {product_catalog} is huge — tens of thousands of items, metadata, and labels. Even if it fits in the context window (it probably won't), you're paying to process all those input tokens on every single query across potentially thousands of users. Cost. Latency.

The next option is retrieval of some sort: find the 5–10 most relevant items first, then pass just those to the model. The first instinct is keyword search — pull the terms out of the query ("oversized", "sweatshirt", "cropped", "cozy") and match against the catalog:

keywords = input_query.split()
top_10_keyword_results = search(product_catalog, keywords, k=10)
 
prompt = """You are an ecommerce agent. Look at the users query and match it to some options we keyword searched on:
{user_query}
{top_10_keyword_results}
"""

The problem is that most of that sentence doesn't live in keywords. "Cozy but nice enough to wear out to class or work" has no SKU match. "A lighter color, probably" isn't a filter you can write. "Kind of cropped" barely returns anything useful.

What we actually need is something like:

semantic_results = semantic_search(user_input, product_catalog, k=10)
 
prompt = """You are an ecommerce agent. Look at the users query and match it to some options we semantic searched on:
{user_query}
{semantic_results}
"""

In practice we wouldn't hardcode it in this order — we'd give the model a tool called semantic search and let it decide when to call it, then plug the results back in. But this is easier to explain.

This problem is everywhere

The same failure shows up anywhere you match a natural-language description against a large corpus — not just ecommerce. Cursor, the AI code editor, ran into the exact same wall. Users type "where does the authentication token get refreshed?" and the old system would grep the codebase for those words — and miss the actual function entirely, because it was called rotate_session_credentials. When they switched to embedding-based semantic search over the codebase, agent accuracy improved by 23.5% over grep alone.

Same underlying fix. Different domain. So how do we build it? We need a few basics first.

Vectors and measuring similarity

A vector is just a list of numbers that represents a point in space. Here are four clothing items placed in two dimensions, just like the x-y plane we're used to:

items = {
    "Sweatshirt": [-1.5,  1.8],
    "Sweater":    [-1.2,  1.4],
    "Tee":        [-1.8, -0.5],
    "Blazer":     [ 1.8, -1.6],
}

Notice how sweatshirt and sweater point in the same direction — more casual loungewear — while blazer is off in the opposite corner, more dressy, and tee sits in between, dressable up or down. In this simple 2D example our axes are "representing" how casual vs dressy something is. Pretty simple.

Now try to imagine adding a third dimension for color. Then a fourth for fit. And so on — not so simple! Good news is, regardless of dimensionality, we can measure the "similarity" of direction using cosine similarity.

import numpy as np
 
def cosine_similarity(a, b):
    a, b = np.array(a), np.array(b)
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
 
query_vec = [-2.0, -2.0]  # "long sleeve tee"
 
for name, vec in items.items():
    score = cosine_similarity(query_vec, vec)
    print(f"{name:25} {score:.3f}")

Sweatshirt                -0.091
Sweater                   -0.077
Tee                        0.871
Blazer                    -0.059

Remember our axes represent how casual vs dressy something is — and "casual vs dressy" is really a direction in this space. So the thing we care about isn't how close two points are, it's whether they point the same way. That's exactly what cosine similarity measures: the angle between two vectors, ignoring how long they are.

That's why the query landing at [-2, -2] scores 0.87 against tee even though it isn't sitting right on top of it — they point in the same direction, and that's the whole thing.

Dissecting cosine similarity

It's worth unpacking the pieces a little more — they show up all over AI.

To start, the dot product: multiply the two vectors position-by-position and add it all up.

a · b = a₁b₁ + a₂b₂

For our query and the tee:

[-2, -2] · [-1.8, -0.5] = (-2)(-1.8) + (-2)(-0.5) = 3.6 + 1.0 = 4.6

a = np.array([-2.0, -2.0])   # query
b = np.array([-1.8, -0.5])   # Tee
print("dot product:", np.dot(a, b))          # 4.6

So what does 4.6 actually mean? Here's the identity that makes it click:

a · b = ||a|| ||b|| cos(θ)

The dot product is big and positive when two vectors point the same way, near zero when they're perpendicular, and negative when they point opposite ways. It's already a similarity signal on its own — query·tee comes out to +4.6 (same direction), while query·blazer is negative (pointing away).

The catch is that the dot product is tangled up with length. Make a vector twice as long and its dot product doubles without the direction changing at all. So the dot product answers "same direction and how big" — but for similarity we only care about the "same direction" part. That's what the magnitude fixes.

Magnitude (also called the norm) is just the length of a vector — good old Pythagoras:

||v|| = sqrt(v₁² + v₂²)      # np.linalg.norm(v)

The key move is getting unit vectors. A unit vector is what you get when you divide a vector by its own magnitude — v / ||v||. It points the exact same direction, but its length is now 1. When every vector is scaled to length 1 they all land on the unit circle (a sphere in higher dimensions), and cosine similarity equals the plain dot product — because both magnitude terms in the denominator are now 1.

a_unit = a / np.linalg.norm(a)
b_unit = b / np.linalg.norm(b)
print("dot(unit_a, unit_b):", np.dot(a_unit, b_unit).round(3))
print("cosine_similarity:  ", cosine_similarity(a, b).round(3))   # same number

dot(unit_a, unit_b): 0.871
cosine_similarity:   0.871

Matrix multiplication — scoring the whole catalog at once

While we're here: matrix multiplication. So far we've looped over the catalog one item at a time. Fine for 4 items, miserable for 4 million. The fix is to stack the whole catalog into a matrix and score it in one shot — a matmul is really just a pile of dot products stacked up. Put each catalog vector on its own row, multiply by the query, and each row's dot product falls out as one entry of the result.

M = np.array(list(items.values()))
M_unit = M / np.linalg.norm(M, axis=1, keepdims=True)
q_unit = np.array(query_vec) / np.linalg.norm(query_vec)
sims = M_unit @ q_unit
for name, s in zip(items.keys(), sims):
    print(f"{name:12}  {s:.3f}")

Sweatshirt    -0.091
Sweater       -0.077
Tee            0.871
Blazer        -0.059

Same numbers, one operation.

The faded arrows are the original vectors at their real lengths; the solid ones are the same vectors normalized onto the circle. Notice sweatshirt and sweater land basically on top of each other — they were nearly the same direction all along — while the query and tee sit close together and the blazer is way off on the other side. Length is gone; only the angle survived.

Once you scale every vector onto the unit circle — that's L2 normalization, dividing each vector by its own length so it lands exactly at distance 1 from the origin — the physical distance between two points and the angle between them become the same signal. Closer points = smaller angle = higher cosine similarity. They're all measuring the same thing.

This is why vector search is described as finding the "nearest neighbor" — on the unit sphere, nearest in distance and most similar in direction are identical. And it's why the term ANN (Approximate Nearest Neighbor) shows up whenever vector search scales up. Exact search scores every vector against your query — fine for hundreds of items, too slow for millions. ANN indexes exploit the geometry of this space to check a smart subset instead of everything, returning the correct answer ~90–95% of the time at a fraction of the cost. For search that tradeoff is almost always worth it — a user can't tell the difference between the true #1 result and the true #3.

And it's why these pieces show up all over AI: the dot product is the atom, a matmul is just a bunch of them stacked together, and GPUs are basically machines built to do enormous matmuls quickly. Attention, linear layers, embedding lookups — matmuls all the way down.

Embedding models — placing vectors automatically

So we know what a vector is and how to measure similarity between them. But we placed those vectors by hand — two axes, four items. A real catalog has thousands of items and meaning that lives across hundreds of dimensions: fit, fabric, occasion, color palette, aesthetic, and a lot more that doesn't even have a clean name.

You can't place those by hand. An embedding model learns them automatically from data. Show the model millions of pairs of text that are similar — product descriptions and their tags, sentences that mean the same thing, questions and their answers. For each pair, push their vectors closer together. For non-matches, push them apart. Do that enough times and the geometry self-organizes: "cozy and worn-in" ends up near "faded fleece pullover" even if those exact words never appeared together in training.

The result is a function: text in, vector out. Let's put one to work on a real catalog.

Keyword search on the real Vuori catalog

Toy catalogs are useful for building intuition, but they're rigged — we hand-placed the items and wrote the descriptions ourselves. Let's test on a real store.

Vuori is a good target: a premium activewear brand with a large catalog and the kind of nuanced language ("worn-in," "relaxed silhouette," "elevated casual") that keyword search struggles with. Their site is a Next.js app backed by Shopify, which means every product page has a structured _next/data endpoint — no HTML scraping needed.

We'll keep this section dead simple: just product names and descriptions. No metadata, no tags, no fit notes — raw descriptions only. We'll run the original query through keyword search, then through semantic search, and compare. Part 2 will show what enriching those descriptions with structured metadata actually buys us.

Step 1 — Get all product URLs from the sitemap

Vuori publishes a standard XML sitemap at /sitemap.xml. Every product has a URL of the form /products/{slug}. We pull the sitemap and parse out every product slug — all 2,600+ of them, every category.

Step 2 — Fetch name and description for every product

Vuori's Next.js app pre-renders each product page and serves its data from /_next/data/{buildId}/products/{slug}.json. The build ID changes on each deploy, so we grab it from the homepage first. We only extract what we need for this section: name, description, color, category, and URL.

A sample product, to see what we're working with:

{
  "slug": "outdoor-trainer-shell-azure-linen-texture",
  "name": "Outdoor Trainer Shell",
  "color": "Azure Linen Texture",
  "description": "Take on the great outdoors with our men's hooded Outdoor Trainer Shell jacket. Crafted from lightweight, 4-way stretch performance material, it's water-resistant, moisture-wicking, and ready for anything.",
  "category": "Jackets & Hoodies",
  "url": "https://vuoriclothing.com/products/outdoor-trainer-shell-azure-linen-texture"
}

Step 3 — Keyword search

Split the query into terms, count hits per item across name + description. No weighting, no IDF — just raw term frequency. The item with the most hits wins.

query = (
    "I need some kind of oversized sweatshirt or hoodie for my girlfriend, "
    "she likes it kind of cropped, a lighter color probably, cozy but also "
    "nice enough to wear out like to class or work"
)
 
def tokenize(text):
    return re.findall(r'\b\w+\b', text.lower())
 
def keyword_search(query, catalog, k=5):
    query_terms = tokenize(query)
    results = []
    for item in catalog:
        doc_tokens = tokenize(item["name"] + " " + item["description"])
        score = sum(doc_tokens.count(t) for t in query_terms)
        results.append((score, item))
    return sorted(results, reverse=True, key=lambda x: x[0])[:k]
 
top_keyword = keyword_search(query, products_deduped, k=5)

The text breakdown — what actually scored:

#1  [22]  ALOHA-Mini Hip Pack
    term hits: 4×"to", 3×"of", 2×"a", 2×"out", 1×"need", 1×"it"
#2  [19]  ALOHA-Day Tripper
    term hits: 4×"to", 3×"of", 2×"a", 2×"out", 1×"it"
#3  [16]  Women's Taika Snow Shell
    term hits: 7×"to", 1×"a", 1×"it"
#4  [15]  Stormbreak Jacket
    term hits: 3×"for", 3×"to", 2×"out", 1×"a", 1×"work", 1×"or"
#5  [15]  Ventana Jacket
    term hits: 4×"a", 2×"it", 1×"of", 1×"for", 1×"work", 1×"or"

The top results are an ALOHA hip pack, a tote bag, and a pile of jackets — for a sweatshirt query. Not a single sweatshirt in sight. Look at what's actually scoring: 4×"to", 3×"of", 2×"a". These are stopwords — "some kind of", "to class or work", "a lighter color" — matching common words in a brand backstory. The ALOHA collection has a long "founded in Cardiff-by-the-Sea" paragraph that's full of them.

Whole-word matching fixed the character-level noise, but the core problem remains: the query is full of filler words, and filler words appear everywhere. The search has no way to know that "to" and "of" are meaningless here — or that "sweatshirt," the one word that actually matters, should count for far more than a stopword that shows up two dozen times in a brand backstory.

Keyword search is a word-matching machine. It cannot bridge vocabulary gaps. The fix isn't to filter stopwords by hand — it's to weight terms by how rare and informative they are across the whole catalog. A term that appears in 800 of 879 products tells you almost nothing; a term that appears in 12 does. That's the idea behind BM25, and it's what we'll use in Part 2 to make the keyword side of hybrid search actually useful.

Semantic search on the same catalog

Same query. Same catalog. Different approach.

Instead of counting words, we embed everything — query and products — into the same 768-dimensional space, then score with a single matmul. The model was trained to push similar meanings together, so "cozy worn-in pullover" and "oversized crewneck sweatshirt, all about coziness" land near each other even with almost no word overlap.

from sentence_transformers import SentenceTransformer
import numpy as np
 
model = SentenceTransformer('BAAI/bge-base-en-v1.5')
 
# One chunk per product: name + description (same as keyword search — raw only)
texts = [f"{p['name']} — {p['description']}" for p in products_deduped]
embeddings = model.encode(texts, normalize_embeddings=True)
# shape: (879, 768)
 
# Encode the query into the same space, then score all 879 products in one matmul
query_vec = model.encode(f"query: {query}", normalize_embeddings=True)
scores = embeddings @ query_vec
 
top_idx = np.argsort(scores)[::-1][:5]
top_semantic = [(float(scores[i]), products_deduped[i]) for i in top_idx]

Results:

#1  [0.699]  Restore Oversized Crew 2.0
    ...this oversized crewneck sweatshirt is all about coziness. Crafted from our ultra-plush organic cotton fleece...

#2  [0.681]  Restore Double Zip Sweatshirt
    ...destined for chill afternoons and keeping cozy on cool nights. An oversized...

#3  [0.680]  Beach Fleece Pullover Hoodie
    ...a stylish men's sweatshirt designed for next-level comfort...

#4  [0.672]  Boyfriend Sweatpant
    ...made with our super soft brushed jersey DreamKnit™, two side pockets, a mid-rise elastic waist...

#5  [0.666]  Restore Oversized Hoodie
    ...made with brushed French terry that's been treated with a bio wash for a broken-in feel...

That's the jump. Rank 1 is the Restore Oversized Crew 2.0 — "oversized crewneck sweatshirt, all about coziness, ultra-plush organic cotton fleece." Ranks 2, 3, and 5 are the same idea in different cuts — a double-zip sweatshirt, a fleece pullover hoodie, an oversized hoodie. Exactly the "oversized sweatshirt or hoodie, cozy, nice enough for class or work" she asked for.

None of those descriptions share the vocabulary of the query. "Nice enough to wear out to class or work," "a lighter color probably" — none of that phrasing appears in the product text. The model found them anyway because it understood what those phrases mean and matched that meaning to the right region of the catalog. Compare that to keyword search's top result: an ALOHA hip pack, ranked first because its brand story happened to contain "to," "of," and "a" a lot.

There's one honest limitation worth naming: at #4 the model slipped in a Boyfriend Sweatpant, right in the middle of the sweatshirts — because "cozy, relaxed, soft fleece" describes both. They're genuinely close in embedding space. The model got the vibe right and the garment type mostly right, but it can't perfectly separate sweatshirt from sweatpant on description text alone. Fixing that — and taking this system from a working prototype to something production-grade — is what Part 2 covers.

Aside — can we name a direction in this space?

We keep saying the 768 dimensions encode meaning — "fit," "fabric," "occasion." But no single dimension is labeled; meaning is smeared across many of them. So here's a fair question: if a concept like how dressy something is really lives in this space, can we point at it?

We can, and the trick is simple. Embed a couple of phrases that clearly mean dressy ("nice enough to wear out to dinner") and a couple that mean the opposite, gym ("sweaty workout training"), then take the difference of their averages. That difference is a direction: the way you'd walk through the space to go from gym-wear toward dressy-wear. Project any product onto it and you get a score — and just like a bulldog is "more dog" than a poodle is, a twill trouser turns out to be "more dressy" than a pair of running tights.

def concept_axis(pos_phrases, neg_phrases):
    """A named direction = mean(positive embeddings) − mean(negative embeddings)."""
    pos = model.encode(pos_phrases, normalize_embeddings=True).mean(axis=0)
    neg = model.encode(neg_phrases, normalize_embeddings=True).mean(axis=0)
    axis = pos - neg
    return axis / np.linalg.norm(axis)
 
dressy_axis = concept_axis(
    ["dressy elevated polished nice enough to wear out to dinner"],
    ["sweaty gym workout athletic training exercise"],
)
 
# Score every product on the axis, then sample evenly across the whole range
score = embeddings @ dressy_axis

Nothing told the model what "dressy" means — we just did a bit of arithmetic on a few word vectors, and a real, recognizable axis fell out of the space. It's a probe, not proof the model has a dedicated "dressy neuron" — these directions are approximate and not perfectly independent — but it's a striking amount of structure for zero supervision.

Visualizing the embedding space

879 items × 768 dimensions is impossible to look at directly. t-SNE collapses it to 2D while trying to preserve neighborhood structure — items that sit close in the 768-d space end up close on the plot.

We'll look at it twice. First the honest way every paper shows it: dots, colored by the category Vuori assigns each product — enough to see whether the space organized itself.

Then we swap every dot for the product's actual photo, because "a cluster of blue dots" and "a cluster of fleece hoodies" are very different levels of convincing.

Two things jump out.

The full catalog self-organized without ever being told what a category is — scan the plot and you'll see jackets pooling in one region, shorts and pants in another, bags and hats drifting to their own edges. That structure came purely from description text run through the embedding model.

The zoom is the payoff. The query star lands in a pocket that's wall-to-wall cozy knitwear — Restore crews, zip hoodies, fleece pullovers. Every nearest neighbor is the kind of thing the query asked for. You can also see the limitation with your own eyes: a few sweatpants and joggers sneak into the neighborhood, because "cozy, relaxed, soft fleece" describes them just as well as a sweatshirt, and on description text alone the model can't fully separate the two garment types.

Still — we went from an ALOHA hip pack at #1 to Restore Oversized Crew at #1, using nothing but a pre-trained embedding model and a matrix multiply. No labeled data, no hand-tuned rules, no product taxonomy. That's a meaningful result from a simple system.

One more thing — search was never the only use

Everything we built points text → catalog: embed a query, find the nearest items. But look at what the machinery actually is — embed something, cosine-similarity against the catalog, take the top-k. Nothing about that says the "something" has to be a search query.

Feed it a customer's purchase history instead and the exact same three lines become a recommendation engine. Same embeddings, same matmul, same nearest-neighbor lookup — different input. Say a shopper has bought these three cozy, casual-athletic pieces:

# Grab three items the shopper "bought" — indices into the catalog
def find(substr):
    return next(i for i, p in enumerate(products_deduped)
               if substr.lower() in p["name"].lower())
 
basket = [find("Venture Quarter Zip"), find("Strato Tech Tee"), find("Denver Waffle Crew Sweater")]

Recommendation #1 — "complete the kit" (a taste vector)

Average the three purchase vectors into a single taste vector — one point that sits in the middle of everything they've bought — then find the catalog items nearest to that. It's the customer's style, distilled to 768 numbers.

def recommend(vector, exclude, k=5):
    """Nearest catalog items to a vector, skipping owned items + colorway dupes."""
    vector = vector / np.linalg.norm(vector)
    sims = embeddings @ vector
    seen = {products_deduped[i]["name"] for i in exclude}
    out = []
    for i in np.argsort(sims)[::-1]:
        if i in exclude or products_deduped[i]["name"] in seen:
            continue
        seen.add(products_deduped[i]["name"])
        out.append((float(sims[i]), products_deduped[i]))
        if len(out) >= k:
            break
    return out
 
taste = embeddings[basket].mean(axis=0)   # their average taste
taste_recs = recommend(taste, exclude=set(basket))

The taste vector blends the whole basket, so the picks span their look — more Strato tees, a Coronado half-zip and a Denver full-zip sweater, a Strato hoodie. It's "here's your kit," not "here's ten of the thing you just bought."

Recommendation #2 — "more like this" (a single anchor)

Want the tighter version? Drop the averaging and point at one item. This is the "customers also viewed" strip on every product page — and it's just a taste vector with a basket of one.

anchor = basket[1]  # the Strato Tech Tee
like_this = recommend(embeddings[anchor], exclude={anchor})

And there it is: anchored on the Strato Tech Tee, you get the Strato family — tank, hoodie, V-neck, muscle tee. Narrower and more literal than the taste-vector picks, exactly as you'd want for a "more like this" shelf.

Two features — personalized recommendations and "customers also viewed" — and neither needed a new model, a new index, or a single line of code you haven't already seen. Search, recommendations, related items: all the same embedding, the same cosine similarity, the same matmul. That's the payoff of getting the retrieval layer right once.

What's next

This is a working system. Not a toy — a real embedding model, a real product catalog, real queries, and results that are meaningfully better than keyword search. But it's also the simplest version of the idea.

Part 2 goes deeper, for engineers who want to actually build this:

• Enriched chunks — name + description + category + material + fit notes + Shopify tags. Better input signal, better retrieval.
• A real index — Supabase pgvector with HNSW. Metadata filtering, namespaces, persistence.
• BM25 — what keyword search looks like when it's done right. Calibrated scores, IDF weighting, document-length normalization.
• RRF — fusing keyword + vector signals without score-scale incompatibility. Rank-based fusion.
• Image embeddings — CLIP ViT-L/14 giving each product a visual fingerprint. Text queries that retrieve by appearance, not just description.
• 3-way hybrid — BM25 + text embedding + image embedding, fused with RRF. The system that catches what each individual signal misses.
• Index architecture deep-dive — when does flat search break? IVF vs HNSW vs quantization vs S3-native indexes. Napkin math at each scale checkpoint.

Closer

We started with a chatbot that returned grey sweatshirts.

We built a system that understands "oversized, cropped, cozy, a lighter color, nice enough for class or work" — and finds the right item even when none of those words appear in the product description. The same primitives — embeddings, cosine similarity, matmul — power Cursor's codebase search, Shopify's product recommendations, Spotify's music discovery.

The retrieval layer is the foundation. Part 2 builds on it.