Skip to content

Deep dive: Vectorization & vector databases ​

Optional deep dive

This page is not required for the workshop. It supports Part 2, Context engineering, specifically the RAG section, but stands alone. Take it at any pace, in any order. All scripts are shown complete on this page — no extra downloads needed.

What you will learn ​

  • What an embedding is and why it is how models represent meaning
  • How similarity search works geometrically
  • What a vector database does and which Azure services support vector search
  • How to do vector arithmetic in Python with a Word2Vec model, with an office twist

Concept: From words to numbers ​

A language model does not read text the way you do. Before it can reason about meaning, every piece of text is converted into a list of floating-point numbers called an embedding (also called a vector). An embedding is a point in a high-dimensional space. The key property: similar meanings land close together.

TextNearby in embedding space
"paper"document, sheet, report, invoice
"warehouse"storage, dock, pallet, distribution
"manager"supervisor, director, executive, boss
"Scranton"Pennsylvania, Pittsburgh, Wilkes-Barre

Nothing in the model was hand-coded to know that "manager" and "supervisor" are similar. The model learned it by reading billions of sentences where those words appeared in similar contexts. The result is a space where context encodes meaning geometrically.

Two generations of embeddings ​

TypeExamplesHow it worksLimitation
Word-level (fixed)Word2Vec, GloVeEach word gets one static vector regardless of context"paper" means the same thing in "paper round" and "paper company"
Context-awareOpenAI text-embedding-3-*, sentence transformersThe vector for a word depends on its surrounding sentenceMore accurate; required for production RAG

The exercises below use GloVe (word-level) because it is small and fast to load. The intuition it teaches applies directly to context-aware embeddings.


Concept: Vector arithmetic ​

Because meaning is geometric, you can navigate the embedding space with addition and subtraction.

The famous example:

$$\vec{\text{king}} - \vec{\text{man}} + \vec{\text{woman}} \approx \vec{\text{queen}}$$

What is happening: subtracting man removes the "male royalty" direction; adding woman inserts the "female" direction; the nearest point in the whole vocabulary is queen. The model was never told this. It emerges from the geometry of how words co-occur in text.

The same pattern works for many relationships:

AnalogyTypical result
Paris - France + GermanyBerlin
walking - walk + runrunning
manager - man + womansupervisor, executive

Results are approximate, not exact. The fun is in exploration, not in memorising outputs.


Concept: Similarity search and vector databases ​

The cosine similarity measure ​

The standard way to measure how close two vectors are is cosine similarity: the cosine of the angle between them. Two identical vectors have a cosine similarity of 1; two completely unrelated vectors are near 0; true opposites approach -1.

$$\text{similarity}(A, B) = \frac{A \cdot B}{|A| |B|}$$

For RAG retrieval, you embed the user's query and then find the stored chunks whose vectors have the highest cosine similarity to the query vector.

Why exact search does not scale ​

Comparing a query vector against one million stored vectors one by one (brute-force nearest neighbour) is $O(n)$ per query. At 1.5 billion documents that becomes prohibitively slow.

Approximate nearest neighbour (ANN) indices ​

Vector databases use index structures that trade a small amount of recall for orders-of-magnitude faster search. The most common algorithm today is HNSW (Hierarchical Navigable Small World): it builds a multi-layer graph of vectors and navigates it like a highway system, skipping most of the comparisons.

In practice, recall losses are negligible (often 99 %+) and query times drop from seconds to milliseconds.

Where the vector database sits in a RAG pipeline ​

User question
  │
  â–¼  embed with the same model used to embed the documents
Query vector
  │
  â–¼  ANN search (cosine similarity) against the vector store
Top-k most similar document chunks
  │
  â–¼  inject into the context window as "retrieved data"
Model answer

This is exactly the "retrieve on demand" decision from your context map in Part 2: the vector store is the mechanism that executes that decision efficiently at runtime.


Vector search in Azure ​

Azure offers vector search across seven services. The right choice depends on your existing infrastructure, latency requirements, and how much you need advanced features like hybrid search and semantic ranking.

Architecture guide

The Microsoft Architecture Center article Choose an Azure Service for Vector Search contains a decision flowchart and a full capability matrix covering indexing algorithms, distance metrics, and Azure OpenAI integration for each service. Use it when making a production choice. The table below is a practical summary.

ServiceBest fitANN indexHybrid search
Azure AI SearchUnstructured content, RAG pipelines, semantic rankingHNSWFirst-class
Azure Cosmos DB for NoSQLNoSQL document workloads with vector fieldsDiskANN, quantizedFlatYes
Azure DocumentDBMongoDB-compatible workloadsHNSW, IVFFlat, DiskANNYes, native RRF
Azure Cosmos DB for PostgreSQLPostgreSQL teams; uses the pgvector extensionHNSW, IVFFlatManual
Azure Database for PostgreSQLManaged Postgres; pgvector + DiskANN extensionHNSW, IVFFlat, DiskANNManual
Azure Managed RedisUltra-low latency, in-memory search; Redis already in stackHNSWYes
Azure SQL DatabaseRelational data with embedded vector columnsDiskANN (preview)Manual

Embeddings: always Azure OpenAI ​

Regardless of which vector store you choose, the embedding model is a separate concern. For Azure-native stacks, all three models below are available in your pre-deployed Azure OpenAI resource. Docs: Azure OpenAI embeddings how-to.

ModelDimensionsNotes
text-embedding-3-small1 536Best price/quality ratio; recommended for new projects
text-embedding-3-large3 072Higher accuracy; larger budget
text-embedding-ada-0021 536Legacy; prefer 3-small for new work

Which to use for the Dunder Mifflin agent?

Azure AI Search is the natural fit for the package manager scenario: unstructured warehouse policy documents, hybrid keyword + vector search over order data, and first-class Semantic Kernel integration. If your team already runs PostgreSQL, Azure Database for PostgreSQL with pgvector is the lowest-friction path to the same result.


Lab: Word2Vec warm-up ​

Setup ​

bash
pip install gensim numpy

The first api.load(...) call downloads and caches the model (~66 MB for the 50-dimension GloVe vectors). Subsequent runs load from the local cache.

python
# warm_up.py
import gensim.downloader as api

print("Loading GloVe vectors (first run downloads ~66 MB)...")
model = api.load('glove-wiki-gigaword-50')
print(f"Vocabulary size: {len(model):,} words\n")

# --- 1. Find neighbours of a word ---
print("Neighbours of 'paper':")
for word, score in model.most_similar('paper', topn=8):
    print(f"  {word:<15} {score:.3f}")

print("\nNeighbours of 'warehouse':")
for word, score in model.most_similar('warehouse', topn=8):
    print(f"  {word:<15} {score:.3f}")

# --- 2. Measure similarity between two words ---
pairs = [
    ('sell', 'ship'),
    ('manager', 'supervisor'),
    ('paper', 'forklift'),
    ('package', 'parcel'),
]
print("\nPairwise similarity:")
for a, b in pairs:
    print(f"  {a} / {b}: {model.similarity(a, b):.3f}")

Questions to discuss:

  • Which words are neighbours of "paper" that surprised you?
  • Is "sell" more similar to "ship" or to "buy"? Does that match your intuition?
  • What does the low similarity between "paper" and "forklift" tell you about the embedding space?

Lab: The classic analogy (anchor) ​

Before exploring office-themed analogies, run the famous one to confirm the model behaves as expected.

python
# classic_analogy.py
import gensim.downloader as api

model = api.load('glove-wiki-gigaword-50')

def analogy(positive, negative, topn=4):
    """Print the top results for: positive[0] - negative[0] + positive[1]"""
    label = f"{positive[0]} - {negative[0]} + {positive[1]}"
    results = model.most_similar(positive=positive, negative=negative, topn=topn)
    print(f"\n{label}:")
    for word, score in results:
        print(f"  {word:<15} {score:.3f}")

# Classic: should return 'queen' near the top
analogy(positive=['king', 'woman'], negative=['man'])

# Geographic: Paris is to France as __ is to Germany
analogy(positive=['paris', 'germany'], negative=['france'])

# Plural: mouse is to mice as goose is to ___
analogy(positive=['mice', 'goose'], negative=['mouse'])

Expected output (approximate):

king - man + woman:
  queen           0.849
  monarch         0.789
  princess        0.782

paris - france + germany:
  berlin          0.928
  munich          0.841
  vienna          0.835

The exact scores vary slightly between GloVe versions, but queen and berlin should appear near the top. If they do not, confirm you loaded glove-wiki-gigaword-50, not a different model.


Lab: Office-themed analogies ​

Now apply the same mechanics to the Dunder Mifflin world. These analogies are exploratory, not guaranteed, which is part of the lesson: vector arithmetic reveals learned associations, not hard-coded facts.

python
# office_analogies.py
import gensim.downloader as api

model = api.load('glove-wiki-gigaword-50')

def analogy(positive, negative, topn=5, label=None):
    if label is None:
        label = f"{positive[0]} + {positive[1]} - {negative[0]}"
    results = model.most_similar(positive=positive, negative=negative, topn=topn)
    print(f"\n{label}:")
    for word, score in results:
        print(f"  {word:<18} {score:.3f}")

# 1. Workplace role shift: what is to a warehouse what a salesperson is to an office?
analogy(
    positive=['salesperson', 'warehouse'],
    negative=['office'],
    label="salesperson + warehouse - office"
)

# 2. Paper company pivot: selling paper vs shipping packages
analogy(
    positive=['paper', 'ship'],
    negative=['sell'],
    label="paper + ship - sell"
)

# 3. Scranton geography: Scranton is to Pennsylvania as ___ is to California
try:
    analogy(
        positive=['scranton', 'california'],
        negative=['pennsylvania'],
        label="scranton - pennsylvania + california"
    )
except KeyError:
    print("\n'scranton' not in vocabulary (proper nouns are model-dependent)")
    # Fallback with a word more likely to be in the vocab
    analogy(
        positive=['pittsburgh', 'california'],
        negative=['pennsylvania'],
        label="pittsburgh - pennsylvania + california (fallback)"
    )

# 4. Authority gradient: boss minus dominance, plus service
analogy(
    positive=['boss', 'service'],
    negative=['authority'],
    label="boss + service - authority"
)

# 5. Explore your own
print("\n--- Your turn ---")
print("Try your own analogy below and see what comes back.\n")

YOUR_POSITIVE = ['manager', 'delivery']   # change these
YOUR_NEGATIVE  = ['sales']                # and this
analogy(positive=YOUR_POSITIVE, negative=YOUR_NEGATIVE, label="your analogy")

Discussion prompts:

  • Analogy 1: did the result feel like a role you would find in a warehouse?
  • Analogy 2: what does the shift from sell to ship say about the model's representation of the paper-company context?
  • Analogy 3: if scranton was not in the vocabulary, what does that tell you about the limits of word-level embeddings for proper nouns? (Hint: context-aware models embed whole phrases, which handles this better.)
  • What analogy of your own did you try? Did it work?

Lab (stretch): Visualize word clusters with PCA ​

Project 50-dimensional vectors down to 2D to see how words cluster visually.

bash
pip install matplotlib scikit-learn
python
# visualize_clusters.py
import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
import gensim.downloader as api

model = api.load('glove-wiki-gigaword-50')

# Words grouped by Dunder Mifflin category
categories = {
    'Office roles':    ['manager', 'salesperson', 'accountant', 'receptionist', 'boss', 'employee'],
    'Warehouse roles': ['forklift', 'pallet', 'loading', 'dock', 'shipping', 'distribution'],
    'Products':        ['paper', 'package', 'parcel', 'invoice', 'delivery', 'shipment'],
}
colours = {'Office roles': '#4C9BE8', 'Warehouse roles': '#E8944C', 'Products': '#5DBF6E'}

# Collect words that exist in the vocabulary
words, vecs, cats = [], [], []
for category, word_list in categories.items():
    for word in word_list:
        if word in model:
            words.append(word)
            vecs.append(model[word])
            cats.append(category)

# Project to 2D
coords = PCA(n_components=2).fit_transform(np.array(vecs))

# Plot
fig, ax = plt.subplots(figsize=(11, 7))
for i, (word, cat) in enumerate(zip(words, cats)):
    ax.scatter(coords[i, 0], coords[i, 1], color=colours[cat], s=80, zorder=2)
    ax.annotate(word, xy=(coords[i, 0], coords[i, 1]),
                xytext=(5, 4), textcoords='offset points', fontsize=9)

# Legend
for cat, colour in colours.items():
    ax.scatter([], [], color=colour, label=cat, s=80)
ax.legend(loc='best')
ax.set_title("Dunder Mifflin words in 2D (PCA of GloVe-50)")
ax.set_xlabel("PC 1")
ax.set_ylabel("PC 2")
plt.tight_layout()
plt.savefig("dunder_mifflin_vectors.png", dpi=150)
print("Saved: dunder_mifflin_vectors.png")
plt.show()

What to look for:

  • Do office-role words cluster away from warehouse-role words?
  • Where does "shipment" land, closer to warehouse or to products?
  • Add a few words from one category and see if they pull towards the expected cluster or land somewhere unexpected.

Why this matters for RAG

The clusters you see here are the reason RAG retrieval works: a query about "fragile package handling" embeds near the product and warehouse clusters, so the retrieval step pulls the right policy sections without you having to write a keyword query. The distance in the plot is the distance that cosine similarity is measuring at retrieval time.


Checkpoint ​

  • You can explain in one sentence what an embedding is and why similar meanings are close together.
  • You ran the classic king - man + woman analogy and got queen near the top.
  • You ran at least two office-themed analogies and can explain one surprising result.
  • You can describe the role of the vector database in the RAG pipeline from the diagram above.
  • (Stretch) You produced the word-cluster plot and identified at least one unexpected placement.

Fallback baseline ​

This deep dive has no separate fallback files — every script (warm_up.py, classic_analogy.py, office_analogies.py, visualize_clusters.py) is shown complete in the sections above, so you can copy each one straight from this page.


Further reading ​

ResourceWhy it is relevant
Word2Vec original paper (Mikolov et al., 2013)The paper that introduced the king-man+woman=queen analogy and the skip-gram/CBOW training methods
GloVe: Global Vectors for Word Representation (Pennington et al., 2014)The model used in the exercises; explains why global co-occurrence statistics improve on Word2Vec
Azure OpenAI embeddings how-toHow to call text-embedding-3-small from Python or REST; the same API your pre-deployed Azure resource exposes
Azure AI Search vector search overviewHNSW-backed vector indexing, hybrid search, and semantic ranking in the Azure AI Search service
HNSW paper (Malkov & Yashunin, 2018)The graph-based ANN algorithm used by most production vector databases today, including Azure AI Search
Choose an Azure Service for Vector SearchMicrosoft Architecture Center decision guide with flowchart and capability matrix comparing all seven Azure vector search services side by side

Previous: Go deeper overview | Workshop: Part 2, Context engineering