Chrysolites Tech Academy

Data and Search for AI Apps · Course book

Chapter 1: Meaning as Numbers

Pairs with Lesson 1: Vector Databases and Search by Meaning

Ordinary search matches letters. Type "car" and you get documents containing the letters c, a, r; a page about "automobiles" is missed. Modern AI systems search by meaning instead, and they do it by turning text into lists of numbers and comparing those numbers. This chapter explains how meaning becomes numbers, how closeness between them is measured, and what a vector database adds on top.

Vectors and what closeness means

A vector is an ordered list of numbers. With two numbers you can draw it as an arrow on a flat sheet: the vector (3, 4) points 3 units across and 4 units up. With three numbers the arrow lives in space. With hundreds of numbers nobody can picture it, but the arithmetic works the same way, and that is all a computer needs.

The key idea is that a vector can stand for a piece of text in such a way that texts with similar meanings get arrows pointing in similar directions. Figure 1.1 shows a toy version with only two numbers per word.

cat (3, 4)dog (4, 3)car (4, -3) 044-3 cat and dog: small angle,similarity 0.96cat and car: right angle,similarity 0
Figure 1.1 — Three toy word vectors. Words with related meanings point in nearly the same direction; unrelated words point in very different directions.

The usual measure of closeness is cosine similarity, which depends only on the angle between two arrows and ignores their lengths. It is found in three steps.

  1. Compute the dot product: multiply matching numbers and add the results. For cat (3, 4) and dog (4, 3) this is 3 × 4 + 4 × 3 = 24.
  2. Compute each vector's length: square the numbers, add them and take the square root. For cat this is the square root of 9 + 16 = 25, which is 5. Dog also has length 5.
  3. Divide the dot product by the two lengths multiplied together: 24 ÷ (5 × 5) = 0.96.

A result of 1 means the arrows point the same way, 0 means they are at right angles and -1 means they are opposite. Cat and dog score 0.96, which is very close. Now try cat (3, 4) and car (4, -3): the dot product is 3 × 4 + 4 × (-3) = 12 - 12 = 0, so the similarity is 0. As far as these toy vectors are concerned, the two words have nothing in common.

10-1 0°60°90°180°angle between the two vectors same direction0.5unrelatedopposite
Figure 1.2 — Cosine similarity as the angle between two vectors opens. The smaller the angle, the closer the score is to 1.
MeasureHow it is calculatedClosest pair hasAffected by vector length?
Cosine similarityDot product divided by both lengthsThe highest score, up to 1No
Dot productMultiply matching numbers and addThe highest scoreYes
Euclidean distanceStraight-line distance between the arrow tipsThe lowest distance, down to 0Yes

For cat and dog the Euclidean distance is the square root of (3 - 4)² + (4 - 3)² = 1 + 1 = 2, which is about 1.41. When all vectors are scaled to length 1, as many embedding models do, the three measures rank results in the same order.

From text to tokens to embeddings

A language model does not read letters or whole words. Text is first cut into tokens, which are common chunks of characters. A short everyday word is usually one token, while a long or rare word is split into several pieces. Each token is replaced by an ID number from the model's fixed vocabulary. As a rough guide for English, one token is about four characters, so 100 tokens is around 75 words.

Tokens matter for two practical reasons. Models have a limit on how many tokens they accept at once, and paid services usually charge by the token.

An embedding model takes a run of tokens, whether a word, a sentence or a paragraph, and returns a single vector that represents its meaning. That vector is called an embedding. Real embeddings have hundreds or thousands of numbers, known as dimensions; 384, 768 and 1,536 are common sizes. Nobody assigns a meaning to each dimension by hand. The model learns them during training on enormous amounts of text, by being adjusted until passages that appear in similar contexts end up with similar vectors.

The result is that "How do I reset my password?" and "I forgot my login details" produce vectors that sit close together, even though they share almost no words. That is the property that makes search by meaning possible.

One model throughout. Vectors from different embedding models are not comparable. The documents and the question must be embedded by the same model, and changing model means embedding every document again.

Inside a vector database

You could keep embeddings in a plain list and compare the question against every one. That exact approach works for a few thousand items. It struggles at scale, and two quick sums show why.

Memory. Each number in a vector normally takes 4 bytes. One million vectors of 384 dimensions need 1,000,000 × 384 × 4 = 1,536,000,000 bytes, about 1.5 gigabytes. At 1,536 dimensions the same collection needs four times as much, about 6.1 gigabytes.

Work. Comparing one question against a million 384-dimension vectors takes 384 million multiplications, for every single query.

A vector database is built for this job. Besides storing each vector together with its original text and any labels, it builds an index: a structure that lets a search skip most of the collection. A widely used design arranges the vectors as a network in which each one is linked to its near neighbours. A search enters the network, hops from neighbour to neighbour towards the question's vector and stops when it can get no closer, having examined only a small fraction of the data.

The price is that the answer is approximate. This is called approximate nearest neighbour search, and it occasionally misses a true closest match in exchange for being dramatically faster. For most applications that is an excellent bargain, and the balance between speed and accuracy can be tuned.

Vector databases also store metadata beside each vector, such as a date, an author or a category, and can combine a filter with the similarity search: "the passages closest to this question, but only from documents published this year".

Semantic search step by step

Putting the pieces together gives semantic search, search by meaning. The process has two phases.

Indexing, done once in advance: split your documents into passages, embed every passage, and store each vector with its text. Splitting matters because one vector for a whole book would blur many topics into a meaningless average. Passages of a few sentences to a few paragraphs usually work well.

Querying, done for each question: embed the question with the same model, ask the database for the nearest vectors, and return the passages they belong to.

With ChromaDB, an open-source vector database that runs on your own computer, the whole cycle is a few lines of Python. By default it creates the embeddings for you.

import chromadb

client = chromadb.Client()
notes = client.create_collection(name="notes")

notes.add(
    ids=["1", "2", "3"],
    documents=[
        "Reset your password from the account settings page.",
        "Our office is closed on public holidays.",
        "Invoices are sent on the first day of each month.",
    ],
)

found = notes.query(query_texts=["I forgot my login details"], n_results=1)
print(found["documents"])

The question shares no important word with the first note, yet that is the one returned, because their embeddings are the closest. A collection here plays the role a table plays in an ordinary database, each entry needs a unique ID, and n_results sets how many matches to bring back.

Where vector search fits, and where it fails

The best-known use is retrieval-augmented generation. A language model knows only what it saw in training, so it cannot answer questions about your private documents. The remedy is to search those documents for the passages most relevant to the question and hand them to the model along with the question, so the answer is grounded in real text. Vector search is the retrieval step. The same technique drives recommendations ("items like this one"), duplicate detection and grouping documents by topic.

It has weaknesses worth knowing. Semantic search is poor at exact strings: a product code, an error number or an unusual surname carries little "meaning" for an embedding model, and a keyword search finds it more reliably. The closest match is also not always a good match. The database always returns the nearest vectors it has, even if none of them is relevant, so sensible systems check the similarity score and decline to answer when it is low. For these reasons many real systems combine keyword search with vector search, which is the subject of the next chapter.

Check your understanding

  • What is the cosine similarity of (1, 2) and (2, 1)? (Answer: the dot product is 2 + 2 = 4, each length is the square root of 5, and 4 ÷ 5 = 0.8.)
  • Two vectors have a cosine similarity of 0. What does that say about the angle between them, and about the texts they represent?
  • How much memory do 2,000,000 vectors of 768 dimensions need at 4 bytes per number? (Answer: 2,000,000 × 768 × 4 = 6,144,000,000 bytes, about 6.1 gigabytes.)
  • Why must the question be embedded with the same model as the documents?
  • What does an approximate nearest neighbour index give up, and what does it gain?
  • Give one kind of query for which keyword search beats semantic search.

Install Chrysolites Tech Academy

Android / Chrome / Edge: open the browser menu and choose Install app or Add to Home screen.

iPhone / iPad (Safari): tap the Share button, then Add to Home Screen.