What Are Embeddings: How AI Understands Meaning, Not Just Words

Have you ever wondered why ChatGPT finds a connection between "car" and "automobile" — even though they are different words? Or why a RAG system finds the right document even if the query doesn't contain a single word from the text? Spoiler: one technology stands behind it — embedding. It's a way to convert any text into a set of numbers so that texts with similar meanings have similar numbers.

📚 Article Content

• 📌 Section 1. From tokens to numbers: why AI cannot work with words directly
• 📌 Section 2. Embedding = coordinates in the space of meaning
• 📌 Section 3. Where do these numbers come from? How the model learned to encode meaning
• 📌 Section 4. Dimensionality: why 768 ≠ 1536 and what it means in practice
• 📌 Section 5. Where embeddings are used: 5 real-world applications
• 📌 Section 6. Limitations of embeddings: when they won't help
• 💼 Section 7. Models of 2026: where to start
• ❓ Frequently Asked Questions (FAQ)
• ✅ Conclusions

🎯 From tokens to numbers: why AI cannot work with words directly

A token is a unit of text that the AI sees. An embedding is what that token *means*, written in numbers.

If you've read the article about tokens, you know: AI divides text into fragments (tokens) — these can be whole words, parts of words, or even individual characters. But a token is not yet meaning. It's just a unit of segmentation.

Embedding is the next step. Each token gets its unique numerical "fingerprint" — a vector. This vector encodes not the spelling of the word, but its meaning in the context of language. That's why "car" and "automobile" are different tokens, but their vectors will be very close.

Analogy: dictionary vs. city map

Imagine two ways to describe a word. The first is a dictionary: "cat — a domestic animal of the feline family." This is useful for humans, but useless for mathematics. The second is a map: in the city of concepts, "cat" is located next to "kitten," "paw," "purring" — and far from "rocket" or "accounting." An embedding is precisely a map where the position of each word is determined by its semantic neighborhood. OpenAI Embeddings documentation on how to convert text into a numerical vector in a high-dimensional space.

• ✔️ Token = unit of text (what is written)
• ✔️ Embedding = numerical vector (what it means)
• ✔️ Without converting text to numbers — no neural network can work with it

Conclusion: Embedding is the essential bridge between text that humans understand and numbers that AI works with.

Why different models give different results: comparison table

Why the same text yields different results in different models

There are three main reasons for discrepancies in results between models:

1. Different training signal. OpenAI trained the model on synthetic pairs from GPT-4 — this provides good general quality, but the dataset details are closed. BGE-M3 was trained on an open 1.2M pair dataset with an explicit multilingual signal — hence better on Cyrillic. E5 uses instruction-tuning: the model expects the prefix "query: " before the query and "passage: " before the document. If the prefix is not added — quality drops by 5–15% even on English content.

2. Different context window. E5-large processes only 512 tokens — a long document will be truncated, and the tail part will simply disappear from the vector. OpenAI and BGE-M3 process up to 8K tokens, allowing entire articles to be embedded without information loss. If your documents are longer than 400 words and you use E5 — chunking is definitely required.

3. Different space geometry. Each model builds its own vector space. The vector for the word "contract" from OpenAI and the vector for the same word from BGE-M3 are incompatible numbers in incompatible spaces. That's why you cannot mix vectors from different models in the same Vector DB and cannot compare them. More details on this are in the MTEB Leaderboard, where models are compared on standardized benchmarks.

📌 Embedding = coordinates in the space of meaning

If regular GPS coordinates are two numbers (latitude and longitude), then an embedding is a GPS in a space with thousands of dimensions, where each "coordinate" encodes some aspect of meaning.

Let's imagine a very simplified space — only two dimensions. The X-axis represents how much the word is related to the living world. The Y-axis represents how much it is related to movement. Then:

• "lawyer" → high X (human), low Y (not about movement) → coordinate [0.9, 0.1]
• "courier" → high X, high Y → [0.9, 0.8]
• "car" → low X, high Y → [0.1, 0.9]
• "rocket" → low X, very high Y → [0.05, 0.95]

In reality, there are not two, but hundreds or thousands of dimensions — each encodes some subtler aspect: emotional tone, syntactic role, thematic relevance, etc. We cannot interpret them individually, but the mathematics of distances between points in this space works flawlessly.

How similarity between vectors is measured: Cosine Similarity

The distance between two vectors is measured not by a straight line (Euclidean distance), but by the angle between them — the so-called cosine similarity. The smaller the angle between two vectors, the more semantically similar the words or sentences are. "Contract" and "agreement" point in almost the same direction in the space of meaning — the angle between them is small, similarity is high. "Contract" and "rocket" are almost perpendicular, similarity is close to zero.

Cosine similarity is the de facto standard for most models and vector DBs. But there is an important nuance that I see in production errors: some models are optimized for a different metric — Dot Product. For example, Google's Gemini Embedding 2 and some Cohere models in certain modes expect Dot Product, not Cosine. If you set the wrong metric in your Vector DB — search results will be worse even with the correct model. Always check the model's documentation before setting the distance in ChromaDB, Qdrant, or pgvector.

Lifehack: normalizing vectors speeds up search

There is a practical trick worth knowing. If you normalize vectors to unit length — that is, bring each vector to a norm of 1 — then Cosine Similarity becomes mathematically equivalent to Dot Product. And Dot Product is calculated faster: it's just the sum of element-wise products, without additional division by norms. Many vector DBs (Qdrant, pgvector with vector_cosine_ops) do this automatically — but if you use your own search or FAISS, normalize vectors before saving and you'll get a speed boost without any loss of quality.

More details on how cosine similarity and Dot Product are used in search in practice are in the article Vector Search for Beginners.

• ✔️ Embedding — a list of numbers from 384 to 3072 values
• ✔️ Similarity is measured by the angle between vectors (Cosine Similarity) — but check the model's documentation: some are optimized for Dot Product
• ✔️ Normalized vectors: Cosine = Dot Product → faster search without quality loss

Conclusion: Embedding transforms abstract "meaning" into concrete geometry — and this is precisely what allows AI to compare text meanings mathematically. But choosing the right similarity metric is no less important than the model itself.

📌 Section 3. Where do these numbers come from? How the model learned to encode meaning

The model doesn't read a dictionary and doesn't know grammar. It reads billions of sentences — and learns which words and phrases co-occur, and which never appear together.

Behind this lies the idea of distributional semantics, formulated by linguist John Firth back in 1957: "A word is characterized by the company it keeps." If two words consistently appear in similar contexts — they most likely have similar meanings. This very principle became the basis for Word2Vec (Google, 2013) — the first large-scale model that converted words into vectors through context statistics, rather than manual labeling. Original paper by Mikolov et al., 2013 — arXiv:1301.3781 .

From Word2Vec to modern contrastive learning

Word2Vec training was simple: for each word — predict neighboring words within a window of 5–10 tokens. If "lawyer" and "attorney" frequently appear in similar sentences — their vectors get closer. This already worked. But in Word2Vec, each word had *one* vector regardless of context: the word "key" received the same vector in the sentence "key to the lock" and "key to success" — even though the meanings are different.

Modern models (BERT by Google, 2018) solved this problem: the vector now depends on the entire sentence, not on a static dictionary. The word "key" gets a different vector depending on what is next to it. This became possible thanks to the transformer architecture and the self-attention mechanism. BERT: Pre-training of Deep Bidirectional Transformers — arXiv:1810.04805 .

What contrastive learning looks like in practice

The current standard for training embedding models is contrastive learning, specifically the SimCSE approach and its derivatives. The model receives three sentences simultaneously:

• Anchor: "How to cancel a service subscription?"
• Positive example: "Instructions for unsubscribing from a plan" — should be close to the anchor
• Negative example: "Recipe for traditional borscht" — should be far from the anchor

The loss function (contrastive loss or InfoNCE loss) penalizes the model if the positive vector is far from the anchor — or if the negative vector is too close. Through billions of such triplets, the model learns to build a space where semantic proximity is reflected by geometric proximity. SimCSE: Simple Contrastive Learning of Sentence Embeddings — arXiv:2104.08821 .

Where do they get pairs for training? Mostly from three sources: natural question-answer pairs (NLI datasets, Stack Overflow, Reddit), parallel translations (for multilingual models), and synthetic pairs generated by LLMs. For example, to train text-embedding-3-small, OpenAI used synthetic data generated by GPT-4, as described in OpenAI's blog on new embedding models .

What the model actually encodes in the vector

Researchers have found that different dimensions of the vector encode different aspects of meaning — but not as straightforwardly as one might wish. One dimension might be responsible for "legal context," another for "emotional tone," a third for "syntactic role in the sentence." But the dimensions are not isolated: meaning is encoded distributively across all numbers simultaneously. This means that no single number in the vector has a human-interpretable meaning — only the entire combination together.

A well-known example from Word2Vec, which has carried over to modern models: vector("king") − vector("man") + vector("woman") ≈ vector("queen"). The arithmetic of meanings works in the vector space — this is one of the most striking confirmations that the model has truly captured the structure of language, not just memorized words.

Why multilingual models understand "договір" and "contract" as the same

Multilingual embedding models (Cohere embed-v4, BGE-M3, multilingual-e5) were trained simultaneously on texts in dozens of languages and on parallel translation corpora. Because of this, "договір" (Ukrainian) and "contract" (English) end up in adjacent points in the same space — because they appeared in similar contexts and were linked through translations.

This enables cross-lingual search: a query in Ukrainian finds documents in English without any translation. In practice — if your knowledge base is filled mostly with English documents, and users query in Ukrainian, a high-quality multilingual model will bridge this gap. In our WebsCraft case, this was a real problem: some content exists only in one language, and queries come in another. BGE-M3 handles this significantly better than OpenAI small, where quality on Cyrillic is noticeably lower than on Latin — which is also confirmed by the MTEB Multilingual Leaderboard results .

• ✔️ Distributional semantics: meaning = co-occurrence context
• ✔️ Contrastive learning: positive pairs move closer, negative ones move apart
• ✔️ Modern models (BERT and beyond): vector depends on context, not static
• ✔️ Multilingual models: translations in the same space → cross-lingual search without a translator

Conclusion: The numbers in a vector are not arbitrary and not hardcoded. They are the result of training on billions of examples and reflect the real statistical structure of language — that's why they capture meaning so well.

📌 Dimensionality: why 768 ≠ 1536 and what it means in practice

The dimensionality of a vector is like the resolution of a photograph: 100×100 pixels are enough for an icon, 4000×3000 for a poster. What you need depends on the task.

More dimensions = more "channels" for encoding meaning. A small vector (384 dimensions, like in all-MiniLM) can distinguish "cat" and "rocket," but might confuse subtle nuances: for example, it might not differentiate "early termination of contract" from "expiration of contract term." A large vector (3072 dimensions, like in text-embedding-3-large) will capture this difference too, but will cost more and take up more space in the database.

Trade-off in practice: speed, quality, cost

Matryoshka embeddings: high dimensionality without overpaying

In 2024–2025, the Matryoshka Representation Learning (MRL) technique gained popularity. The idea is simple: the model is trained such that the first N dimensions of the vector already carry most of the useful information. This means you can generate a 3072-dimensional vector but store and search only the first 256 — with minimal loss of quality. OpenAI text-embedding-3-large and Gemini Embedding 2 support this approach: generate once, and choose the dimensionality for the task.

• ✔️ Higher dimensionality = more nuances, but more expensive and slower
• ✔️ For most projects, 768–1536 dimensions is optimal
• ✔️ Matryoshka (MRL): you can truncate vectors without re-generation

Section conclusion: Dimensionality is a trade-off between quality and resources; start with 768–1536 and increase only if there's a specific search quality problem.

📌 Where embeddings are used: 5 real-world applications

If keyword search is like searching by address, then embedding search is like searching for what's inside the house.

1. Semantic search

Traditional search looks for exact word matches. Semantic search finds documents by meaning: a query "how to cancel subscription" finds an article "instructions for tariff cancellation" — even if there are no common words. This is how internal searches in Notion, Confluence, and corporate knowledge bases work.

2. RAG (Retrieval-Augmented Generation)

In RAG systems, embeddings are used twice: during indexing (document → vector → store in vector DB) and during search (query → vector → find similar documents → LLM response). Without quality embeddings, RAG returns irrelevant fragments, and even the best LLM will hallucinate. More details in the article LLM vs RAG in 2026 and the complete RAG guide.

3. Recommendation systems

"You might like..." is often embeddings in action. The article you're reading is converted into a vector. The system finds other articles with similar vectors and suggests them. The same applies to streaming services: a movie you watched → vector → search for similar ones by mood and theme, not just genre.

4. Zero-shot classification

Traditional classification requires thousands of labeled examples for each class. With embeddings, you can classify texts without any examples: calculate the vector of the text and compare it with the vectors of class names. "Is this a complaint or praise?" — calculate the cosine similarity with the vectors of the words "complaint" and "praise" and see which one is closer.

5. Duplicate detection and cross-lingual search

Embeddings find nearly identical texts even if they are paraphrased or written in different languages. This is useful for knowledge base deduplication, plagiarism detection, or — especially important for Ukrainian content — searching when the query is in one language, but the documents exist in another. A multilingual embedding model places "rental agreement" (Ukrainian) and "rental agreement" (English) in adjacent points in space.

• ✔️ Semantic search: by meaning, not by words
• ✔️ RAG: the foundation of retrieval quality
• ✔️ Recommendations, classification, deduplication, cross-lingual search

Conclusion: Embeddings are not a standalone technology, but a horizontal tool: they appear everywhere AI needs to understand or compare text meaning.

📌 Limitations of embeddings: when they won't help

An embedding answers "what is this text about?", but not "what is the exact value in row 7?"

Problem 1: exact numbers, dates, IDs

The query "contract №4521" and "contract №4522" are almost the same to an embedding model: both are "about a contract with a number." The difference of one digit is not reflected in the vector as it is important to a human. The same applies to dates: "what happened on February 15, 2023?" — the embedding doesn't "know" it's a specific date, not just "February."

Problem 2: proper nouns and terms

Rare proper nouns, abbreviations, product names, or technical terms are often poorly represented in the embedding space. If a name appeared rarely in the training data, its vector will be "blurry" and unreliable.

Problem 3: short queries

A query like "RAG" or "Spring" provides too little context for a quality vector. The model doesn't know what you're interested in: RAG as a technology, RAG as a movie title, or Spring as a season. The vector will be averaged and fuzzy. The solution is to fall back to keyword search for short queries, as described in the article Vector Search for Beginners.

What to do: hybrid search

For production systems, the right answer is to combine semantic search with traditional keyword search (BM25). Hybrid search provides a +15–40% quality improvement compared to pure vector search and solves all three problems above. More details in the complete RAG guide.

• ✔️ Exact numbers, IDs, dates → keyword search or metadata filters
• ✔️ Short queries → fallback to BM25
• ✔️ Production → hybrid search (BM25 + vector)

Conclusion: Embeddings are a powerful tool for semantic tasks, but they don't replace exact search; know their limits and combine approaches.

Common mistakes with embeddings and how to avoid them

💼 2026 Models: Where to Start

Don't spend weeks choosing the "perfect" model. Start, log real results, and only replace if you see specific problems.

This section is a navigation guide. A full comparison of 10+ models with prices, benchmarks, and a selection matrix can be found in the anchor article Embedding Models for RAG in 2026. Here are three scenarios for a quick start.

Scenario 1: Local Development Without Costs

nomic-embed-text via Ollama — free, 768 dimensions, 8K context window. Runs with a single command: ollama pull nomic-embed-text. Suitable for prototypes and learning, your data never leaves your computer.

Scenario 2: API Start with Minimal Costs

text-embedding-3-small from OpenAI — $0.02 per million tokens, 1536 dimensions. For a blog with 500 articles, this is less than $1 per month. The broadest integration ecosystem, the easiest start via API. Downside: quality on Ukrainian is lower than on English.

Scenario 3: Multilingual Content with Cyrillic

BGE-M3 (open-source, self-hosted) or Cohere embed-v4 ($0.10/1M tokens) — both support 100+ languages with equal quality for Latin and Cyrillic. BGE-M3 also supports hybrid search "out of the box": dense + sparse vectors in one model.

• ✔️ Start simple — nomic or OpenAI small
• ✔️ Multilingual content → BGE-M3 or Cohere
• ✔️ Details and benchmarks → anchor article

My recommendation: model choice is a less critical decision than the quality of your data and your chunking strategy; start fast and optimize on real queries.

❓ Frequently Asked Questions (FAQ)

How does an embedding differ from a token?

A token is a unit of text (a word or part of it) that AI identifies when breaking down text. An embedding is a numerical vector that encodes the meaning of that token or an entire sentence. Tokens are at the syntax level, embeddings are at the semantics level. More details on tokens can be found in the article about tokens.

Do I need to re-index the database when changing the embedding model?

Yes, absolutely. Different models generate vectors in incompatible spaces: a vector from OpenAI and a vector from BGE-M3 cannot be compared to each other. If you change the model, you need to regenerate all vectors from scratch. This is precisely why the initial model choice is an important architectural decision.

Can I use one embedding for search and classification?

Technically, yes, but the quality will be lower. Some models (e.g., Cohere embed-v4 or Jina v5) support "task-specific" modes: you specify the task type (retrieval, classification, matching), and the model optimizes the vector for it. If you have multiple tasks, consider such models.

How can I check the quality of embeddings for my content?

The simplest way: take 10–20 real queries, perform a search, and look at the top 3 results with their cosine similarity scores. If relevant results have scores below 0.6 or irrelevant ones above 0.7, the model is not suitable for your content. Benchmarks (MTEB) are a useful guide, but testing on real data is always more important. For a systematic evaluation approach, see the guide to metrics and evaluation pipelines for RAG.

Can I create embeddings for images?

Yes. Multimodal models (e.g., Google Gemini Embedding 2) generate vectors for both text and images in the same space. This allows you to search for images by text description or vice versa, and find images that are similar in meaning, not just pixels.

✅ Conclusions

• 🔹 Embedding is the conversion of text into a numerical vector, where the proximity of numbers reflects the proximity of meaning.
• 🔹 The model learns through contrastive learning on billions of text pairs — without manual labeling.
• 🔹 Dimensionality (384–3072) is a trade-off between quality and resources; 768–1536 is sufficient for most tasks.
• 🔹 Embeddings are the foundation of RAG, semantic search, recommendations, and classification.
• 🔹 They are not precise with numbers, IDs, and short queries — hybrid search is needed for this.
• 🔹 For starting: nomic-embed-text locally or text-embedding-3-small via API; for Cyrillic — BGE-M3 or Cohere.

Main takeaway: Embedding is the language AI uses to describe meaning. By understanding how it works, you can build systems that find the right information — not just the right words.

If you want to understand how embeddings are used in search at a mathematical level, read Vector Search for Beginners. If you're ready to choose a specific model, read Embedding Models for RAG in 2026.