Embedding Models Explained: Dimensions, Storage Cost and the Re-Embedding Bill (2026)

Every "chat with your documents" setup has two models in it. Everyone obsesses over the chat model. The other one, the embedding model, decides which paragraphs the chat model ever gets to see. If it fetches the wrong chunk, the smartest LLM in the world answers the wrong question very confidently.

I did not benchmark or install anything for this piece. On 3 October 2026 I read the vendors' docs and price pages, the model cards and the MTEB papers, and did the arithmetic. If you only want a pick, our best local embedding model guide has one. This piece is about the numbers behind the pick.

What is an embedding, in plain terms?

An embedding is a list of numbers that stands for the meaning of a piece of text. Texts that mean similar things get lists that sit close together. Your RAG app embeds every chunk of your documents once, then embeds each question and fetches the chunks whose numbers sit nearest to it.

OpenAI's guide puts it plainly: "An embedding is a vector (list) of floating point numbers. The distance between two vectors measures their relatedness." Small distance, related. Large distance, unrelated. That is the whole trick behind semantic search, which means search by meaning rather than by matching words.

Think of a map where every paragraph you own is a pin. "Refund policy" and "how do I get my money back" land on the same street, even though they share no words. The embedding model is the cartographer. A bad one puts the refund policy in the wrong town.

Two practical rules follow. First, the question and the documents must be embedded by the same model, or the distances mean nothing. Second, many models want a label on each input. Nomic's model card says documents must start with search_document: and questions with search_query:. Mixedbread's mxbai-embed-large wants the prompt "Represent this sentence for searching relevant passages:" in front of queries. Skip those and retrieval quietly gets worse.

What does "dimensions" mean, and why does it cost money?

Dimensions is the length of that list of numbers. More dimensions can hold finer shades of meaning, but every one is stored for every chunk. At the usual 32-bit float, each dimension costs 4 bytes, so storage is dimensions x 4 bytes x number of vectors.

Here is what the common lengths mean for a million chunks, which is a big company wiki or a few years of support tickets:

These are floor numbers. The pgvector README says each vector row takes 4 * dimensions + 8 bytes, so add 8 bytes a row. The search index, usually an HNSW graph (a structure that finds near neighbors fast), sits on top, and its size depends on its settings. I am not printing a number for it.

Two cheaper storage types exist. Per pgvector's docs, halfvec stores 2 bytes per dimension, half the float32 size. A bit vector stores one bit per dimension, so 1,024 dimensions shrink to 128 bytes. Both trade some precision for space.

One limit matters for anyone on Postgres. pgvector's HNSW and IVFFlat indexes accept vector columns of up to 2,000 dimensions, halfvec up to 4,000 and bit up to 64,000. A 3,072-dimension OpenAI or Gemini vector cannot be indexed as a plain vector. You either store it as halfvec or ask the API for fewer dimensions. Our vector database guide covers where those vectors live.

How do I choose an embedding model without being fooled by MTEB?

Use the MTEB leaderboard to make a shortlist, then test the shortlist on your own documents. MTEB, the Massive Text Embedding Benchmark, averages many task types. RAG cares mostly about retrieval. Compare models on the same benchmark version, and check the zero-shot column before trusting a high score.

Four caveats, all from the benchmark's own papers and docs.

• Versions are not comparable. The MMTEB paper says the English benchmark went from 56 tasks in v1 to 40 tasks in v2. OpenAI's guide lists text-embedding-3-large at 64.6% "on MTEB", Nomic lists 62.28 for its model, and Google's EmbeddingGemma card lists 69.67 on "MTEB (English, v2)". Those three numbers come from different tests, so ranking them against each other tells you nothing.
• Training on the test. The paper says v2 is meant to be zero-shot, "excluding tasks like MS MARCO… and Natural Questions… which are frequently used in fine-tuning." Some small English models scored worse on v2, "likely because they were trained on MS MARCO and Natural questions." A model that trained on the test's cousins looks smarter than it is.
• Who ran the numbers. MTEB's docs explain that results reach the leaderboard by a pull request to its results repository, including results "that you ran yourself." Model makers also declare their own training datasets. That is open and auditable, but it is not an independent lab run.
• The average hides your task. The results tables carry a separate Retrieval score next to classification, clustering and the rest. For RAG, sort by retrieval, and by your language.

Then run your own check. Take 30 to 50 real questions your users ask, note which chunk should answer each, and see how often each model puts that chunk in its top five. That small test beats any leaderboard rank, because it is your data. Our chat with your documents guide shows the full pipeline the test plugs into.

What are the local options in Ollama and llama.cpp?

Ollama lists a dedicated embedding category, and you call any of them through its /api/embed endpoint. llama.cpp's server runs embedding models with the --embedding flag and an OpenAI-compatible /v1/embeddings route. Local embedding costs you no per-token fee, only your own hardware and time.

Two traps hide in that table. First, the bare tag qwen3-embedding pulls the 8B model, a 4.7 GB download with 4,096-dimension vectors. If you wanted the small one, ask for qwen3-embedding:0.6b. Second, Ollama's /api/embed has a truncate option that "truncates the end of each input to fit within context length" and defaults to true. Feed a 1,000-token chunk to a model with a 512-token window and the end of that chunk is silently dropped. Size your chunks to the window, or set truncate to false so you get an error instead.

The Ollama API also takes a dimensions parameter. Its docs do not say whether it re-normalizes after cutting, so if you shorten vectors, normalize them yourself. In llama.cpp, --embd-normalize defaults to 2, which is the usual Euclidean normalization, and --pooling picks how token vectors are combined. Most embedding models ship with the right default.

# Ollama: embed two chunks with the small Qwen3 model
curl http://localhost:11434/api/embed -d '{
  "model": "qwen3-embedding:0.6b",
  "input": ["first chunk", "second chunk"]
}'

# llama.cpp: serve an embedding model on an OpenAI-style route
llama-server -m embedding-model.gguf --embedding --port 8080

These models are small enough that many run fine without a GPU. For running the server side of things, see local AI as an API server.

What do the API embedding models cost?

Embedding APIs bill per input token and are cheap. On 3 October 2026, OpenAI listed text-embedding-3-small at $0.02 per million tokens and 3-large at $0.13. Google listed gemini-embedding-2 at $0.20 for text. Voyage AI listed voyage-4-lite at $0.02, voyage-4 at $0.06 and voyage-4-large at $0.12.

Some fine print from the same pages. OpenAI's Batch API supports /v1/embeddings at a 50% discount, with up to 50,000 embedding inputs per batch and a 24-hour turnaround. Google's free tier for gemini-embedding-2 costs nothing, but its pricing table says free-tier data is used to improve Google's products, while paid-tier data is not. Voyage gives each account 200 million free tokens on the 4 series, and its docs say those free credits do not apply to batch jobs.

What is Matryoshka truncation?

Matryoshka truncation means cutting a vector short and still getting useful results. Models trained with Matryoshka Representation Learning pack the most important meaning into the first numbers, like nesting dolls. You keep the first 256 of 768 numbers and lose a little accuracy for a lot of storage.

The idea comes from a paper by Aditya Kusupati and colleagues. They describe a single embedding that can "adapt to the computational constraints of downstream tasks." Google says both of its Gemini embedding models are trained this way, and recommends 768, 1,536 or 3,072 output dimensions.

The model cards print what you give up. Google's EmbeddingGemma card scores each cut length on MTEB English v2:

Going from 768 to 256 dimensions cuts storage to a third and the score by 1.30 points. Nomic's card shows a similar curve for nomic-embed-text v1.5: 62.28 at 768, 61.04 at 256 and 56.10 at 64. Google's own table for gemini-embedding-001 is flatter still: 68.17 at 1,536 and 67.99 at 768.

Three rules for doing it safely:

OpenAI's guide gives the most striking example. It says a text-embedding-3-large embedding shortened to 256 dimensions still outperforms the older ada-002 at its full 1,536 on MTEB. That is a 6x smaller vector from a newer model.

What does it cost to switch embedding models?

The token bill is small. The work is not. Vectors from two different models live in different spaces, so you cannot search old vectors with a new model's query. Switching means re-embedding every chunk, rebuilding the index and swapping over, ideally while the old index keeps serving.

Google's docs are blunt about its own upgrade: the spaces of gemini-embedding-001 and gemini-embedding-2 "are incompatible," and if you upgrade, "you must re-embed all of your existing data." That applies to any switch between models. One documented exception: Voyage says "all embeddings created with the 4 series are compatible with each other," so moving between voyage-4-lite, voyage-4 and voyage-4-large does not force a rebuild, per Voyage.

Here is the bill for a typical case. Assume 1,000,000 chunks of about 500 tokens each, which is 500 million tokens. The cost is 500 x the price per million tokens.

So a full re-embed runs from $10 on text-embedding-3-small to $100 on gemini-embedding-2, or $5 to $50 through the batch APIs. A local model has no token bill, but it is not free. It costs machine hours, and I have no measured throughput to give you, so I won't guess one.

The hidden costs are the ones nobody prices. You store two indexes during the cutover. Every cached query vector is stale. Your chunking may need to change if the new model's context window is different. And your retrieval test has to run again. That is why the cheapest decision is choosing carefully the first time, and writing down the model name and version next to every vector you store.

So which one should you pick?

Start with your constraint, not the leaderboard. If data must stay on your machine, pick a local model sized to your chunks. If you want no hardware, a $0.02 API model costs about $10 per million chunks. Then test retrieval on your own questions and pick a length you will keep.

My rough order of operations: decide local or API first. Pick the shortest vector that passes your own test, because storage and memory scale with it. Prefer a Matryoshka model so you can shrink later without re-embedding. And record the model name, version, dimensions and input prefixes in your schema, because six months from now someone will ask why search got worse after "a small upgrade."