Vector stores in practice: Chroma, Weaviate, and Milvus

The query shape is identical across all three. What differs is who operates the index and how many tenants share it.

Nov 20, 2026The avbbc desk
Vector stores in practice: Chroma, Weaviate, and Milvus
The query shape is identical across all three. What differs is who operates the index and how many tenants share it.

A vector search call looks the same everywhere: embed a query, return the nearest neighbours, attach metadata. That sameness is why comparisons in this category drift into feature lists. The differences that matter are operational: who runs the index, how you isolate tenants, and what happens when the index is wrong.

Three servers in the directory cover the range — Chroma, Weaviate, and Milvus.

The short answer

Chroma for a prototype on one machine. Weaviate when you need a self-hosted index with hybrid search. Milvus when the index is already large and someone operates it. The MCP server is the smallest decision in the stack; the corpus and the chunking are bigger.

What each one is really for

Store Shape Best moment
Chroma Embedded, local Prototype, one developer
Weaviate Self-hosted service Hybrid search, small team ops
Milvus Distributed service Scale with an owner

Read the third column rather than the second. A team without an operator should not pick the store that needs one, regardless of benchmark numbers.

Chroma: the prototype path

Chroma runs locally, stores collections on disk, and answers queries in milliseconds for a small corpus. The advantage is the disappearance of infrastructure: no server, no account, no deployment. The limitation arrives with success — multi-tenancy and concurrent writers are not what it is built for.

Weaviate: hybrid by default

Weaviate is a service you run, with hybrid search that combines keyword and vector results. That hybrid matters when identifiers and exact phrases are part of the queries, which is common in product documentation. It is a real server, so it needs backups and upgrades like one.

Milvus: the index somebody else owns

Milvus is built for large collections and horizontal scaling. An agent pointing at it is a client of infrastructure that already exists, which is the right relationship: retrieval is a feature, not a project. Configuring a Milvus cluster to serve one agent is the wrong order of operations.

The part that decides quality

None of these stores decides whether retrieval works. Three decisions do: how documents are split, whether metadata filters are applied, and whether the agent is allowed to answer from a single weak passage.

The chunking choice is the one teams regret. Chunks that are too small split an answer across two results; chunks that are too large dilute the passage with irrelevant text. A useful default is section-level chunks with a parent reference, so a hit can be expanded.

Tenancy and the tool boundary

A vector index usually holds more than one corpus. If the agent's tool can reach every collection, a metadata filter is the only thing between two customers' documents. Put the collection or namespace in the tool configuration rather than the prompt, and verify it with a query that should return nothing.

What to measure

Twenty questions with known answers beats any benchmark. For each, check whether the expected passage appeared in the top five, and whether the final answer cited it. Those are different failures and they point at different fixes. See RAG Eval.

Frequently asked questions

Do I need a vector database at all? For a small corpus, keyword search plus re-ranking is often enough. Vectors pay off for semantic questions at scale.

Which is easiest to start? Chroma, by a wide margin.

Can I move later? The data yes, the tuning no. Treat the first choice as a prototype decision and expect to re-index. See MCP servers.

Related reading

All posts