
A vector search call looks the same everywhere: embed a query, return the nearest neighbours, attach metadata. That sameness is why comparisons in this category drift into feature lists. The differences that matter are operational: who runs the index, how you isolate tenants, and what happens when the index is wrong.
Three servers in the directory cover the range — Chroma, Weaviate, and Milvus.
The short answer
Chroma for a prototype on one machine. Weaviate when you need a self-hosted index with hybrid search. Milvus when the index is already large and someone operates it. The MCP server is the smallest decision in the stack; the corpus and the chunking are bigger.
What each one is really for
| Store | Shape | Best moment |
|---|---|---|
| Chroma | Embedded, local | Prototype, one developer |
| Weaviate | Self-hosted service | Hybrid search, small team ops |
| Milvus | Distributed service | Scale with an owner |
Read the third column rather than the second. A team without an operator should not pick the store that needs one, regardless of benchmark numbers.
Chroma: the prototype path
Chroma runs locally, stores collections on disk, and answers queries in milliseconds for a small corpus. The advantage is the disappearance of infrastructure: no server, no account, no deployment. The limitation arrives with success — multi-tenancy and concurrent writers are not what it is built for.
Weaviate: hybrid by default
Weaviate is a service you run, with hybrid search that combines keyword and vector results. That hybrid matters when identifiers and exact phrases are part of the queries, which is common in product documentation. It is a real server, so it needs backups and upgrades like one.
Milvus: the index somebody else owns
Milvus is built for large collections and horizontal scaling. An agent pointing at it is a client of infrastructure that already exists, which is the right relationship: retrieval is a feature, not a project. Configuring a Milvus cluster to serve one agent is the wrong order of operations.
The part that decides quality
None of these stores decides whether retrieval works. Three decisions do: how documents are split, whether metadata filters are applied, and whether the agent is allowed to answer from a single weak passage.
The chunking choice is the one teams regret. Chunks that are too small split an answer across two results; chunks that are too large dilute the passage with irrelevant text. A useful default is section-level chunks with a parent reference, so a hit can be expanded.
Tenancy and the tool boundary
A vector index usually holds more than one corpus. If the agent's tool can reach every collection, a metadata filter is the only thing between two customers' documents. Put the collection or namespace in the tool configuration rather than the prompt, and verify it with a query that should return nothing.
What to measure
Twenty questions with known answers beats any benchmark. For each, check whether the expected passage appeared in the top five, and whether the final answer cited it. Those are different failures and they point at different fixes. See RAG Eval.
Frequently asked questions
Do I need a vector database at all? For a small corpus, keyword search plus re-ranking is often enough. Vectors pay off for semantic questions at scale.
Which is easiest to start? Chroma, by a wide margin.
Can I move later? The data yes, the tuning no. Treat the first choice as a prototype decision and expect to re-index. See MCP servers.


