Vector Stores: Which One, When, and Why It Matters
Vector Database Selection and Implementation Guide
---
Your AI assistant just briefed a client using a project proposal that was revised six months ago. The vector store indexed the wrong version and nothing flagged it as outdated. The AI was confident. The information was wrong. The client noticed.
That's a vector store problem, not a model problem.
A vector store is your AI's indexed library — every document, report, and record your organization has taught it, converted into mathematical representations of meaning so the AI can search by concept rather than keyword. When you ask the AI "what are our standard contract payment terms," it's not searching for the words "payment terms." It's searching for the meaning of the question and finding documents that match conceptually. That search takes under 100 milliseconds on a well-configured vector store — compared to 45 seconds or more for traditional keyword search across the same document archive.
That process — retrieval augmented generation, or RAG — is how enterprise AI works in practice. The model provides the intelligence. The vector store provides the memory. The quality of your AI's answers is directly tied to how well its memory retrieves the relevant information.
Every regulated organization needs one clear answer: where is your AI's memory stored, and who controls access to it?
---
The Hidden Infrastructure Decision
Most vector store choices happen during a developer's prototype phase. Setup takes minutes with Pinecone or Chroma. By the time the prototype becomes production, a database containing your organization's indexed knowledge base is already live — and nobody in IT governance signed off on it.
A 2024 LayerX Security survey found that 43 percent of organizations that deployed AI couldn't identify the specific vector store holding their indexed data. It was chosen by a developer on a deadline. It never went through the same review as any other system storing sensitive organizational data.
This matters because of what gets indexed. Not just publicly available documents — your internal contracts, client correspondence, acquisition research, proprietary processes. Once indexed, that content lives in the vector store. Moving it out means re-embedding every single document — breaking each one into small sections and converting them all again — a compute-intensive process that took one Paris-based law firm six weeks and significant engineering cost to complete after their auditor flagged the CLOUD Act exposure.
---
The Metric That Doesn't Matter
Most vector store comparisons benchmark query speed — how many milliseconds to retrieve results.
Speed is the wrong question. Recall is what matters.
Recall@10 measures whether the document your AI actually needs appears in the top ten results it retrieves. A store answering in 40 milliseconds but missing the relevant document in one out of five searches is worse than one answering in 150 milliseconds with 95 percent recall. Your users don't feel the 110-millisecond difference. They do feel the answers built on incomplete information.
The evaluation that produces better AI answers is not a speed benchmark. It's checking whether your AI's memory actually finds the right document when it needs to.
---
What Makes the Cloud Option Complicated
Pinecone is the most widely deployed managed vector database. It's also architecturally incompatible with sovereign AI requirements for regulated European organizations.
With no on-premises deployment option and infrastructure running exclusively on US-hosted servers, Pinecone means your organization's indexed knowledge base lives under US legal jurisdiction. That means every document the AI has learned from sits on servers in the United States, where US law governs who can access it.
The CLOUD Act — a US law that allows authorities to compel US companies to hand over data from any server they control, anywhere in the world — applies to Pinecone. A European organization that selects a non-US server region for Pinecone still stores data within a US company's infrastructure. That exposure applies to every indexed document, regardless of the server region selected.
EU AI Act Article 13 adds a second dimension. Article 13 requires technical documentation sufficient to explain AI system outputs — which means your vector store must support complete retrieval logging, a record of exactly which documents the AI accessed to generate each answer. Most managed vector services require custom engineering work to add this capability. Without it, an EU regulatory audit leaves you with an explainability gap you cannot fill retroactively.
---
The Self-Hosted Options, Explained
Three open-source vector databases cover the majority of enterprise deployments at different scales.
Qdrant runs on a single commodity server and delivers sub-10 millisecond query response times on collections up to 10 million documents. Installation takes hours. The interface is clean. For organizations under 10 million indexed documents, Qdrant handles the workload without requiring specialist database administration. A Luxembourg financial services company selected Qdrant for their first sovereign AI deployment and had it operational in a week, integrated with their existing infrastructure team's skill set.
Milvus is built for scale. At collections above 100 million documents, Milvus distributes its work across multiple servers and maintains consistent performance where Qdrant's single-server design reaches its limits. Organizations with large document repositories — ten years of client records, regulatory filings, research archives — should evaluate Milvus from the start rather than migrating later. Migration from a wrong choice at scale is expensive.
pgvector adds vector search directly to PostgreSQL, the database that runs a significant portion of enterprise applications already. No new infrastructure. No additional operational complexity for the IT team. For organizations that already run PostgreSQL in production, pgvector deserves evaluation first — the total cost of adding a capability to existing infrastructure is almost always lower than deploying a dedicated system.
All three are open-source, self-hostable, and EU-jurisdiction-capable. None charges per document stored. The operational cost is infrastructure cost only.
---
When Shadow Indexing Becomes a Compliance Problem
A developer at a European bank chose Pinecone for a knowledge base prototype in early 2023. Two years later, the bank's complete underwriting knowledge base — client data, property valuations, risk models — is indexed there. No senior sign-off. No data residency review. No documented exit strategy.
This is shadow indexing: a prototype decision that matured into production infrastructure without governance review. The pattern repeats across industries because the friction of setting up a cloud managed service is genuinely lower than setting up a self-hosted alternative.
At the Paris law firm, migration away from Pinecone happened after their external auditor flagged the issue — 2.3 million indexed document fragments, six weeks of engineering time, significant compute cost to re-process everything. Their CTO later noted that the right governance checkpoint would have been at prototype approval, before any client correspondence was indexed.
Vector store governance needs to happen before indexing begins. Re-indexing after the fact compounds in difficulty as data volume grows. At 100,000 documents, migration is manageable. At 10 million, it's a significant project. At 100 million, it can take months.
---
Three Questions That Resolve the Choice
For most enterprise deployments, the vector store selection comes down to three questions in order.
Is this data regulated or sensitive — legal, financial, healthcare, defense, or under GDPR? If yes: self-hosted, EU jurisdiction. Cloud managed options are off the table for this data type regardless of their technical performance.
Does your organization already run PostgreSQL in production? If yes: evaluate pgvector first. Adding a capability to existing infrastructure is almost always cheaper operationally than deploying a new system. Your IT team already knows the operational model.
Do you need to index more than 100 million documents? If yes: deploy Milvus. Its distributed architecture handles that scale without the performance degradation that single-server systems experience above this threshold. If no: start with Qdrant. It handles up to 10 million documents on a single server with sub-10 millisecond response times and straightforward administration.
For organizations that don't want the vector store to be a standing decision — the Leeloo Framework includes the Vault component, which pre-selects and pre-configures the appropriate vector store based on your sovereignty tier and data volume, integrates retrieval audit logging from day one, and manages incremental indexing so your AI's knowledge stays current. The infrastructure decision is resolved as part of deployment, not before it.
---
What a Review Cycle Looks Like
Vector store technology moves faster than most enterprise infrastructure categories. Qdrant barely existed at production scale in 2021. By 2024 it was outperforming established players in independent benchmarks. Organizations locked into managed cloud stores from 2022 are paying migration costs they couldn't have predicted, because the self-hosted open-source alternatives weren't viable when they made the original choice.
A 12 to 18 month review cycle on vector store selection is appropriate — not to migrate constantly, but to confirm that the current choice still fits the organization's scale, compliance requirements, and performance needs. The sovereignty requirements only increase as AI adoption grows and regulatory enforcement arrives.
The question that used to be "which vector store is fastest" is now "which one keeps our data where it belongs while giving our AI the best chance of finding the right answer." Those two requirements have a clear answer today: self-hosted, EU-jurisdiction, retrieval-logged, with Recall@10 as the benchmark that actually measures what matters.
---
The Leeloo Framework's Vault component handles vector store selection, configuration, and retrieval audit logging as a pre-built production component. Learn more at [leeloo.ai](https://leeloo.ai).