Vector Database Design and Implementation
The storage and retrieval layer under your search and RAG systems, designed around the decisions that actually determine quality: how content is chunked, how it is filtered, and what happens when the embedding model changes.
Standing up a vector database takes an afternoon. Getting good answers out of it takes the rest of the engagement, and almost none of that time goes on the database. It goes on chunking, on metadata, and on measuring whether the right passage was actually retrieved.
A vector database stores high-dimensional embeddings and retrieves the nearest ones to a query vector using approximate nearest-neighbour indexes. It is the retrieval substrate under semantic search, RAG and recommendation systems, and its practical quality is determined far more by how content is prepared and filtered than by which database is chosen.
Chunking is the decision nobody budgets for
How a document is split determines what can ever be retrieved from it. Chunk too small and answers lose their context; too large and the relevant sentence is diluted by surrounding text until it stops matching. Neither problem is visible in a demo with ten documents.
- Structure-aware splitting. Sections, clauses and headings rather than a fixed character count, because a contract clause split across two chunks is retrievable from neither.
- Context preserved on the chunk. Document title, section path and date carried alongside, so a retrieved fragment is interpretable on its own.
- Overlap where boundaries are genuinely ambiguous, sized deliberately rather than left at a library default.
- Tables and lists handled separately, because splitting a table mid-row produces chunks that are worse than useless.
We test chunking strategies against real queries and measure which one retrieves the right passage, rather than picking a strategy and hoping. It is unglamorous and it moves retrieval quality more than any other single choice.
How we design the store
Choose the embedding model on your content
Embedding models differ substantially by domain, language and text length, and benchmark leaderboards are a weak predictor of performance on your corpus. We evaluate candidates on your content against your queries, and factor in dimensionality, because storage and query cost scale with it.
Design metadata as a first-class concern
Most production retrieval is filtered: this department, this date range, documents this user may see. Filters have to be designed into the schema and the index rather than applied afterwards, because post-filtering silently returns fewer results than requested and quietly degrades quality.
Enforce permissions at retrieval
Access control belongs in the query, not in a post-processing step. A vector store that returns a passage from a document the user cannot open is a data leak with extra steps, and it is one of the most common production RAG failures we are asked to fix.
Tune the index for your recall target
Approximate nearest-neighbour indexes trade recall for speed through parameters most teams leave at their defaults. We measure recall against exhaustive search on a sample and tune to your actual requirement, since the right setting for legal discovery and for a product search box are not the same.
Plan reindexing before you need it
Embedding models improve, and re-embedding a large corpus is a project. We design for it: versioned embeddings, dual-write or shadow index capability, and a rehearsed cutover, so improving your model later is an operation rather than an outage.
Keep it current
Documents change and are deleted. The store needs incremental update, deletion propagation and a reconciliation job that catches drift between source and index, because a retrieval system that confidently serves last quarter's policy is worse than no retrieval system.
The database is rarely the bottleneck
Managed and open-source vector stores are all fast enough for typical enterprise workloads. We choose on operational fit, filtering capability, permission model and cost, then spend the engagement on chunking, metadata and measurement, because that is where retrieval quality actually comes from.
Measuring retrieval, not vibes
| Question | How it is measured | Why it matters |
|---|---|---|
| Is the right passage retrieved at all? | Recall at k against expert judgements | The ceiling on every answer built on top |
| Is it near the top? | Mean reciprocal rank, precision at k | Later positions are frequently ignored downstream |
| Do filters degrade results? | Same metrics measured per filter combination | Filtered performance is often far worse and rarely tested |
| Is it fast enough under load? | Latency at p95 and p99 at expected concurrency | Averages hide the experience users complain about |
| What does it cost per query? | Embedding plus query cost at projected volume | Determines whether the design survives success |
The judgements come from your experts, on your queries. Fifty well-chosen query and answer pairs are enough to make every subsequent decision evidence-based, and building that set is part of the engagement rather than an optional extra.
How the engagement runs
A judged query set is built early, because every later decision is measured against it.
Corpus and query analysis
Content profiled, real queries collected, a judged evaluation set built with your experts.
Chunking and embedding trials
Strategies and models benchmarked on your content against the judged set.
Build
Store deployed, schema and metadata designed, ingestion and permission filtering implemented.
Tuning and load testing
Index parameters tuned to the recall target; latency and cost measured at projected concurrency.
Handover
Reindexing procedure rehearsed, runbook and evaluation harness handed over.
What you receive
A retrieval layer with measured quality and a rehearsed path to changing the embedding model later.
Deployed vector store
Provisioned, indexed and integrated, with incremental update and deletion handling.
Ingestion pipeline
Chunking, embedding, metadata extraction and reconciliation against source.
Permission filtering
Access control enforced within the query rather than after it.
Judged evaluation set
Your queries with expert relevance judgements, reusable for every future change.
Benchmark report
Chunking and embedding options compared on recall, latency and cost.
Reindexing runbook
Rehearsed procedure for changing embedding model without downtime.
Is this the right engagement?
Worth being direct. Vector Database Design and Implementation is the wrong spend in some situations, and those are listed rather than buried.
Good fit if
- A search or RAG system returns plausible but wrong passages.
- Retrieval must respect per-user permissions on the underlying documents.
- The corpus is large enough that chunking and index tuning materially affect quality.
- You expect to change embedding models and want that to be routine.
- Costs at projected query volume need to be understood before launch.
Choose something else if
- The corpus is a few hundred documents; simpler retrieval may be sufficient.
- The need is the full application rather than the retrieval layer. See RAG development.
- The requirement is ranked search over structured records rather than passages.
- No expert time is available to judge relevance, which leaves nothing to tune against.
Frequently asked questions
Marked up with FAQPage schema so these answers can surface directly in search results and inside AI assistant responses.
Which vector database should we use?
For most enterprise workloads the managed and open-source options are all fast enough, so we choose on filtering capability, permission model, operational fit with your existing platform, and cost at your volume. Where your warehouse or lakehouse already offers competent vector support, using it avoids a whole system to operate — which is frequently the right answer.
How is this different from semantic search?
Semantic and vector search is the end-to-end search experience: query understanding, hybrid keyword and vector retrieval, reranking and the interface. This engagement is the storage and retrieval layer underneath it, which also serves RAG and recommendation systems. Where you need the whole search product, start there.
What chunk size should we use?
There is no universal answer, which is why we test rather than assert. Structure-aware chunks that follow sections or clauses generally outperform fixed-size splitting, and the right size depends on your document type and query style. We benchmark options against judged queries on your own corpus.
What happens when we want a better embedding model?
The whole corpus has to be re-embedded, which is why reindexing is designed in from the start: versioned embeddings, the ability to build a shadow index alongside the live one, and a rehearsed cutover. Without that, a model upgrade becomes a project people keep deferring.
Can the vector store enforce who is allowed to see what?
Yes, and it must. Permissions are applied as filters within the retrieval query so that a user can never receive a passage from a document they are not entitled to open. Applying access control after retrieval is a common shortcut and a reliable source of leaks.
Often paired with this
Most clients combine two or three engagements from the Data Engineering and AI Readiness pillar. These are the ones that most often run immediately before or after.
Data Lakehouse and Warehouse Build
A governed lakehouse or warehouse with a modelled core layer, access control and predictable cost.
Read more →Data Cleaning, Quality and Enrichment
Profiling, deduplication, standardisation and enrichment, with automated checks that stop quality decaying again.
Read more →Data Labelling and Annotation
Labelled datasets with a written guideline, measured annotator agreement and audited quality at every batch.
Read more →Is this the right engagement?
Tell us what you are trying to build. If a different service fits better, or if you do not need us at all, we will say so.