each AI engineer is aware of effectively. You’ve got simply shipped a proof of idea. The demo went brilliantly. The LLM answered questions fluently, synthesised info on the fly, and impressed everybody within the room. Then somebody requested it concerning the firm’s refund coverage, and it confidently gave the improper reply, one which had not been true for eight months.
That second just isn’t a mannequin failure. It’s an structure failure. And it’s precisely the issue that Retrieval-Augmented Era, or RAG, was designed to resolve.
This text walks via constructing a production-grade RAG system for an enterprise inside data base, utilizing a totally open-source stack. We’ll transfer from the issue to the design, via every stage of the pipeline, and end with the way you truly know whether or not the system is working. The purpose is to not cowl each potential variation however is to present you a transparent psychological mannequin and a sensible basis you may construct on.
What We Will Cowl
- Why LLMs alone aren’t sufficient for enterprise data retrieval
- The RAG structure: how the 2 pipelines match collectively
- Constructing the indexing pipeline: loading, chunking, embedding, and storing
- Constructing the retrieval and era pipeline: search, re-ranking, and prompting
- Analysis: measuring high quality at each stage, not simply the top
- The place RAG ends and fine-tuning begins
The Drawback Value Fixing
Most medium-to-large organisations sit on 1000’s of inside paperwork, Engineering runbooks, HR insurance policies, Compliance pointers, Onboarding guides, Product specs. They dwell throughout Confluence, SharePoint, Notion, shared drives, and e-mail threads that no person has touched in three years.
The typical worker spends two to 3 hours per week merely wanting for info that already exists someplace. Senior engineers develop into unintended assist brokers. New joiners take months to develop into independently productive, not as a result of they lack capacity, however as a result of institutional data is scattered and unsearchable.
The naive response is to level an LLM in any respect of this and ask it questions. The issue is that LLMs are static. As soon as skilled, they haven’t any data of your newest product launch, the coverage that modified final quarter, or the autopsy your workforce printed yesterday. Nice-tuning helps with model and tone, however it’s costly, sluggish to replace, and it doesn’t let you know the place a solution got here from. In a regulated trade, that auditability hole just isn’t acceptable.
RAG threads the needle. At question time, the system retrieves essentially the most related paperwork out of your data base and offers them to the LLM as context. The mannequin generates a solution grounded in these paperwork, not in what it realized throughout coaching. Each reply is traceable to a supply. The data base might be up to date in minutes. And nothing wants to depart your infrastructure.
The Structure
Earlier than going into the person elements, it helps to see the form of the entire system. RAG just isn’t a single mannequin, it’s two pipelines working collectively.
The indexing pipeline runs as soon as while you first arrange the system, after which incrementally every time paperwork are added or modified. Its job is to take uncooked paperwork, break them into significant chunks, convert these chunks into vector representations, and retailer them.
The retrieval and era pipeline runs on each person question. It takes the query, finds essentially the most related chunks, assembles them right into a immediate, and asks the LLM to generate a solution grounded in that context.
The 2 pipelines share the vector retailer as their assembly level. That single design resolution, separating indexing from retrieval, is what makes the entire system updatable with out retraining.
Part One: The Indexing Pipeline
Loading Your Paperwork
The primary problem is solely getting your paperwork right into a usable type. Enterprise data is never in a single place or one format.
For this, we use LlamaIndex. The place LangChain affords doc loaders, LlamaIndex goes additional: it ships over 100 native connectors for programs like Confluence, Notion, SharePoint, Google Drive, and S3, and it tracks doc hashes in order that solely modified information are re-indexed on subsequent runs. For a data base that’s always evolving, that incremental sync just isn’t a nice-to-have, it’s important.
from llama_index.readers.confluence import ConfluenceReader
from llama_index.core import SimpleDirectoryReader
# Pull from Confluence
confluence_docs = ConfluenceReader(
base_url="https://yourcompany.atlassian.internet/wiki",
oauth2={"client_id": "...", "token": "..."}
).load_data(space_key="ENGG", page_status="present")
# Pull from an area listing (PDFs, Markdown, DOCX)
local_docs = SimpleDirectoryReader(
input_dir="./knowledge_base",
required_exts=[".pdf", ".docx", ".md"],
recursive=True
).load_data()What to examine right here: Log what number of paperwork loaded efficiently, what number of had been skipped, and whether or not any failed silently. A loader failure at this stage creates a data hole that may manifest as a improper or lacking reply later and it is going to be very tough to hint again.
Chunking: The Step That Most Groups Get Incorrect
Should you take one factor from this text, let or not it’s this: the standard of your chunking has extra affect in your system’s efficiency than your alternative of LLM and even your embedding mannequin.
The reason being simple. When a person asks a query, the system retrieves chunks not full paperwork. If a piece cuts off mid-argument, or splits a desk throughout two segments, or is so giant it dilutes the sign, the retrieval system can’t do its job correctly.
Easy fixed-size splitting: reducing each 512 tokens with no consciousness of sentence or paragraph boundaries is fast to implement and persistently mediocre. For enterprise content material, we use LlamaIndex’s SentenceWindowNodeParser, which indexes on the sentence degree for exact retrieval however expands to a surrounding window of sentences when producing the reply. You get surgical retrieval with out shedding the context that makes a solution coherent.
from llama_index.core.node_parser import SentenceWindowNodeParser
parser = SentenceWindowNodeParser.from_defaults(
window_size=3, # 3 sentences both facet at era time
window_metadata_key="window",
original_text_metadata_key="original_text"
)
nodes = parser.get_nodes_from_documents(all_docs)For longer paperwork like coverage information or technical runbooks, a hierarchical strategy works higher: index on the paragraph degree, however return the total part when producing. The correct chunking technique relies on your content material kind; there is no such thing as a common reply.
What to examine right here: Manually evaluation round fifty random chunks. Ask your self whether or not each may stand alone as a significant reply to some query. If multiple in 5 really feel like sentence fragments or orphaned clauses, your chunk dimension is just too small or your overlap is inadequate.
Turning Textual content Into Vectors
Every chunk must be transformed right into a numerical vector in order that we will measure similarity between a question and a doc. That is the job of the embedding mannequin, and the selection issues greater than many engineers realise.
We use BAAI/bge-large-en-v1.5, an open-source mannequin from the Beijing Academy of AI, which is among the many top-performing open-source fashions on the MTEB benchmark. It runs solely regionally, which for many enterprises just isn’t non-obligatory however obligatory. Sending inside paperwork to an exterior embedding API is a knowledge residency concern that may cease a manufacturing rollout in its tracks.
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
embed_model = HuggingFaceEmbedding(
model_name="BAAI/bge-large-en-v1.5",
query_instruction="Signify this sentence for looking out related passages: "
)The instruction prefix on the final line is restricted to BGE fashions and value holding. It’s an uneven retrieval optimisation that measurably improves precision. One rule to deal with as absolute: the identical embedding mannequin should be used for each indexing and querying. These two operations produce vectors that dwell in the identical mathematical area. Mixing fashions, even upgrading to a more recent model mid-deployment, breaks that area and renders your index meaningless.
What to examine right here: Run your twenty most typical queries towards a small check index and examine the similarity scores. Persistently scoring beneath 0.6 on queries you realize ought to match effectively alerts a website mismatch. Take into account fine-tuning the embedder on a pattern of your inside corpus.
Storing Vectors: Why Weaviate
The vector retailer is the place all of the listed chunks dwell, able to be searched. We use Weaviate, self-hosted, and the explanations are value being specific about.
Most vector databases do one factor: retailer vectors and discover the closest neighbours. Weaviate does that, nevertheless it additionally affords one thing that enterprise deployments genuinely want: native hybrid search, combining dense semantic vectors with BM25 key phrase search in a single question name. This issues as a result of enterprise customers don’t search the best way a basic net person does. They search with precise product names, inside ticket IDs, workforce abbreviations, and jargon that embedding fashions deal with poorly. A question for “GDPR Article 17 compliance guidelines” comprises a particular time period that semantic similarity will dilute. BM25 will discover it instantly.
Past hybrid search, Weaviate affords native multi-tenancy – you may partition the index by division, so an HR question by no means unintentionally surfaces engineering structure paperwork, and entry management is enforced on the database degree relatively than bolted on in utility code.
import weaviate
from weaviate.courses.config import Configure, Property, DataType
shopper = weaviate.connect_to_local(host="localhost", port=8080, grpc_port=50051)
shopper.collections.create(
title="EnterpriseKB",
vectorizer_config=Configure.Vectorizer.none(),
properties=[
Property(name="text", data_type=DataType.TEXT),
Property(name="source", data_type=DataType.TEXT),
Property(name="department", data_type=DataType.TEXT),
Property(name="classification", data_type=DataType.TEXT),
Property(name="updated_at", data_type=DataType.DATE),
]
)Qdrant is a powerful various in case you are beginning small and need less complicated operations. pgvector is cheap in case you are already on Postgres and don’t want horizontal scale. However for an enterprise deployment the place hybrid search, entry management, and multi-team isolation matter, Weaviate is the correct device.
Part Two: Retrieval and Era
Discovering the Proper Chunks
When a person submits a question, the primary job is retrieval: discover the chunks most certainly to include the reply. We embed the question utilizing the identical mannequin as indexing, then search Weaviate with hybrid mode enabled.
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.vector_stores import MetadataFilter, MetadataFilters
retriever = VectorIndexRetriever(
index=index,
similarity_top_k=10,
vector_store_query_mode="hybrid",
alpha=0.75, # Mix: 75% semantic, 25% key phrase
vector_store_kwargs={
"filters": MetadataFilters(filters=[
MetadataFilter(key="department", value="engineering"),
MetadataFilter(key="classification", value="confidential",
operator=FilterOperator.NE)
])
}
)The alpha parameter controls the mix between semantic and key phrase search. A worth of 0.75 tilts in the direction of semantic similarity whereas nonetheless giving key phrase matches significant weight. Chances are you’ll must tune this primarily based in your content material, domains with lots of exact technical terminology typically profit from a decrease alpha.
Measuring retrieval high quality requires a labelled analysis set: a group of queries paired with the paperwork that ought to be returned. Your IT helpdesk ticket historical past is a sensible supply for this, actual worker questions with documented resolutions. The metrics to trace are Hit Price at Okay (does the correct doc seem within the prime Okay outcomes?), Imply Reciprocal Rank (how excessive within the listing does the primary appropriate end result seem?), and Context Precision (what quantity of retrieved chunks are literally related?).
An affordable goal for a manufacturing system is a Hit Price above 0.80 at Okay=5.
Re-ranking: The Refinement Move
Vector search is quick and scales effectively, nevertheless it has a identified weak spot: it compares question and doc independently as separate vectors. Two paperwork may need related vectors to a question however just one genuinely solutions it.
A cross-encoder re-ranker addresses this by reading the query and each document together and scoring true semantic alignment. It is slower, but applied only to the top ten candidates from retrieval, the added latency is fifty to a hundred milliseconds and is usually acceptable.
We use ms-marco-MiniLM-L-6-v2, a well-tested open-source cross-encoder trained on search relevance data. LlamaIndex integrates it cleanly into the query engine as a post-processor, so there is no custom orchestration required.
Re-ranking is worth adding when your queries are long or ambiguous, or when you notice that retrieval finds vaguely relevant documents but misses the best one. If your embedding model is already well-suited to your domain and retrieval precision is high, skip it the latency cost is not always justified.
The Local LLM: Keeping Data In-House
For many enterprises, especially those in regulated sectors, sending internal documents to an external LLM API is simply not on the table. GDPR, data residency requirements, and commercial confidentiality concerns all push towards on-premise inference.
Ollama makes this straightforward. It packages open-source models with a runtime and a simple API, letting you run Llama 3.1 locally with a single command. For an 8-billion parameter model, a single 16 GB GPU is sufficient. For higher accuracy at the cost of compute, the 70-billion parameter variant requires roughly 80 GB of GPU memory; achievable on a small cluster.
from llama_index.llms.ollama import Ollama
llm = Ollama(
model="llama3.1:8b",
temperature=0.1, # Low temperature for factual retrieval tasks
context_window=8192,
request_timeout=120.0
)Temperature deserves a word here. For factual question-answering against a knowledge base, you want the model to be deterministic and conservative. A temperature of 0.1 keeps the model tightly grounded in the provided context. Raising it above 0.4 increases the risk of the model interpolating beyond what the retrieved chunks actually say.
Assembling the Prompt
Prompt engineering for RAG is often treated as an afterthought, which is a mistake. The way you frame the context and the instruction directly determines whether the model stays grounded or drifts into hallucination.
The essentials are: tell the model explicitly that it must answer using only the provided context; give it a clear fallback instruction for when the answer is not in the context; and ask it to cite the source document. The last point is not just useful for users but it makes errors auditable.
from llama_index.core import PromptTemplate
qa_prompt = PromptTemplate(
"""You are a knowledgeable assistant for the internal knowledge base.
Answer the question using only the context provided below.
If the answer is not clearly present in the context, say so honestly and suggest
the employee contact the relevant team directly.
Always end your answer by citing the source document(s) you used.
Context:
{context_str}
Question: {query_str}
Answer:"""
)LlamaIndex’s RetrieverQueryEngine wires retrieval, re-ranking, prompt assembly, and generation together. The MetadataReplacementPostProcessor handles expanding the compressed sentence chunks back to their full window before they are passed to the LLM.
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.postprocessor import (
MetadataReplacementPostProcessor,
SentenceTransformerRerank
)
query_engine = RetrieverQueryEngine.from_args(
retriever=retriever,
llm=llm,
node_postprocessors=[
MetadataReplacementPostProcessor(target_metadata_key="window"),
SentenceTransformerRerank(model="cross-encoder/ms-marco-MiniLM-L-6-v2", top_n=3)
],
text_qa_template=qa_prompt
)
response = query_engine.query("What is the process for requesting production database access?")
print(response.response)
for node in response.source_nodes:
print(f"Source: {node.metadata.get('source')} - score: {node.score:.3f}")Evaluating the Full Pipeline
Building a RAG system without an evaluation framework is like shipping software without tests. You cannot know whether a change improved or degraded the system unless you have a baseline to compare against.
RAGAS (Retrieval Augmented Generation Assessment) is the standard open-source framework for this. Its most valuable property is that it does not require pre-labelled gold answers for every question, it uses an LLM as a judge internally, which makes it scalable to hundreds of evaluations per run.
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall, context_precision
result = evaluate(
dataset=eval_dataset, # query, contexts, answer, ground_truth
metrics=[faithfulness, answer_relevancy, context_recall, context_precision],
llm=LlamaIndexLLMWrapper(llm),
embeddings=LlamaIndexEmbeddingsWrapper(embed_model)
)The four core metrics – each one catches a different category of failure:
Faithfulness checks whether the answer is actually supported by the retrieved context. A low faithfulness score means the LLM is hallucinating, generating claims that go beyond what the documents say. This is the most critical metric for enterprise use.
Answer Relevancy measures whether the response actually addresses the question asked. A model can be perfectly faithful (only saying things the context supports) but still give an irrelevant answer.
Context Recall checks whether the retrieval step surfaced the information that was needed. If this is low, the problem is in your retrieval, not your generation.
Context Precision measures what proportion of the retrieved chunks were genuinely useful. High retrieved chunks with low precision means you are passing noise to the LLM, which degrades generation quality.
For a production system, reasonable targets are faithfulness above 0.90, answer relevancy above 0.85, context recall above 0.80, and context precision above 0.75. These are not fixed rules, but if you are significantly below any of them, you have a clear signal of where to focus your debugging effort.
RAG or Fine-tuning? The Honest Answer
This question comes up in almost every conversation about LLMs in enterprise, and it is worth addressing directly rather than hedging.
Fine-tuning is the right tool when you want to change how a model behaves: its tone, its reasoning pattern, how it structures responses, the vocabulary it uses. It bakes those properties into the model weights. Updating that knowledge later requires another fine-tuning run.
RAG is the right tool when you want to change what a model knows: the facts, policies, and documents it can draw on. Updating knowledge is a matter of re-indexing documents, which takes minutes.
The two are not in competition. The most robust production systems use both: a model fine-tuned on the company’s writing style and internal terminology, combined with RAG for knowledge grounding. Fine-tuning gives you consistency of voice; RAG gives you factual accuracy and auditability.
The common mistake is reaching for fine-tuning when a document is “too important to risk the model getting wrong.” Fine-tuning does not guarantee accuracy it just makes the model more confident. RAG, with a well-maintained index and a strict grounding prompt, gives you something fine-tuning cannot: a direct line from every answer back to its source.
Common Failure Modes
A few patterns appear often enough to be worth naming explicitly.
The most common problem is not hallucination but it is the retrieval failure. The model cannot answer correctly if the right chunk was never retrieved. Before blaming the LLM, check your Hit Rate on your evaluation set. If it is below 0.70, start with chunking and embedding quality, then consider hybrid search if you are not using it already.
Stale knowledge is the second most common issue in production. A document was updated, but the index was not. The fix is operational: set up an incremental re-indexing job triggered by document change events in Confluence or SharePoint, rather than running a full re-index on a schedule.
The third pattern is context that is technically retrieved but ignored by the model: the “lost in the middle” problem. LLMs weight the beginning and end of the context window more heavily than the middle. If you are passing ten chunks, the most relevant one should be first. Reduce your top-K and ensure your re-ranker is ordering correctly.
Before You Ship
A short checklist that reflects the gap between a working prototype and a system you would stake your reputation on:
- Evaluate Hit Rate at K=5 on at least 150 labelled queries; target above 0.85
- Run RAGAS faithfulness on 100 or more query-answer pairs; target above 0.90
- Configure Weaviate tenant isolation if deploying across multiple departments
- Set up incremental re-indexing triggered by document change events
- Add a low-confidence fallback: if the top retrieval score is below 0.55, return an honest “I could not find a reliable answer” rather than guessing
- Implement query logging with a user feedback mechanism: this becomes your ongoing evaluation dataset
Conclusion
RAG does not make your LLM smarter. It makes it honest.
The difference between a system your colleagues trust and one they quietly stop using after a fortnight usually has nothing to do with which model you picked. It comes down to whether the retrieval is precise enough to find the right chunk, whether the prompt is disciplined enough to keep the model grounded in it, and whether you have the evaluation in place to know when either of those things starts to degrade.
The pipeline described in this article is not the only way to build a RAG system. It is a set of deliberate choices each one made for a specific reason that collectively produce something you can deploy in a regulated environment, hand to a non-technical stakeholder, and stand behind when someone asks where an answer came from.
That last part matters more than any benchmark score. In enterprise settings, trust is the product. Everything else is just infrastructure.
