RAG Architecture: Building Reliable LLM Applications on Private Data
Retrieval Augmented Generation combines vector search with LLMs. We cover chunking, embeddings, re-ranking, and RAGAS evaluation.
Get one deep-dive article every Saturday. No spam. Unsubscribe anytime.
Large Language Models hallucinate. They confidently produce wrong answers when asked about information they weren't trained on — your internal documents, your product data, your customers' history. Retrieval Augmented Generation (RAG) is the standard pattern to fix this.
How RAG Works
RAG has two phases:
Indexing (offline): Your documents are chunked, embedded into vectors, and stored in a vector database. Each chunk is a searchable unit.
Retrieval + Generation (online): When a user asks a question, the question is embedded and the top-K most similar chunks are retrieved. These chunks are injected into the LLM prompt as context, and the LLM answers based on that context — not from memory.
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_pinecone import PineconeVectorStore
from langchain.chains import RetrievalQA
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = PineconeVectorStore.from_existing_index("my-index", embeddings)
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
chain = RetrievalQA.from_chain_type(
llm=llm,
retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
)
answer = chain.invoke("What is our refund policy for enterprise customers?")Chunking Strategy
How you split documents dramatically affects retrieval quality. Get this wrong and you'll retrieve chunks that are semantically incomplete.
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=64,
separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_documents(documents)Get unlimited access to all LLMs tutorials & animated diagrams
1,000+ animated tutorials · 100+ AI tools · Mock interviews · Certifications
Vector Database Options
Re-ranking for Better Precision
Vector similarity retrieves semantically similar chunks but doesn't guarantee relevance. A re-ranker scores (query, chunk) pairs and reorders them.
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
def rerank(query: str, chunks: list[str], top_n: int = 3) -> list[str]:
pairs = [(query, chunk) for chunk in chunks]
scores = reranker.predict(pairs)
ranked = sorted(zip(scores, chunks), reverse=True)
return [chunk for _, chunk in ranked[:top_n]]Evaluating RAG Quality with RAGAS
RAGAS measures:
Get unlimited access to all LLMs tutorials & animated diagrams
1,000+ animated tutorials · 100+ AI tools · Mock interviews · Certifications
Common RAG Failure Modes
1. Wrong chunk retrieved — Fix: add keyword search (BM25) alongside vector search (hybrid search).
2. Chunk boundary cuts off context — Fix: use semantic chunking or parent document retrieval.
3. LLM ignores retrieved context — Fix: explicit prompt instruction, lower temperature.
4. Too many irrelevant chunks — Fix: reduce K, add re-ranking, increase similarity threshold.
Get unlimited access to all LLMs tutorials & animated diagrams
1,000+ animated tutorials · 100+ AI tools · Mock interviews · Certifications