RAG with LangChain — Chat With Your Own Documents

An LLM only knows what it was trained on — not your lecture notes, your PDF syllabus, or last week's circular. RAG (Retrieval-Augmented Generation) fixes that: before answering, you fetch the relevant chunks of your documents and hand them to the model as context. It's the single most useful pattern in applied LLMs.

The RAG pipeline in five steps

Load your documents (text, PDF, web page).
Split them into small chunks.
Embed each chunk into a vector (a list of numbers capturing meaning).
Store the vectors in a vector store.
Retrieve the closest chunks to the question, then generate an answer from them.

Steps 1-4: build the knowledge base

pip install langchain-community langchain-text-splitters langchain-huggingface

from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_core.vectorstores import InMemoryVectorStore

# 1. load
docs = TextLoader("anna_univ_notes.txt").load()

# 2. split — small overlapping chunks so meaning is not cut mid-sentence
chunks = RecursiveCharacterTextSplitter(
    chunk_size=500, chunk_overlap=50).split_documents(docs)

# 3. embed — a FREE local model, no API key needed
embeddings = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-MiniLM-L6-v2")

# 4. store
store = InMemoryVectorStore(embeddings)
store.add_documents(chunks)
retriever = store.as_retriever(search_kwargs={"k": 3})   # top 3 chunks

Why split? Retrieval works best on small, focused chunks — and you can only fit so much in the prompt. Why embed? Vectors let you find text by meaning, not keywords: "exam re-check" matches a chunk about "revaluation" even with no shared words.

Step 5: retrieve + generate (the RAG chain)

from langchain.chat_models import init_chat_model
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

model = init_chat_model("claude-opus-5", model_provider="anthropic")

prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer ONLY from the context below. If it is not there, "
               "say you do not know.\n\nContext:\n{context}"),
    ("user", "{question}"),
])

def format_docs(docs):
    return "\n\n".join(d.page_content for d in docs)

rag = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt | model | StrOutputParser()
)

print(rag.invoke("What is the revaluation deadline?"))

Read the chain: the question goes to retriever (fetch 3 chunks) and passes through unchanged, both feed the prompt, then the model answers grounded in your text.

Why "answer only from context" matters

That instruction is your guardrail against hallucination. Grounding the model in retrieved facts — and telling it to admit when the answer isn't there — is what makes RAG trustworthy enough to ship.

From toy to real

Bigger docs? Swap InMemoryVectorStore for a persistent one (Chroma, FAISS, or pgvector) so you embed once, not every run.
PDFs? Use PyPDFLoader; websites, WebBaseLoader.
Cite sources? Return the chunks' metadata alongside the answer.

RAG answers questions. To let the model take actions — search, calculate, call an API — you need tools and agents, next. Related: LCEL chains power the pipeline above.

← All Articles