How to Build a RAG System: Step-by-Step Tutorial with LangChain and Pinecone

You have learned what RAG is. Now it is time to build one. This tutorial walks you through creating a fully functional document Q&A chatbot that can answer questions from your own documents using LangChain, Pinecone, and OpenAI.

By the end of this tutorial, you will have a working RAG system that you can adapt for any knowledge base, deploy as an API, and add to your portfolio.

What You Will Build

A chatbot that reads your PDF documents, stores them in a vector database, and answers questions based on the actual content of those documents, with source citations. No hallucinations, no outdated training data, just accurate answers from your own knowledge base.

Prerequisites

• Basic Python knowledge (variables, functions, installing packages)
• An OpenAI API key (sign up at platform.openai.com)
• A Pinecone account (free tier at pinecone.io)
• Python 3.9+ installed on your machine

Step 1: Set Up Your Environment

Install the required Python packages:

pip install langchain langchain-openai langchain-pinecone pinecone-client pypdf python-dotenv

Create a .env file with your API keys:

OPENAI_API_KEY=your-key-here

PINECONE_API_KEY=your-key-here

Step 2: Load and Chunk Your Documents

The first step is reading your documents and splitting them into chunks that are small enough for precise retrieval but large enough to maintain context.

Use LangChain's PyPDFLoader to read PDFs and RecursiveCharacterTextSplitter to break them into chunks of roughly 500 characters with 50 characters of overlap between chunks. The overlap ensures that concepts split across chunk boundaries are still captured.

Why chunk size matters: Chunks that are too large return irrelevant information alongside the relevant parts. Chunks that are too small lose context. 500 characters with 50-character overlap is a good starting point for most document types.

Step 3: Create Embeddings and Store in Pinecone

Convert each text chunk into a numerical vector using OpenAI's embedding model (text-embedding-3-small). These vectors capture the semantic meaning of each chunk. Then store these vectors in Pinecone, where they can be searched by similarity.

When you create your Pinecone index, set the dimension to match your embedding model (1536 for text-embedding-3-small) and use cosine similarity as the distance metric.

Step 4: Build the Retrieval Chain

This is where LangChain brings everything together. Create a retrieval chain that: (1) takes the user's question, (2) converts it to an embedding, (3) searches Pinecone for the most similar document chunks, (4) passes those chunks as context to the LLM, and (5) generates an answer grounded in the retrieved documents.

Use LangChain's RetrievalQA chain with a prompt template that instructs the LLM to answer based only on the provided context and to cite which documents it used.

Step 5: Test and Refine

Ask your chatbot questions and evaluate the responses:

• Accuracy: Does it answer correctly based on your documents?
• Relevance: Does it retrieve the right chunks for each question?
• Completeness: Does it provide enough detail?
• Honesty: Does it say "I don't know" when the answer is not in the documents?

Common refinements include adjusting chunk size, adding metadata filtering, tuning the number of retrieved chunks (k parameter), and improving your prompt template.

Taking It Further

• Add a web interface using Streamlit or Gradio for a user-friendly chatbot experience.
• Deploy as an API using FastAPI and Docker for production use.
• Add metadata filtering to restrict retrieval to specific document categories or date ranges.
• Implement conversation memory so the chatbot remembers previous questions in the session.
• Support multiple document types including web pages, Word documents, and database records.

Want Expert Guidance Building This?

Our Generative AI Course includes this exact project in Module 3, with live instructor guidance, code reviews, and deployment support. You build a production-quality RAG system, not just a tutorial notebook.

Frequently Asked Questions

How do I build a RAG system?

Build a RAG system in 5 steps: (1) Load and chunk your documents, (2) Create embeddings using an embedding model, (3) Store embeddings in a vector database like Pinecone, (4) Build a retrieval chain with LangChain, (5) Connect to an LLM for answer generation. The entire pipeline can be built in under 100 lines of Python code.

What tools do I need for RAG?

Essential tools: Python, LangChain (orchestration), an embedding model (OpenAI text-embedding-3), a vector database (Pinecone or ChromaDB), and an LLM (GPT-4, Claude, or Llama). LangChain connects all these components with pre-built integrations.

Can beginners build a RAG chatbot?

Yes, with basic Python knowledge and the right tutorial. The concepts are straightforward, and LangChain handles most of the complexity. A beginner can build a working RAG chatbot in a weekend. Making it production-quality takes more experience, which is where guided projects help.

What is a good chunk size for RAG?

500 characters with 50-character overlap is a good starting point. Adjust based on your documents: shorter chunks for precise factual retrieval (FAQs, definitions), longer chunks for narrative content (reports, articles). Always test and measure retrieval quality.