A comprehensive Retrieval-Augmented Generation (RAG) implementation combining Llama2 language model with LangChain framework and ChromaDB vector database for advanced document question-answering capabilities.
This project demonstrates a complete RAG pipeline using:
- Llama2: Open-source large language model for generation
- LangChain: Framework for building LLM applications
- ChromaDB: Vector database for semantic search
- Local Models: Fully local setup without external APIs (optional)
Perfect for building private, offline-capable document QA systems.
- Implement complete RAG system
- Use open-source models (Llama2)
- Demonstrate LangChain capabilities
- Leverage ChromaDB for vector storage
- Build offline-capable applications
- Create scalable document indexing
- Enable multi-document QA
RAG-using-llama2-langchain-and-chromadb/
βββ RAG using llama2 langchain and chromadb.ipynb # Main notebook
βββ README.md # Documentation
βββ data/ # Documents (optional)
βββ documents/
βββ chroma_db/
βββ models/
-
LLM: Llama2 (via Ollama or HuggingFace)
-
Framework: LangChain
-
Vector DB: ChromaDB
-
Embeddings: Sentence-Transformers or HuggingFace
-
Document Processing:
- PyPDF2
- LangChain Document Loaders
- TextSplitter
-
Utilities: NumPy, Pandas
-
Complete RAG Pipeline:
- Document ingestion
- Text chunking and splitting
- Embedding generation
- Vector storage
- Semantic search
- Answer generation
-
LangChain Integration:
- Document loaders
- Text splitters
- Vector stores
- LLM chains
- Memory management
- Prompt templates
-
ChromaDB Features:
- Persistent storage
- Similarity search
- Metadata filtering
- Collection management
- Scalable indexing
-
Llama2 Support:
- Local inference (via Ollama)
- HuggingFace integration
- GGML quantized models
- CPU and GPU support
- Python 3.8+
- Ollama (for local Llama2) - Install
- 8GB+ RAM (16GB+ for Llama2)
- Document files for indexing
-
Clone the repository:
git clone <repository-url> cd RAG-using-llama2-langchain-and-chromadb
-
Install dependencies:
pip install -r requirements.txt
-
Set up Llama2 locally (optional):
# Install Ollama from ollama.ai ollama pull llama2 ollama serve -
Open the notebook:
jupyter notebook "RAG using llama2 langchain and chromadb.ipynb"
Document Upload
β
Load Documents (PDF, TXT, etc.)
β
Split Text into Chunks
β
Generate Embeddings
β
Store in ChromaDB
β
User Query
β
Generate Query Embedding
β
Semantic Search in ChromaDB
β
Retrieve Top-K Documents
β
Build Prompt with Context
β
Llama2 Generation
β
Answer + Source Citation
import chromadb
from langchain.vectorstores import Chroma
# Initialize ChromaDB
chroma_client = chromadb.Client()
collection = chroma_client.create_collection(
name="documents",
metadata={"hnsw:space": "cosine"}
)
# Or use persistent client
persistent_client = chromadb.PersistentClient(
path="/path/to/data"
)from langchain.llms import Ollama
# Use local Ollama
llm = Ollama(model="llama2", base_url="http://localhost:11434")
# Or via HuggingFace
from langchain.llms import HuggingFacePipeline
llm = HuggingFacePipeline(model_name="meta-llama/Llama-2-7b-chat-hf")from langchain.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.vectorstores import Chroma
from langchain.chains import RetrievalQA
# Load documents
loader = PyPDFLoader("document.pdf")
documents = loader.load()
# Split text
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200
)
texts = splitter.split_documents(documents)
# Create embeddings
embeddings = HuggingFaceEmbeddings()
# Create vector store
vectorstore = Chroma.from_documents(
texts,
embeddings,
persist_directory="./chroma_db"
)
# Create QA chain
qa = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=vectorstore.as_retriever()
)# Simple query
question = "What are the main topics discussed?"
answer = qa.run(question)
print(f"Answer: {answer}")from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate
# Custom prompt
template = """Use the provided context to answer the question.
Context: {context}
Question: {question}
Answer: """
prompt = PromptTemplate(
template=template,
input_variables=["context", "question"]
)
# Create QA chain with custom prompt
qa = RetrievalQA.from_chain_type(
llm=llm,
chain_type="refine", # Better for complex docs
retriever=vectorstore.as_retriever(k=5),
chain_type_kwargs={
"prompt": prompt,
"verbose": True
}
)
# Query
result = qa({"query": "What is the document about?"})
print(result)from langchain.document_loaders import DirectoryLoader
# Load all PDFs from directory
loader = DirectoryLoader(
"documents/",
glob="**/*.pdf",
loader_cls=PyPDFLoader
)
documents = loader.load()
# Process and store
texts = splitter.split_documents(documents)
vectorstore = Chroma.from_documents(
texts,
embeddings,
persist_directory="./chroma_db"
)from langchain.chains import RetrievalQAWithSourcesChain
# Create QA chain with sources
qa_with_sources = RetrievalQAWithSourcesChain.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=vectorstore.as_retriever()
)
# Query returns answer with sources
result = qa_with_sources("What is the main topic?")
print(f"Answer: {result['answer']}")
print(f"Sources: {result['sources']}")# Text splitting
CHUNK_SIZE = 1000
CHUNK_OVERLAP = 200
# Embedding
EMBEDDING_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
# Retrieval
TOP_K = 5
SIMILARITY_THRESHOLD = 0.5
# Generation
TEMPERATURE = 0.7
MAX_TOKENS = 500| Model | Size | Speed | Quality | Notes |
|---|---|---|---|---|
| Llama2 7B | 3.8GB | Fast | Good | Via Ollama |
| Llama2 13B | 7.3GB | Medium | Better | Needs more VRAM |
| Llama2 70B | 39GB | Slow | Excellent | Requires A100 GPU |
- Save ChromaDB periodically
- Use proper chunk sizes for your domain
- Test retrieval quality
- Include metadata in documents
- Monitor embedding quality
- Validate LLM outputs
- Use appropriate chain types
- Store API keys securely
- Metadata Filtering: Filter by date, author, etc.
- Hybrid Search: Combine BM25 + semantic search
- Conversation History: Maintain context across turns
- Multi-turn QA: Complex reasoning chains
- Answer Confidence: Score answer reliability
- Document Versioning: Track document updates
- Quantization: Use GGML quantized Llama2
- Batch Processing: Process multiple queries
- Caching: Cache embeddings
- Chunking: Optimize chunk size
- GPU Acceleration: Use CUDA if available
- Model Compression: Use smaller variants
Contributions are welcome! Please:
- Fork the repository
- Create a feature branch
- Commit your changes
- Push to the branch
- Submit a pull request
- Retrieval-Augmented Generation
- LLaMA: Open and Efficient Foundation Language Models
- LangChain Documentation
- ChromaDB Documentation
- Ollama
This project is open source and available under the MIT License.
- β Fully local - no API costs
- β Privacy-first - data stays on your machine
- β Open-source - full transparency
- β Customizable - modify any component
- β Offline-capable - works without internet
- β Scalable - handle large document collections
- Slower than cloud APIs
- Requires local compute resources
- Model quality depends on size/version
- Setup complexity vs. managed solutions
For questions or suggestions, please open an issue or contact the project maintainers.
Built for privacy-first, offline-capable RAG systems using open-source models π