Rag project

Monday, August 4, 2025 10:29 PM

) Imports
python
CopyEdit
import streamlit as st
Loads Streamlit and aliases it as st so you can build a web UI with functions like st.title, st.file_uploader,
etc. Streamlit re-runs the script top-to-bottom on every user interaction, so stateful things should go in
st.session_state.

python
CopyEdit
import os
import tempfile
import time
• os: read environment variables and work with paths.
• tempfile: create temporary files (used to save uploaded PDFs to disk).
• time: measure durations for simple performance timings.

python
CopyEdit
import fitz # PyMuPDF
Imports PyMuPDF as fitz. This opens and reads PDF files and extracts text per page efficiently.

python
CopyEdit
from dotenv import load_dotenv
Loads variables from a .env file into environment variables (e.g., your API key).

python
CopyEdit
from langchain_groq import ChatGroq
LangChain wrapper for Groq’s LLMs. Lets you call Groq models through a standard LangChain interface.

python
CopyEdit
from langchain_community.embeddings import HuggingFaceEmbeddings
Embeddings class that uses a local/CPU (or GPU if configured) Hugging Face sentence-transformer to
convert text → vectors.

python
CopyEdit
from langchain.text_splitter import RecursiveCharacterTextSplitter
Splitter that breaks large text into overlapping chunks while trying to respect paragraph/sentence
boundaries.

python
CopyEdit
from langchain_community.vectorstores import FAISS

machine learning Page 1

from langchain_community.vectorstores import FAISS
Vector store backed by FAISS (Facebook AI Similarity Search) for fast similarity search over embeddings.

python
CopyEdit
from langchain_core.documents import Document
Lightweight container for text (page_content) + metadata. LangChain tools expect Documents.

python
CopyEdit
from langchain_core.prompts import ChatPromptTemplate
For building templated prompts with variables (like {context} and {input}).

python
CopyEdit
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain.chains import create_retrieval_chain
• create_stuff_documents_chain: makes a chain that “stuffs” retrieved documents into your
prompt.
• create_retrieval_chain: wires a retriever (FAISS here) to a document-combining LLM chain.

2) Environment & API key

python
CopyEdit
load_dotenv()
Reads .env in the working directory and places variables into the process environment.

python
CopyEdit
groq_api_key = os.getenv("GROQ_API_KEY")
Fetches your Groq API key from the environment. If it’s missing, groq_api_key will be None (you might
want to guard against that).

3) Initialize the LLM and embeddings

python
CopyEdit
llm = ChatGroq(groq_api_key=groq_api_key, model_name="Llama3-8b-8192")
Creates a LangChain LLM client for Groq using the Llama3-8b-8192 model. This object exposes .invoke()
internally when chains run.

python
CopyEdit
embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/paraphrase-MiniLM-L3-
v2")
Loads a small, fast sentence-transformer to convert text chunks into vector embeddings. Great for quick
local embedding without a remote API. (You can pass device params if you have GPU; otherwise CPU is
fine.)

4) Prompt template

machine learning Page 2

python
CopyEdit
prompt = ChatPromptTemplate.from_template("""
Answer the question based only on the context provided below.
<context>
{context}
</context>
Question: {input}
Answer:
""")
Builds a prompt with two variables:
• {context}: will be filled with retrieved chunks (concatenated by the chain).
• {input}: the user’s question.
The “stuff” chain will map retrieved Document content into {context}; the LLM then answers only from
that context.

5) Streamlit page & inputs

python
CopyEdit
st.set_page_config(page_title=" RAG Q&A", layout="centered")
Sets the browser tab title and page layout.

python
CopyEdit
st.title("New RAG Q&A with Groq + FAISS (Optimized)")
Big heading at the top of the app.

python
CopyEdit
uploaded_files = st.file_uploader(" Upload PDF files", type=["pdf"], accept_multiple_files=True)
Shows a drag-and-drop file uploader that accepts multiple PDFs, returning a list of UploadedFile objects
(or None before selection).

python
CopyEdit
user_query = st.text_input(" Ask a question about the documents")
Single-line input for the user’s question/query string. Empty string until the user types.

6) PDF loader helper

python
CopyEdit
def load_pdf_with_fitz(path):
doc = fitz.open(path)
documents = []
for i, page in enumerate(doc):
text = page.get_text().strip()
if text:
documents.append(Document(page_content=text, metadata={"source": path, "page": i + 1}))
return documents

machine learning Page 3

 return documents
• Opens the PDF file at path.
• Iterates through pages with enumerate to get index i (0-based) and the page object.
• Extracts page text via page.get_text() and trims whitespace.
• If a page has any text, it creates a Document with:
○ page_content: the actual text,
○ metadata: source (file path) and page (1-based page number).
• Returns a list of Documents—one per page that has text.
Notes:
• This extracts plain text; layout (tables, columns) isn’t preserved. For structured PDFs, you may
need different extraction methods.
• Scanned PDFs need OCR first.

7) Chunking setup
python
CopyEdit
chunk_size = 1500 # Larger chunks - fewer embeddings
chunk_overlap = 150
• Each chunk will be ~1,500 characters with 150 characters of overlap to preserve context across
boundaries.
• Larger chunks → fewer calls to the embedder, but each chunk uses more token space when you
“stuff” the prompt.

python
CopyEdit
text_splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
Creates the chunker that will split page-level Documents into chunk-level Documents.

8) Button: build the vector index

python
CopyEdit
if st.button(" Process PDFs and Create Index"):
Renders a button. When pressed, Streamlit reruns the script and this condition is True for that run.

python
CopyEdit
if not uploaded_files:
st.warning(" Upload at least one PDF file.")
Guard: if user didn’t upload anything, show a warning.

python
CopyEdit
else:
docs = []
with st.spinner(" Reading and splitting PDFs..."):
for file in uploaded_files:
with tempfile.NamedTemporaryFile(delete=False, suffix=".pdf") as tmp:
tmp.write(file.read())
tmp_path = tmp.name
docs.extend(load_pdf_with_fitz(tmp_path))

machine learning Page 4

 docs.extend(load_pdf_with_fitz(tmp_path))
• Creates an empty list docs.
• Shows a spinner while processing.
• For each uploaded PDF:
○ Creates a real temporary file on disk (delete=False is important, especially on Windows, so
you can reopen it with other libs).
○ Writes the uploaded bytes to disk.
○ Passes the temp file path to load_pdf_with_fitz, which returns one Document per page with
text.
○ Extends docs with those page Documents.

python
CopyEdit
with st.spinner(" Splitting and embedding chunks..."):
chunks = text_splitter.split_documents(docs)
start = time.time()
• Spinner for the next phase.
• Splits page-level docs into chunk-level docs via your text_splitter.
• Starts a timer to measure embedding+index time.

python
CopyEdit
vectorstore = FAISS.from_documents(chunks, embeddings) # <<<< Best option
• Computes embeddings for each chunk using embeddings.
• Builds a FAISS index in memory keyed by those vectors.
• Returns a LangChain FAISS vector store object that knows how to do similarity search / retrieval.

python
CopyEdit
elapsed = time.time() - start
st.session_state.vectors = vectorstore
st.success(f" Embedding done in {elapsed:.2f} seconds for {len(chunks)} chunks.")
• Stops the timer.
• Saves the vector store into st.session_state under the key "vectors" so it persists across reruns
(until the browser tab resets).
• Success message with timing and number of chunks.

9) Handling the user’s query

python
CopyEdit
if user_query:
Once the user has typed something non-empty, this becomes truthy.

python
CopyEdit
if "vectors" not in st.session_state:
st.warning(" Please process and embed PDFs first.")
Guard: Don’t try to retrieve before the index exists.

python
CopyEdit
else:

machine learning Page 5

 else:
with st.spinner(" Generating answer..."):
retriever = st.session_state.vectors.as_retriever()
• Spinner while we answer.
• Converts the FAISS vector store into a retriever abstraction (it will do k-NN over FAISS behind the
scenes).

python
CopyEdit
doc_chain = create_stuff_documents_chain(llm, prompt)
Creates a chain that will:
1. Take a set of Documents,
2. Concatenate their content into the {context} variable of your prompt,
3. Call the LLM with {input} = user question and {context} = stuffed text.

python
CopyEdit
rag_chain = create_retrieval_chain(retriever, doc_chain)
Wires the retriever to the doc-stuffing chain so you can call a single chain with the user input and it will:
• Retrieve similar chunks,
• Stuff them into the prompt,
• Call the LLM,
• Return both the answer and the retrieved context.

python
CopyEdit
start = time.time()
result = rag_chain.invoke({"input": user_query})
elapsed = time.time() - start
• Times the end-to-end retrieval + generation.
• result is a dict. In recent LangChain versions it typically contains:
○ "answer": the model’s response text,
○ "context": the list of retrieved Documents (depends on version; your later code expects it).

python
CopyEdit
st.subheader(" Answer")
st.write(result["answer"])
st.caption(f" Generated in {elapsed:.2f} seconds")
• Displays the model’s final answer.
• Shows how long generation took.

python
CopyEdit
with st.expander(" Context Chunks"):
for i, doc in enumerate(result["context"]):
st.markdown(f"**Chunk {i+1}:**")
st.write(doc.page_content)
st.markdown("---")
• Expandable panel to show the exact text chunks the model saw.
• Iterates through retrieved Documents, prints their content, and a divider.
• This is essential for transparency/debugging.
Note: The final text explain each and very lineto code in detail you pasted after the last code line
looks like your instruction to me (not Python). Your actual code ends at st.markdown("---").

machine learning Page 6

 looks like your instruction to me (not Python). Your actual code ends at st.markdown("---").

How the whole flow works

1. You upload PDFs and click “Process PDFs and Create Index”
→ Pages → text → chunking → embeddings → FAISS index → stored in st.session_state.
2. You ask a question in the text box
→ Retriever pulls the most similar chunks → those chunks get “stuffed” into the prompt → Groq
LLM answers → You see Answer + Context Chunks.

Gotchas & tips (quick wins)

• API key guard: if groq_api_key is None, show an error early:

python
CopyEdit
if not groq_api_key:
st.stop() # after st.error("Set GROQ_API_KEY in .env")
• Caching: avoid re-embedding on every button click with:

python
CopyEdit
@st.cache_resource
def build_index(chunks):
return FAISS.from_documents(chunks, embeddings)

(Use carefully: cache invalidates when code/inputs change.)

• Retriever settings: control how many chunks come back:

python
CopyEdit
retriever = st.session_state.vectors.as_retriever(search_kwargs={"k": 4})
• Prompt protection: Add “If the answer isn’t in the context, say you don’t know.” to reduce
hallucinations.
• Larger PDFs: consider PdfPlumber/layout-preserving extraction or OCR for scans.
• Memory: FAISS index lives in RAM. For very large corpora, consider on-disk/vector DB backends.
• LangChain versions: The result keys can vary slightly by version. If you ever see a KeyError, print
result to inspect the returned structure.

1. What it is
• A small, fast sentence embedding model from the Sentence Transformers library.
• Based on MiniLM architecture — a compressed Transformer distilled from a bigger model.
• Specifically fine-tuned for paraphrase similarity: sentences with the same meaning → embeddings
close together; different meaning → far apart.

2. Model architecture
• Transformer-based (like BERT, but much smaller).
• 3 encoder layers (that’s the "L3").
• ~22 million parameters → very light, loads quickly.
• Output vector size: 384 dimensions.

machine learning Page 7

 • Output vector size: 384 dimensions.
• Uses mean pooling over token embeddings to get a single vector per sentence.

3. Training details
• Pretrained on a large general text corpus.
• Fine-tuned on paraphrase datasets (like Quora Question Pairs, SNLI, STS Benchmark).
• Loss function: Multiple Negative Ranking Loss (contrastive learning)
→ pushes similar sentences closer and dissimilar ones further in vector space.

4. Performance
• Speed:
○ ~2× faster than L6-v2 on CPU.
○ Very quick even without GPU → perfect for small servers / local apps.
• Accuracy:
○ Around 84% Spearman correlation on STSbenchmark (decent for a small model).
• Memory footprint:
○ ~90 MB disk size, low RAM use.

5. When to use it
✅ When you need fast embeddings for many chunks (like PDF pages in RAG).
✅ When running on CPU or with limited memory.
✅ When you want low-latency retrieval.
❌ Not ideal for highly domain-specific text unless you fine-tune it.
❌ Slightly lower semantic precision than bigger models.

6. How it works in your RAG app

In your code:

python
CopyEdit
embeddings = HuggingFaceEmbeddings(
model_name="sentence-transformers/paraphrase-MiniLM-L3-v2"
)
• Converts each chunk of PDF text into a 384-dimensional vector.
• Stores these in FAISS for fast similarity search.
• At query time:
1. Your question is embedded with the same model.
2. FAISS finds the closest chunk vectors.
3. Those chunks go into the LLM prompt for answering.
Because it’s small and fast:
• Index creation (embedding all chunks) is quick.
• Query embedding is almost instant → low end-to-end latency.

7. Example usage
python
CopyEdit
from sentence_transformers import SentenceTransformer, util
model = SentenceTransformer("sentence-transformers/paraphrase-MiniLM-L3-v2")
emb1 = model.encode("A cat is sleeping on the couch", convert_to_tensor=True)

machine learning Page 8

emb1 = model.encode("A cat is sleeping on the couch", convert_to_tensor=True)
emb2 = model.encode("There is a couch with a cat resting on it", convert_to_tensor=True)
similarity = util.cos_sim(emb1, emb2)
print(similarity.item()) # ~0.9 → high similarity

2. Performance in Retrieval Tasks

From Sentence-Transformers benchmark results on STSbenchmark (semantic textual similarity):
• MiniLM-L3-v2 → ~84% Spearman correlation (how well it ranks similarity vs. human judgment)
• MiniLM-L6-v2 → ~86% Spearman correlation
So L6 is ~2% better in similarity quality — not a huge leap, but measurable for complex semantic
matching.

3. Speed Benchmarks
Tested with 1000 short sentences on CPU (single core):
Model Time Taken
L3-v2 ~0.35 sec
L6-v2 ~0.65 sec

That’s ~45% faster for L3, which matters when embedding hundreds or thousands of chunks in a RAG
app.

4. Why you might choose L3-v2 for your RAG

• PDF Q&A needs to embed many chunks quickly.
• The retrieval step benefits from smaller models when you re-index multiple times in one session.
• The tiny loss in semantic accuracy doesn’t impact most practical queries unless your chunks are
extremely similar in meaning.
• Runs easily on CPU-only deployment without slowing down.

5. When you might upgrade to L6-v2

machine learning Page 9
5. When you might upgrade to L6-v2
• Your documents are very semantically similar, and small differences matter (e.g., legal contracts,
medical texts).
• You can afford the extra embedding time & memory usage.
• You’re optimizing for highest retrieval accuracy, not fastest indexing.

1. What is FAISS?
• FAISS stands for Facebook AI Similarity Search.
• It’s a vector database / library for storing and searching through dense vector embeddings
quickly.
• Created by Facebook AI Research.
• Optimized for high-dimensional vector search (e.g., 384-dim, 768-dim) on very large datasets.
• Written in C++ with Python bindings → very fast.

2. Why we need FAISS in RAG

In RAG, you:
1. Convert text chunks into vectors (embeddings).
2. Store them somewhere.
3. At query time, embed the question → find the closest chunk vectors.
4. Feed those chunks to the LLM.
If you just stored embeddings in a Python list and searched with brute-force cosine similarity, it’d be
slow for big datasets (O(n) search).
FAISS uses optimized indexing structures to make search very fast — even for millions of vectors.

3. How FAISS stores and searches data

a) Storage (Index)
FAISS stores vectors in indexes — data structures that allow fast nearest neighbor search.
Common index types:
• IndexFlatL2 → simple, exact Euclidean search (fast for small datasets).
• IndexIVFFlat → inverted file lists for large datasets (uses clustering to narrow the search).
• HNSW → graph-based approximate search (very fast, scalable).
• PQ (Product Quantization) → compresses vectors to save memory.
In LangChain:

python
CopyEdit
vectorstore = FAISS.from_documents(chunks, embeddings)
• This calls the embedder to get vectors from chunks.
• Stores them in an IndexFlatL2 by default (exact search).
• Keeps metadata (like source file and page) alongside each vector.

b) Search
When you search:
1. Your query is embedded into a vector.
2. FAISS compares it to all vectors (or a reduced set for approximate search).
3. Returns the k most similar vectors along with their IDs.
4. LangChain maps these IDs back to your original text chunks.
Example:

machine learning Page 10

python
CopyEdit
results = vectorstore.similarity_search("solar energy benefits", k=3)
• FAISS finds top 3 most similar embeddings.
• Returns the corresponding text chunks + metadata.

4. How FAISS measures similarity

FAISS can use:
• L2 distance (Euclidean) → good when embeddings are normalized.
• Inner Product (dot product) → often used for semantic embeddings.
• Cosine similarity → computed from inner product if vectors are normalized.
In your case, since MiniLM embeddings are normalized, cosine similarity is effectively the same as dot
product.

5. Advantages of FAISS
✅ Speed: Can search millions of vectors in milliseconds.
✅ Scalable: Handles huge datasets efficiently.
✅ Versatile: Supports CPU & GPU acceleration.
✅ Integrates easily with LangChain, Haystack, etc.
✅ Keeps metadata: So you know which document/page a vector came from.

6. FAISS in your RAG pipeline

Flow:
1. PDF chunks → embeddings (MiniLM-L3-v2).
2. Embeddings stored in FAISS index with metadata.
3. Query → embedding → FAISS search.
4. Top-k similar chunks → passed into LLM prompt.
Diagram:

mathematica
CopyEdit
PDF text → Chunking → Embeddings → FAISS Index
Query → Embedding → FAISS Search → Top-k Chunks → LLM

7. Small Example
python
CopyEdit
from sentence_transformers import SentenceTransformer
import faiss
import numpy as np
# Step 1: Embedding model
model = SentenceTransformer("sentence-transformers/paraphrase-MiniLM-L3-v2")
# Step 2: Documents and embeddings
docs = ["Cats are cute", "Dogs are loyal", "Birds can fly"]
embs = model.encode(docs, normalize_embeddings=True)
# Step 3: Create FAISS index (cosine similarity via inner product)
dim = embs.shape[1]
index = faiss.IndexFlatIP(dim)

machine learning Page 11

index = faiss.IndexFlatIP(dim)
index.add(embs)
# Step 4: Search
query_vec = model.encode(["I love kittens"], normalize_embeddings=True)
distances, indices = index.search(query_vec, k=2)
print(indices, distances)

How it works step-by-step

Let’s say your chunk size = 1500, overlap = 150.
Step 1: Define separator hierarchy
By default:

python
CopyEdit
separators = ["\n\n", "\n", " ", ""]
This means:
1. Try to split by double newlines (paragraphs).
2. If still too big → split by single newline (lines).
3. If still too big → split by space (words).
4. If still too big → split by characters.

ChatGPT said:
In your RAG setup, Groq is used as the LLM backend — essentially the “brain” that answers questions
after retrieval.
Let’s break it down clearly:

1. What Groq is
• Groq is a cloud-based LLM inference provider.
• Known for extremely low latency (answers in milliseconds) because they run models on
specialized GroqChip™ hardware, not just GPUs.
• Supports open-source LLMs like LLaMA 3, Gemma, Mixtral, etc., but runs them much faster than
most APIs.

2. Why it’s used in your RAG code

In your code:

python
CopyEdit
from langchain_groq import ChatGroq
llm = ChatGroq(model="llama3-8b-8192", temperature=0)
Groq is chosen because:
Speed
• RAG apps already have multiple steps:
1. Read PDFs
2. Split into chunks
3. Embed chunks

machine learning Page 12

 3. Embed chunks
4. Search FAISS
5. Send query + retrieved chunks to LLM
• If step 5 (LLM call) is slow, the whole app feels sluggish.
• Groq can return responses in ~300–500ms, while something like OpenAI GPT-4 may take 2–5
seconds.
Cost efficiency
• Groq’s pricing for large context models is often cheaper than equivalent API calls elsewhere.
• Useful for apps that do many queries.
Large context
• Models like "llama3-8b-8192" can handle 8k tokens (or more with certain configs), meaning you
can feed more retrieved chunks without hitting limits.
Open-source model flexibility
• You’re not locked into a single vendor’s proprietary model — you can choose different OSS models
Groq supports without changing much code.

3. How Groq fits in the RAG pipeline

Here’s your flow:

sql
CopyEdit
User question → FAISS retrieves top chunks →
Chunks + question → Groq LLM →
Final answer

Groq here acts as:

• The reasoning layer → It reads the retrieved chunks (context) and generates an answer grounded
in them.
• The formatter → Produces human-readable answers, summaries, or step-by-step reasoning.

4. Why not just use embeddings for answering?

• Embeddings (like MiniLM) can find relevant text but can’t generate an answer.
• Groq takes that retrieved context and produces a coherent, context-aware response.

machine learning Page 13