POST GRADUATE PROGRAM

IN
DATA SCIENCE AND ANALYTICS

GUIDED PROJECT:
NATURAL LANGUAGE PROCESSING WITH
GENERATIVE AI

SUBJECT:
MEDICAL ASSISTANT
 CONTENT
1. CONTEXT

2. OBJECTIVE

3. DATA DICTIONARY

4. EXECUTIVE SUMMARY

5. BUSINESS PROBLEM AND

SOLUTION APPROACH

6. QUESTION ANSWERING USING LLM

7. QUESTION ANSWERING USING LLM WITH

PROMPT ENGINEERING

8. DATA PREPARATION FOR RAG

9. QUESTION ANSWERING USING RAG

10. OUTPUT EVALUATION

11. CONCLUSION

12. ACTIONABLE INSIGHTS

13. RECOMMENDATIONS
CONTEXT
The healthcare industry is rapidly evolving, and professionals face increasing
challenges in managing vast volumes of medical data while delivering accurate
and timely diagnoses. Quick access to comprehensive, reliable, and up-to-date
medical knowledge is critical for improving patient outcomes and ensuring
informed decision-making in a fast-paced environment.
Healthcare professionals often encounter information overload, struggling to sift
through extensive research and data to create accurate diagnoses and treatment
plans. This challenge is amplified by the need for efficiency, particularly in
emergencies, where time-sensitive decisions are vital. Furthermore, access to
trusted, current medical information from renowned manuals and research
papers is essential for maintaining high standards of care.
To address these challenges, healthcare centers can focus on integrating systems
that streamline access to medical knowledge, provide tools to support quick
decision-making, and enhance efficiency. Leveraging centralized knowledge
platforms and ensuring healthcare providers have continuous access to reliable
resources can significantly improve patient care and operational effectiveness.

OBJECTIVE
The objective of my project is to develop a Retrieval-Augmented Generation
(RAG)-based AI system that uses trusted medical manuals to help healthcare
professionals overcome the challenge of information overload. This system
aims to provide quick, accurate, and relevant medical information, thereby
improving decision-making, enhancing diagnostic accuracy, and supporting the
standardization of care practices. Through this approach, the project seeks to
streamline clinical workflows and contribute to better patient outcomes.

DATA DICTIONARY
The Merck Manuals are medical references published by the American
pharmaceutical company Merck & Co. that cover a wide range of medical
topics, including disorders, tests, diagnoses, and drugs. The manuals have been
published since 1899, when Merck & Co. was still a subsidiary of the German
company Merck.
The manual is a PDF with over 4,000 pages divided into 23 sections.

EXECUTIVE SUMMARY
This project focuses on developing a Retrieval-Augmented Generation (RAG)-
based AI solution leveraging renowned medical references such as The Merck
Manuals to address critical challenges in healthcare decision-making. The
healthcare industry is facing unprecedented information overload, making it
difficult for professionals to quickly locate accurate and relevant data needed
for diagnoses and treatment planning.
By combining advanced AI retrieval techniques with generative capabilities, the
proposed system aims to provide healthcare providers with rapid, reliable, and
context-specific answers to medical queries. This will enhance diagnostic
accuracy, streamline decision-making processes, and promote the
standardization of care practices across diverse medical settings.
The Merck Manuals, a trusted medical reference with over 4,000 pages of
authoritative content, will serve as the primary knowledge source. The AI
system will be trained to extract relevant information, synthesize it into concise,
actionable insights, and present it in an accessible interface for healthcare
professionals.
This approach is expected to reduce cognitive load, improve patient outcomes,
and increase operational efficiency in both routine and emergency care
situations.
KEY INSIGHTS
• Information Overload is a Barrier: Clinicians spend valuable time
searching for relevant data in a sea of medical literature, which delays
decision-making and can impact patient outcomes.
• Trusted Sources are Crucial: Using The Merck Manuals ensures medical
information is reliable, evidence-based, and up-to-date.
• AI Can Bridge the Gap: RAG technology can connect vast medical
databases with real-time, context-aware responses, making medical
knowledge instantly usable.
• Standardization Improves Care Quality: Providing uniform, guideline-
based answers helps reduce variation in treatment practices.
• Emergency Scenarios Benefit Most: Quick retrieval of critical protocols
(e.g., for sepsis or pulmonary embolism) can directly impact survival
rates.
RECOMMENDATIONS:
• Integrate RAG into Clinical Workflows: Ensure seamless access through
hospital EMR (Electronic Medical Record) systems and mobile devices
for real-time usage.
• Prioritize High-Impact Use Cases: Start with critical care protocols,
common disease diagnostics, and drug information retrieval.
• Implement Continuous Knowledge Updates: Regularly refresh the AI
system with the latest guidelines and research to maintain accuracy.
• Focus on User-Centric Design: Create an intuitive, fast, and searchable
interface tailored to the needs of healthcare professionals.
• Measure Impact with Pilot Programs: Test the system in controlled
hospital environments to assess improvements in diagnostic accuracy,
decision speed, and patient outcomes.
BUSINESS PROBLEM AND SOLUTION APPROACH
BUSINESS PROBLEM
The healthcare industry is experiencing rapid growth in medical knowledge,
guidelines, and research publications, resulting in information overload for
healthcare professionals. Clinicians often struggle to efficiently locate and
interpret relevant, up-to-date information from trusted sources while making
time-sensitive decisions. This challenge is especially critical in emergency care,
where delays or inaccuracies can have life-threatening consequences.
Additionally, the lack of standardized access to authoritative medical information
leads to variations in diagnostic approaches and treatment plans, impacting the
quality and consistency of patient care. Without effective tools to filter, retrieve,
and synthesize medical data, healthcare providers face reduced efficiency,
increased cognitive burden, and potential risks to patient outcomes.

SOLUTION APPROACH
The solution approach involves developing a Retrieval-Augmented Generation
(RAG)-based AI system that uses The Merck Manuals as a trusted knowledge
base to provide quick, accurate, and context-specific medical information. The
system will retrieve relevant content using semantic search, then generate
concise, actionable responses for healthcare professionals. Designed with an
intuitive interface, it will support queries related to diagnostics, treatment plans,
drug information, and critical care protocols, aiming to reduce information
overload, standardize care practices, and improve decision-making efficiency.

QUESTION ANSWERING USING LLM

QUERY 1:
What is the protocol for managing sepsis in a critical care unit?
OBSERVATIONS
The response generated by the LLM is detailed and medically appropriate
regarding the management of sepsis in a critical care unit.

QUERY 2:
What are the common symptoms for appendicitis, and can it be cured via
medicine? If not, what surgical procedure should be followed to treat it?
OBSERVATIONS
The answer, similar to the first query, is medically accurate,
demonstrating that the model is performing effectively.
QUERY 3:
What are the effective treatments or solutions for addressing
sudden patchy hair loss, commonly seen as localized bald spots
on the scalp, and what could be the possible causes behind it?
OBSERVATIONS
The answer is medically appropriate and lists effective
treatemetns for sudden patchy hair loss.

QUERY 4:
What treatments are recommended for a person who has
sustained a physical injury to brain tissue, resulting in
temporary or permanent impairment of brain function?
OBSERVATIONS
The answer provided is medically appropriate and offers a
comprehensive list of common treatments for brain injury,
which aligns well with the query.

QUERY 5:
What are the necessary precautions and treatment steps for a
person who has fractured their leg during a hiking trip, and
what should be considered for their care and recovery?
OBSERVATIONS
This answer is comprehensive and medically appropriate, outlining the necessary
precautions and treatment steps for a person who has fractured their leg while
hiking.

STRENGTHS AND LIMITATIONS OF MISTRAL 7B MODEL IN THIS

SCENARIO

STRENGTHS
 Comprehensive Responses: The Mistral 7B model provides detailed and
medically appropriate answers, covering a broad spectrum of information.
 Structured Answer Format: The model outputs answers in a clear, structured
format, using numbered lists to break down the response into digestible steps.
 Contextual Relevance: The model generates contextually relevant and precise
information based on the query, addressing each aspect of the question.

LIMITATIONS
 Incomplete Responses: All of the answers provided by the this model are cut
off mid-sentence (e.g., the answer about treatments for traumatic brain injury
 or fractures), suggesting that the model might be truncating its output due to
the token limits. This results in incomplete answers, which could be
problematic for users seeking thorough information.
 Generic Responses: While the answers are medically appropriate in
structure, some responses are quite general (e.g., treatment steps or protocols
for common injuries). This could imply that the model might need further
fine-tuning or more specific context to generate highly detailed, situation-
specific answers.
 Need for Contextual Depth: The model tends to provide useful but
somewhat high-level responses. This could make the answers seem more
like generic guidelines rather than tailored advice for specific scenarios.

Therefore, we will leverage prompt engineering to enhance the quality of our

responses.

QUESTION ANSWERING USING LLM WITH

PROMPT ENGINEERING

QUERY 1:
What is the protocol for managing sepsis in a critical care unit?
OBSERVATIONS
Basic parameters
Modified parameters - Temperature = 0.0
 Answer 1, generated with a temperature of 0.7, is more comprehensive and
nuanced, offering a clear step-by-step clinical protocol with specific metrics
(e.g., MAP ≥ 65 mmHg, urine output ≥ 0.5 ml/kg/h). This reflects a more
dynamic and human-like response, likely enhanced by the higher
temperature which allows the model to explore a broader range of relevant
tokens, contributing to richer content.
 Answer 2, with temperature 0.0, is more conservative and deterministic,
sticking closely to the most probable next token at each step. While it
remains medically accurate, it is less informative and cuts off mid-sentence,
indicating a lack of flexibility in adapting to the context or expanding on
critical aspects.
 Overall, Answer 1 is superior in terms of completeness, clinical clarity, and
usefulness for decision-making in a critical care setting.

QUERY 2:
What are the common symptoms for appendicitis, and can it be cured via
medicine? If not, what surgical procedure should be followed to treat it?
OBSERVATIONS
Basic parameters
Modified Parameters - Top_p = 0.85
 Answer 1, generated with a top_p of 0.95, allows the model to consider a
wider range of possible next tokens, leading to slightly more diverse and
expansive content. The answer is informative but a bit general in structure.
 Answer 2, generated with a top_p = 0.85, that is a narrower top_p, the
model restricts its output to more probable and focused responses.
 Overall, Answer 2 is better because the lower top_p value helps the model
generate more focused and relevant content, minimizing unnecessary
variation.

QUERY 3:
What are the effective treatments or solutions for addressing sudden patchy hair
loss, commonly seen as localized bald spots on the scalp, and what could be the
possible causes behind it?
OBSERVATIONS
Basic parameters
Modified Parameters - Top_k = 80
 Answer 1, generated with a top_k of 50, provides a broader overview by
listing multiple causes of patchy hair loss and briefly mentioning treatment
options for each, which makes it informative but slightly general.
 Answer 2, with top_k of 80, dives deeper into specific treatments for
different types of hair loss, especially Alopecia Areata and Traction
Alopecia, offering more clinical detail and therapeutic options.
 Overall, in the context of this project, Answer 2 is better because it shows
more medical depth and actionable insights.

QUERY 4:
What treatments are recommended for a person who has sustained a physical
injury to brain tissue, resulting in temporary or permanent impairment of brain
function?
OBSERVATIONS
Basic parameters
Modified Parameters - Max_tokens = 512
 Answer 1, generated with a Max_tokens of 1024, provides a
comprehensive overview of brain injury treatment, listing specific
interventions such as medications, surgery, rehabilitation, and supportive
care, covering a wide range of recovery aspects, including physical,
cognitive, and emotional support.
 Answer 2, with Max_tokens of 512, is slightly more concise, focusing on
key areas like initial care, rehabilitation, medications, and supportive care.
It emphasizes rehabilitation methods and includes mention of surgery when
necessary for specific issues such as hematomas or clots.
  Overall, Answer 1 is better as it offers a more detailed and well-rounded
view, covering additional aspects. With more tokens, the model can cover
additional aspects of the topic, giving a fuller answer.

QUERY 5:
What are the necessary precautions and treatment steps for a person who has
fractured their leg during a hiking trip, and what should be considered for their
care and recovery?
OBSERVATIONS
Basic parameters
Modified Parameters -
emp = 0.7
Top_p = 0.85
Top_k = 80
Max_tokens = 1024
 Answer 1, generated with default parameters, provides a more detailed
approach to the immediate steps for handling a fractured leg during a hiking
trip, emphasizing both the severity assessment and infection prevention,
while focusing on the necessity of seeking medical attention in case of
complications.
 Answer 2, with modified parameters, is more structured and offers a
comprehensive recovery plan, addressing not only the immediate care but
also long-term considerations like nutrition, rest, and physical therapy. It
also focuses on the steps that a layperson can take, such as cleaning the
wound and elevating the leg for pain management.
 Overall, Answer 2 is more holistic and includes follow-up care, making it
slightly better for a complete response that covers both the immediate
treatment and recovery process.

DATA PREPARATION FOR RAG

LOADING THE DATA

DATA OVERVIEW

DATA CHUNKING

EMBEDDING
VECTOR DATABASE

From the retrieved chunks, we observe that all the chunks are related to the key
terms Alopecia Areata.

RETRIEVER

RESPONSE FUNCTION

The retriever is configured using the Maximal Marginal Relevance (MMR)

search method with cosine similarity to get accurate and diverse medical
information. It returns the top 8 most relevant chunks (k=8) from a pool of 60
candidates (fetch_k=60), balancing relevance and diversity with
lambda_mult=0.5. This setup ensures precise, non-repetitive, and trustworthy
answers from the Merck Manual.

QUESTION ANSWERING USING RAG

QUERY 1:
What is the protocol for managing sepsis in a critical care unit?
OBSERVATIONS
Basic Parameters
Fine Tuning - Removing Chunk Overlap
Chunk size = 512, overlap = 0
 Removing chunk overlap did not change the answers in this case,
probably because the relavant information for the question was already
contained within a single chunk.
 Removing chunk overlap might affect the answer if the question required
information that spanned across multiple chunks from different parts of
the text.

QUERY 2:
What are the common symptoms for appendicitis, and can it be cured via
medicine? If not, what surgical procedure should be followed to treat it?
OBSERVATIONS
Basic Parameters
Fine Tuning - Retreiver Parameters
k=2
 Answer 1, with k=3, is slightly more detailed and includes examples of
cases (like when surgery is impossible).
 Answer 2, with k=2, is slightly shorter and cuts off earlier, suggesting
that retrieving more chunks (k=3) provided additional helpful context for
a more complete answer.

QUERY 3:
What are the effective treatments or solutions for addressing sudden patchy hair
loss, commonly seen as localized bald spots on the scalp, and what could be the
possible causes behind it?
OBSERVATIONS
Basic Parameters
Fine Tuning - LLM Parameters
Temperature = 0.7
 Answer 1, with temperature = 0, is more precise and structured.
  Answer 2, with temperature = 0.7, introduces a bit more variability,
with slight changes in wording, reflecting the increased randomness
from the higher temperature.

QUERY 4:
What treatments are recommended for a person who has sustained a physical
injury to brain tissue, resulting in temporary or permanent impairment of brain
function?
OBSERVATIONS
Basic Parameters
Fine Tuning - LLM Parameters
Top_p = 0.8
 Answer 1 with Top_p = 0.95 means the model will consider the top 95%
probability mass of the next token and select from that pool. This makes
the answer generated more focused, coherent responses because it
prioritizes more likely words and phrases.
 Answer 2 with Top_p = 0.8 means the model restricts the token selection
to only the top 80% of the probability distribution, including less likely
but more diverse options.
 Thus, in Answer 1, the model focuses more on the main facts (like
supportive care and rehab) without adding unnecessary or unrelated
details. In Answer 2, the model explores a wider range of possibilities,
leading to additional information about severe cases, surgery, and
treatment phases.

QUERY 5:
What are the necessary precautions and treatment steps for a person who has
fractured their leg during a hiking trip, and what should be considered for their
care and recovery?
OBSERVATIONS
Basic Parameters
Fine Tuning - LLM Parameters
Top_k = 25
• In this case, the generated answers are the same, probably because the
topic is very structured, and the model chooses almost the same tokens
even when it has a broader range of options (top_k=50).

OUTPUT EVALUATION

QUERY 1:
What is the protocol for managing sepsis in a critical care unit?
OBSERVATIONS
The rating of 5 for groudness and relavance indicates that the answer is fully
grounded in the context and is also highly relevant to the query.

QUERY 2:
What are the common symptoms for appendicitis, and can it be cured via
medicine? If not, what surgical procedure should be followed to treat it?
OBSERVATIONS
The rating of 5 for groudness and relavance indicates that the answer is fully
grounded in the context and is also highly relevant to the query.

QUERY 3:
What are the effective treatments or solutions for addressing sudden patchy hair
loss, commonly seen as localized bald spots on the scalp, and what could be the
possible causes behind it?
OBSERVATIONS
The rating of 5 for groudness and relavance indicates that the answer is fully
grounded in the context and is also highly relevant to the query.

QUERY 4:
What treatments are recommended for a person who has sustained a physical
injury to brain tissue, resulting in temporary or permanent impairment of brain
function?
OBSERVATIONS
The rating of 5 for groudness and relavance indicates that the answer is fully
grounded in the context and is also highly relevant to the query.

QUERY 5:
What are the necessary precautions and treatment steps for a person who has
fractured their leg during a hiking trip, and what should be considered for their
care and recovery?
OBSERVATIONS
The rating of 5 for groudness and relavance indicates that the answer is fully
grounded in the context and is also highly relevant to the query.

CONCLUSION
The development of a RAG-based AI solution using authoritative medical
references like The Merck Manuals demonstrates significant potential to
transform healthcare decision-making. By delivering accurate, context-specific,
and trustworthy medical information in real time, the system addresses the
critical challenge of information overload faced by healthcare professionals. Its
ability to enhance diagnostic accuracy, improve patient outcomes, and support
standardized care practices makes it a valuable tool in both routine and
emergency scenarios. Furthermore, the potential for cost savings, improved
efficiency, and adaptability across multiple medical specialties positions this
solution as a scalable and sustainable innovation in modern healthcare.
Continuous fine-tuning, ethical compliance, and specialization will ensure its
long-term relevance and effectiveness in a rapidly evolving medical landscape.

ACTIONABLE INSIGHTS
1. ACCURACY AND TRUSTWORTHINESS
The RAG-based AI system delivers context-specific, reliable, and evidence-
backed medical information sourced from authoritative references such as The
Merck Manuals. This accuracy builds trust among healthcare professionals and
patients, leading to higher satisfaction and improved adoption in clinical
settings.
2. ENHANCED USER EXPERIENCE
By combining advanced retrieval techniques with natural language generation,
the system provides personalized, relevant, and precise responses to medical
queries. This significantly improves the user experience by reducing search time
and presenting complex medical information in a clear, concise manner.
3. COST REDUCTION IN MEDICAL CONSULTATIONS
The AI model can automate routine medical question-and-answer tasks,
reducing the dependency on direct consultations for basic information needs.
This has the potential to lower operational costs in telemedicine services and
healthcare support centers, while allowing healthcare professionals to focus on
critical cases.

RECOMMENDATIONS
1. CONTINUOUS FINE-TUNING
Regularly update and fine-tune the AI system to incorporate the latest medical
guidelines, research findings, and treatment protocols to maintain accuracy and
relevance over time.
2. ETHICAL AND LEGAL COMPLIANCE
Prioritize data privacy and ensure compliance with healthcare regulations such
as HIPAA and GDPR. Implement robust encryption and access control
measures to protect sensitive patient information.
3. DEVELOPMENT OF SPECIALIZED MODELS
Create dedicated, domain-specific AI models for various medical specialties
such as orthopedics, gastroenterology, oncology, and cardiology. This
specialization can further improve accuracy, relevance, and clinical decision-
making support.