OPTIMIZING DIABETIC RETINOPATHY

DETECTION USING DEEP LEARNING

PROJECT PHASE- 2 REPORT

Submitted by

AKASH VISHNU R (711521BAD003)

HARI PRASANNA M (711521BAD018)
RAVEE RAGHUL C (711521BAD044)

in partial fulfilment for the award of the degree

of

BACHELOR OF TECHNOLOGY
IN

ARTIFICIAL INTELLIGENCE AND DATA SCIENCE

KIT - KALAIGNARKARUNANIDHI INSTITUTE OF TECHNOLOGY

(AN AUTONOMOUS INSTITUTION)
Coimbatore - 641402
Affiliated to Anna University, Chennai
Accredited by NAAC with ‘A’ Grade | Accredited by NBA (AERO, CSE, ECE, EEE,
MECH and MBA)

MAR/APR – 2025
 OPTIMIZING DIABETIC RETINOPATHY
DETECTION USING DEEP LEARNING

PROJECT PHASE- 2 REPORT

Submitted by

AKASH VISHNU R (711521BAD003)

HARI PRASANNA M (711521BAD018)
RAVEE RAGHUL C (711521BAD044)

In partial fulfillment for the award of the degree of

BACHELOR OF TECHNOLOGY
IN

ARTIFICIAL INTELLIGENCE AND DATA SCIENCE

KIT - KALAIGNARKARUNANIDHI INSTITUTE OF TECHNOLOGY

(AN AUTONOMOUS INSTITUTION)
Coimbatore - 641402
Affiliated to Anna University, Chennai
Accredited by NAAC with ‘A’ Grade | Accredited by NBA (AERO, CSE, ECE,
EEE, MECH and MBA)

MAR/APR - 2025

I
 KIT - KALAIGNARKARUNANIDHI INSTITUTE
OF TECHNOLOGY
(AN AUTONOMOUS INSTITUTION)
Affiliated to Anna University, Chennai
Accredited by NAAC with ‘A’ Grade | Accredited by NBA(AERO,CSE,
ECE, EEE, MECH and MBA)

BONAFIDE CERTIFICATE
Certified that this project report “OPTIMIZING DIABETIC RETINOPATHY
DETECTION USING DEEP LEARNING” is the bonafide work of “AKASH
VISHNU R (711521BAD003), HARI PRASANNA M (711521BAD018),
RAVEE RAGHUL C (711521BAD044)” who carried out the project work
under my supervision.

SIGNATURE SIGNATURE
Ms.R.KAVITHA, Dr.C.DEEPA M.E., Ph.D.,
SUPERVISOR HEAD OF THE DEPARTMENT
Assistant Professor / AI&DS Associate Professor & Head
Department of Artificial Intelligence Department of Artificial Intelligence
and Data Science, and Data Science,
KIT-Kalaignarkarunanidhi Institute of KIT-Kalaignarkarunanidhi Institute of
Technology, Technology,
Coimbatore- 641 402. Coimbatore- 641 402.

Submitted for the University Project Viva - Voce held on

II
INTERNAL EXAMINER EXTERNAL EXAMINER

II
 ACKNOWLEDGEMENT
Owing deeply to the supreme, we extend our sincere thanks to God almighty who
has made all things possible.
We extend our heart full gratitude toward our revered Founder Chairman Thiru
Pongalur N. Palanisamy, Vice chairperson Mrs. P. Indhu Murugesan and
Executive Trustee Mr. A. Suriya all other trust members for having provided us
with necessary infrastructure to undertake this project.
We are grateful to our Chief Executive Officer Dr. N. Mohan Das Gandhi and
our beloved Principal Dr. M. Ramesh and academic Dean Dr. K. Ramasamy for
the facilities provided to complete this project work.
We wish to extend our gratitude to Dr. C. Deepa, Associate Professor and Head
of the Department of Artificial Intelligence and Data Science and our project
guide Ms. R. Kavitha and project coordinator Dr. R.P. Narmadha for their
constant support and encouragement.
We thank our beloved parents for their constant support and blessings. Finally, we
would like to thank the teaching and non-teaching staff members of the
Department of Artificial Intelligence and Data Science for their suggestions and
guidance.

III
 ABSTRACT

This project represents a cutting-edge approach to diagnosing diabetic retinopathy, a

serious complication of diabetes that can lead to blindness if not detected and treated

early. Leveraging the power of deep learning, the model utilizes multi-modal retinal

image fusion to enhance diagnostic accuracy. By combining multiple retinal images,

the system captures a more comprehensive view of the retinal structure, allowing for

the detection of subtle abnormalities that might be missed by traditional methods.

The core of this model is a Transformer-based architecture, a state-of-the-art

technology that excels in both local feature extraction and global pattern recognition.

This combination ensures that the model not only identifies fine-grained details,

such as microaneurysms or hemorrhages, but also understands the broader patterns

associated with different stages of DR. In addition to its diagnostic accuracy, the

model is designed for real-time use on edge devices such as smartphones and PCs,

making it accessible for a wide range of healthcare environments. By streamlining

the screening process and improving early detection rates, this project offers a

significant contribution to diabetic retinopathy management, potentially reducing the

global burden of diabetes-related vision loss.

Keywords: Diabetic Retinopathy (DR), Grad-CAM, Multi-modal Image Fusion,

Edge Computing, Retinal Image Analysis, Real-time Predictions.

4
 TABLE OF CONTENTS

CHAPTER TITLE PAGE

NO. NO.
ABSTRACT IV
LIST OF FIGURES
LIST OF ABBREVIATION VIII
1 INTRODUCTION 9
1.1 INTRODUCTION 9
1.1.1 RETINAL FUNDAS IMAGES 9
1.1.2 ARTIFICAL INTELLIGENCE 10
1.1.3 MACHINE LEARNING 10
1.1.4 DEEP LEARNING 11
1.1.5 CONVOLUTIONAL NEURAL NETWORK 11
1.1.6 GRAD-CAM 12
1.2 PROBLEM STATEMNT 13
1.3 OBJECTIVE OF THE PROJECT 14
1.4 ORGANIZATION OF THE PROJECT 14
2 LITERATURE SURVEY 15
2.1 INTRODUCTION 15
2.2 LITERATURE REVIEW 16
3 SYSTEM REQUIREMENT SPECIFICATION 18
3.1. INTRODUCTION 18
3.2 SOFTWARE REQUIREMENTS 18
3.2.1 JUPYTER NOTEBOOK
3.2.2 FEATURES OF JUPYTER NOTBOOK
3.3 PYTHON
3.3.1 FEATURES OF PYTHON
5
 4 PROJECT DESCRIPTION 28
4.1 EXISTING SYSTEM 28
4.2 PROPOSED SYSTEM 30
5 FUNCTIONAL DESIGNS 31
5.1 FUNCTIONAL DIAGRAM 31
5.2 FUNCTIONAL MODULES

6 IMPLEMENTATION AND RESULT 39

6.1 IMPLEMETATION 39
6.2 RESULT ANALYSIS 40

7 CONCLUSION AND FUTURE WORK 42

7.1 CONCLUSION
7.2 FUTURE ENHANCEMENT

APPENDIX
SOURCE CODE
REFERENCES

6
 LIST OF FIGURES

Fig.No Description Page.No

1.1 Retinal Fundus Image 1

3.1 Jupyter Notebook 19

3.2 Python 22

5.1 Functional Diagram for Detection of Diabetic 31

Retinopathy

5.2 Block Diagram for Detection of Diabetic 35

Retinopathy

7
 LIST OF ABBREVIATIONS

ACC Accuracy

AI Artificial Intelligence

ANN Artificial Neural Network

CNNs Convolutional Neural Networks

DNN Deep Neural Network

DR Diabetic Retinopathy

FCNN Fully Convolutional Neural Network

F-CNN Faster Convolutional Neural Network

KNN K Nearest Neighbour

LSTM Long Short Term Memory

MAE Mean Absolute Error

ML Machine Learning

MSE Mean Squared Error

NPDR Non-Proliferative Diabetic Retinopathy

NPV Negative Predictive Value

P Precision

7
R-CNN Region-based Convolutional Neural Network

RNNs Recurrent Neural Networks

SE Sensitivity

SP Specificity

YOLO You Only Look Once

8
9
 CHAPTER 1

INTRODUCTION

1.1 INTRODUCTION

This project introduces an innovative deep learning model for the diagnosis
of diabetic retinopathy, a vision-threatening complication of diabetes. By
leveraging the strengths of EfficientNetB0 and ResNet50 architectures, the model
enhances diagnostic accuracy through the integration of both fine-grained local
features and high-level global representations from retinal fundus images.
EfficientNetB0 contributes to the system’s efficiency and scalability, while
ResNet50 offers deep feature extraction capabilities for complex lesion
identification. The model classifies diabetic retinopathy into five distinct stages,
from No DR to Proliferative DR. To ensure transparency, interpretable AI
technique Grad-CAM are employed, allowing healthcare providers to visualize the
decision-making process and build trust in the automated diagnostic system.

1.1.1 RETINAL FUNDUS IMAGES

Retinal fundus images are high-resolution photographs of the interior surface
of the eye, particularly the retina, optic disc, macula, and blood vessels. These
images are captured using a specialized camera called a fundus camera, which
provides a clear view of the retinal structures. Fundus imaging is a vital tool in
ophthalmology for diagnosing various eye conditions, including diabetic
retinopathy, glaucoma, and macular degeneration.

9
Fig 1.1 Retinal Fundus image

10
 1.1.2 ARTIFICIAL INTELLIGENCE

Artificial Intelligence (AI) is a branch of computer science focused on

creating systems capable of performing tasks that typically require human
intelligence. These tasks include learning, reasoning, problem-solving, perception,
and natural language understanding. AI systems can be divided into narrow AI,
which is designed to perform specific tasks like facial recognition or language
translation, and general AI, which hypothetically would possess the ability to
understand and reason across a wide range of tasks, similar to human cognition. The
development of AI is driven by techniques such as machine learning, neural
networks, and deep learning, enabling computers to recognize patterns, make
decisions, and improve over time without explicit programming. AI is
revolutionizing industries like healthcare, finance, and transportation by automating
complex processes, improving efficiency, and driving innovations such as
autonomous vehicles and personalized medicine.

1.1.3 MACHINE LEARNING

Machine Learning (ML) is a subset of artificial intelligence that enables

computers to learn from data and improve their performance without being
explicitly programmed. By utilizing algorithms and statistical models, ML systems
can automatically detect patterns, make decisions, and predict outcomes based on
historical data. The learning process typically involves training a model on a dataset,
allowing it to recognize relationships and refine its predictions over time. There are
several types of machine learning, including supervised learning (where the model is
trained with labeled data), unsupervised learning (where the model identifies
patterns without labeled outputs), and reinforcement learning (where the model
learns through trial and error with feedback from its actions). Machine learning is
widely used in applications such as recommendation systems, image and speech
recognition, fraud detection, and autonomous systems, driving advancements across
11
industries.

12
 1.1.4 DEEP LEARNING

Deep Learning is a specialized branch of machine learning that focuses on

neural networks with many layers, often referred to as deep neural networks. These
models are designed to automatically learn and extract complex features from vast
amounts of data, enabling them to excel in tasks such as image recognition, natural
language processing, and speech synthesis. Inspired by the structure of the human
brain, deep learning networks consist of interconnected layers of artificial neurons
that process data in a hierarchical manner, with each layer learning more abstract
representations of the input. This ability to capture intricate patterns has made deep
learning a key technology behind breakthroughs in areas like autonomous driving,
medical imaging, and AI-driven personal assistants. By leveraging large datasets
and high computational power, deep learning systems can improve their accuracy
and performance over time, contributing to significant advancements in artificial
intelligence.

1.1.5 CONVOLUTIONAL NEURAL NETWORK

A Convolutional Neural Network (CNN) is a type of deep learning model that

is particularly effective for tasks involving image and visual data. CNNs are
designed to automatically and adaptively learn spatial hierarchies of features from
input images, making them highly efficient in tasks like image classification, object
detection, and facial recognition. The core components of a CNN are convolutional
layers, which apply filters to detect features such as edges, textures, and shapes, and
pooling layers, which reduce the spatial dimensions of the data to retain essential
features while reducing computational complexity. CNNs can capture both low-level
and high-level features from images through multiple layers of convolutions,
enabling them to learn complex visual patterns. Due to their ability to handle high-
dimensional data, CNNs have become the go-to architecture for most computer

13
vision applications, contributing to advancements in fields like medical imaging,
autonomous vehicles.

14
 1.1.6 GRAD-CAM

Grad-CAM (Gradient-weighted Class Activation Mapping) is an

interpretability technique used in deep learning, particularly in convolutional neural
networks (CNNs), to visualize and understand the decisions made by a model. It
highlights the important regions in an input image that significantly influence the
model’s predictions by utilizing the gradients flowing into the final convolutional
layer of the network. Grad-CAM produces heatmaps that overlay on the original
image, showing which areas the model focused on while making a specific
classification or prediction. This interpretability is crucial in sensitive domains such
as healthcare, where understanding why a model has made a certain diagnosis, such
as identifying diabetic retinopathy from retinal images, is as important as the
prediction itself. Grad-CAM helps build trust in AI systems by offering a transparent
view into the decision-making process, allowing users to verify the model’s focus on
relevant and meaningful image features.

1.2 PROBLEM STATEMENT

1.2.3 Classify retinal images into five stages of diabetic retinopathy.

1.2.4 AI system can identify and highlight critical retinal regions for
Diabetic Retinopathy diagnosis.
1.2.5 Improving the detection accuracy can increase the early prediction of
Diabetic Retinopathy.
1.2.6 Edge-computing can help in real-time diabetic retinopathy screening.

15
1.3 OBJECTIVE OF THE PROJECT

1.3.3 Developing a high-accuracy deep learning model to detect and classify

diabetic retinopathy into its stages using retinal images.
1.3.4 This system enhances the diagnostic precision by incorporating multi-
modal retinal image fusion for comprehensive visual analysis.
1.3.5 The objective is to ensure model interpretability with Grad-CAM for
transparent, explainable predictions.
1.3.6 Optimizing the model for real-time diagnosis on edge devices like
smartphones and PCs for fast, accessible screening.
1.3.7 Improving early detection and management to enable timely interventions
and prevent vision loss.
1.3.8 Supporting scalable deployment in resource-constrained settings, making
the model usable in remote regions.

1.4 ORGANIZATION OF THE REPORT

The report is organized as follows: Chapter 1 introduces the project and its
key terms. Chapter 2 presents a literature review, examining current relevant
initiatives. Chapter 3 details the software and hardware requirements for the
project. Chapter 4 outlines the existing system, highlighting its disadvantages, and
proposes the new system. Chapter 5 describes the functional design and block
diagram of the project. Chapter 6 concludes the report, discussing potential
improvements for the project.

16
 CHAPTER 2

LITERATURE SURVEY

2.1 INTRODUCTION

Diabetic retinopathy (DR) is one of the leading causes of blindness worldwide,

particularly among individuals with long-term diabetes. As the disease progresses, it
damages the blood vessels in the retina, potentially leading to vision loss if not
detected and treated early. Traditional methods of detecting DR rely heavily on manual
examination of retinal images by ophthalmologists, a time-consuming process that is
prone to human error, especially in the early stages when symptoms may be subtle.
With the increasing prevalence of diabetes and the subsequent rise in DR cases, there
is a pressing need for automated, accurate, and efficient diagnostic tools. Deep
learning, a subset of artificial intelligence, has emerged as a transformative solution in
medical imaging, offering the potential to revolutionize DR detection by automating
the analysis of retinal images with high precision.

Deep learning models, particularly convolutional neural networks (CNNs), have

demonstrated remarkable success in image-based diagnostics due to their ability to
learn intricate patterns and features from large datasets. In the context of DR detection,
CNNs can analyze retinal images to detect various stages of the disease, ranging from
no DR to proliferative DR, which is the most severe form. These models not only
enhance diagnostic accuracy but also reduce the time and effort required for screening,
making them suitable for large-scale implementation in healthcare systems.
Furthermore, by incorporating techniques such as ensemble modeling and data
augmentation, deep learning approaches can address challenges like imbalanced
datasets and improve the detection of early-stage DR, which is crucial for timely
intervention. As a result, the application of deep learning to diabetic retinopathy

17
detection holds immense promise in improving both the quality and accessibility of
eye care.

18
2.2 LITERATURE REVIEW
2.2.1 Data Diversity in Convolutional Neural Network Based Ensemble Model
for Diabetic Retinopathy:
Recent studies on CNN-based ensemble models for diabetic retinopathy
detection highlight the critical role of data diversity in enhancing model
performance and improving generalization. This ensemble learning techniques,
including popular methods such as bagging and boosting, are designed to combine
the strengths of multiple models to increase predictive accuracy. The effectiveness
of ensemble learning is significantly influenced by the diversity within the training
datasets, which helps the models learn from variations in image quality, patient
demographics, and the different stages of disease severity. By training CNN
models on diverse datasets, researchers can ensure that the models are better
equipped to generalize across different populations and real-world scenarios,
making the systems more reliable and robust. Incorporating data diversity has
proven to be an effective strategy for addressing issues like overfitting, where a
model becomes too specialized in its training data and performs poorly on unseen
data. Data augmentation techniques, such as rotation, scaling, flipping, and
cropping, are commonly applied to artificially expand the dataset, introducing
variability and ensuring that the CNN models learn more comprehensive features.
However, despite the clear benefits, there are still challenges associated with data
diversity in CNN-based ensemble models for diabetic retinopathy detection.

2.2.2 A Hybrid Convolutional Neural Network Model for Automatic

Diabetic Retinopathy Classification from Fundus Images:

Recent advancements in diabetic retinopathy (DR) classification have

increasingly focused on hybrid CNN models to improve both accuracy and
efficiency in automatically analyzing and classifying fundus images. These hybrid

19
models combine the strengths of convolutional neural networks (CNNs) with other

20
machine learning techniques like support vector machines (SVMs), decision trees,
or advanced feature extraction methods to achieve better predictive performance.
CNNs are highly effective in capturing spatial features from retinal images, but
when combined with additional machine learning classifiers or feature selection
algorithms, the overall model gains the ability to harness both low-level and high-
level image details. This multi-layered approach enhances the precision of diabetic
retinopathy classification by allowing different techniques to focus on distinct
aspects of image processing, resulting in more robust and nuanced predictions. For
instance, CNNs can be employed to extract deep spatial features from retinal
fundus images, such as identifying subtle patterns or lesions, while classifiers like
SVMs or decision trees can process these features to make more accurate and
discriminative predictions regarding the stage of DR. Studies have shown that
hybrid models outperform standalone CNNs because they incorporate additional
feature selection mechanisms, which can better capture the complex nature of
diabetic retinopathy. To further enhance the robustness of these models, data
augmentation techniques like image rotation, flipping, zooming, and scaling are
commonly applied, artificially increasing the diversity of the training dataset and
helping the model generalize better to unseen data. Additionally, ensemble
methods, which combine the predictions of multiple CNNs or integrate CNNs with
traditional machine learning models, have demonstrated success in improving the
generalization of DR classification, especially across diverse patient populations
and varying image qualities.

2.2.3 Risk Stratification for Diabetic Retinopathy Screening Order Using

Deep Learning: A Multicenter Prospective Study:

Recent research on risk stratification for diabetic retinopathy (DR) screening

using deep learning has shown remarkable promise in transforming how screening

21
programs prioritize patients based on their risk of developing severe stages of DR.

22
Traditionally, DR screening programs have treated patients uniformly, with little
distinction in terms of the urgency of their cases, potentially leading to
inefficiencies in managing healthcare resources. By employing deep learning,
particularly convolutional neural networks (CNNs), researchers are now able to
analyze retinal fundus images and stratify patients into different risk categories.
This approach allows for a more targeted screening process, where patients at
higher risk of progressing to severe or proliferative DR are identified early and
given priority for further evaluation and treatment. This risk-based prioritization
not only improves patient outcomes by enabling earlier intervention but also
optimizes resource allocation, ensuring that healthcare providers focus their
attention on those who need it most. Multicenter prospective studies have been
pivotal in demonstrating the efficacy of these deep learning models. By training
CNNs on large and diverse datasets sourced from multiple clinical centers,
researchers can create models that are more generalizable and adaptable to
different populations. These multicenter datasets capture a wide range of variations
in disease presentation, imaging quality, and patient demographics, allowing the
deep learning models to automatically assess disease progression with high
accuracy. Through this automated risk stratification, the models can flag high-risk
patients who require immediate attention, while low-risk patients can be monitored
with less frequent check-ups. This stratified approach enhances the efficiency of
DR screening programs, potentially reducing the burden on healthcare systems and
preventing vision loss in high-risk individuals.

2.2.4 Dual-Path and Multi-Scale Enhanced Attention Network for Retinal

Diseases Classification Using Ultra-Wide-Field Images:

Recent studies on retinal disease classification have increasingly focused on

the innovative application of dual path and multi-scale enhanced attention

23
networks, particularly in leveraging the rich details captured by ultra-wide
field (UWF)

24
imaging. UWF imaging represents a significant advancement in retinal
photography, allowing for the visualization of a much larger portion of the retina
compared to traditional fundus images. This capability provides clinicians with
more comprehensive data that can be critical for the early detection and accurate
classification of retinal diseases, as it encompasses more anatomical features and
potential pathologies. Traditional imaging techniques often miss peripheral lesions
and abnormalities that can be crucial indicators of diseases like diabetic
retinopathy, age-related macular degeneration, and retinal vein occlusion. By
utilizing UWF images, researchers can enhance the data available for machine
learning models, ultimately leading to improved diagnostic performance, and
detection of the diabetic retinopathy. The dual path network approach is
particularly effective in processing UWF images because it enables the
simultaneous analysis of both global and local features. This capability is vital
when examining retinal images where macroscopic features, such as overall retinal
architecture, must be assessed alongside microscopic details, such as small lesions
or vascular changes. By capturing and integrating these varying scales of
information, dual path networks ensure that models do not overlook subtle yet
significant pathological features that might be crucial for an accurate diagnosis.
Complementing this approach, the multi-scale enhanced attention mechanism
dynamically adjusts the model's focus across different regions of the image at
various resolutions. This allows the model to prioritize and highlight critical areas
during classification, thereby enhancing its sensitivity to pathological changes.

2.2.5 MTRA-CNN: A Multi-Scale Transfer Learning Framework for

Glaucoma Classification in Retinal Fundus Images:

The MTRA CNN, or Multi-Scale Transfer Learning CNN, represents a significant

breakthrough in the automated detection of glaucoma through the analysis of

25
retinal fundus images. As a multi-scale transfer learning framework, it effectively
utilizes

26
pre-trained convolutional neural network (CNN) models to extract both low-level
and high-level features from retinal images, streamlining the classification
process. This approach addresses one of the main challenges in developing deep
learning models for medical imaging: the need for extensive, annotated datasets
that can be both costly and time-consuming to compile. By leveraging transfer
learning, the MTRA CNN can build upon existing knowledge captured in pre-
trained models, which have already learned to identify relevant features from vast
amounts of data. This allows for the effective classification of glaucoma without
necessitating the same level of extensive, specific annotation, thus reducing the
barrier to entry for deploying advanced diagnostic tools in clinical settings.A
notable feature of the MTRA CNN framework is its multi-scale capability, which
enhances the model’s performance by capturing features at different resolutions.
This is particularly important for glaucoma classification, where the pathological
signs are often subtle and vary in scale. For example, the optic nerve head and the
retinal nerve fiber layer are critical indicators of glaucoma, and detecting changes
in these structures requires a nuanced understanding of both localized details and
broader patterns. By integrating multiple scales, the MTRA CNN is able to
perform a more thorough analysis of retinal structures, ensuring that it can detect
and classify glaucoma effectively. Recent studies have highlighted that this multi-
scale approach results in superior performance metrics compared to traditional
CNN models, particularly in terms of classification accuracy and sensitivity. This
improvement is crucial, as it can lead to earlier detection and intervention, which
is vital for preserving vision in glaucoma patients.

2.2.6 Convolutional Neural Networks for Automated Diagnosis of Diabetic

Retinopathy in Fundus Images:
Recent literature on Convolutional Neural Network (CNN)-based networks for the
automated diagnosis of diabetic retinopathy (DR) in fundus images underscores
27
their increasing efficacy in the domain of medical image analysis. CNNs have
gained widespread traction in the field due to their exceptional capability to
automatically extract hierarchical features from complex retinal images. This
ability is pivotal for accurately classifying various stages of DR, from mild to
proliferative forms. Through deep learning, CNNs are able to learn intricate
patterns that signify the presence of DR, such as microaneurysms, hemorrhages,
and exudates—key indicators that often appear subtle to the human eye but are
critical for timely diagnosis and intervention. The sophisticated architectures of
CNNs allow for multi-layered feature extraction, enabling models to identify both
local features and broader contextual information, which enhances the overall
diagnostic accuracy. Numerous studies have demonstrated that CNN models
consistently outperform traditional machine learning methods in terms of
classification performance. Specifically, architectures like ResNet, Inception, and
DenseNet have been prominently utilized in this context, achieving high
sensitivity and specificity in detecting DR. These models benefit from residual
connections, varying kernel sizes, and dense connectivity patterns that improve
their ability to learn from complex datasets. The integration of advanced data
augmentation and pre-processing techniques, such as image normalization,
contrast enhancement, and rotation, further enhances model robustness by
artificially expanding the training dataset and helping the model generalize better
to unseen data. By mitigating the risks of overfitting and improving model
stability across varying imaging conditions, these techniques play a crucial role in
ensuring reliable predictions.

2.2.7 An Efficient DenseNet for Diabetic Retinopathy Screening:

Recent studies focusing on DenseNet-based models for diabetic retinopathy

(DR) screening have underscored their significant effectiveness in enhancing both

28
classification accuracy and computational efficiency. DenseNet, short for Dense
Convolutional Network, is a sophisticated deep learning architecture characterized

29
by its unique connectivity pattern, where each layer is connected to every
subsequent layer in a feed-forward manner. This innovative structure allows for
efficient feature reuse throughout the network, significantly reducing the number
of parameters needed while still enabling the model to learn and capture complex
features present in retinal fundus images. This is particularly advantageous in the
context of DR screening, where accurately identifying and classifying subtle
retinal abnormalities is essential for effective diagnosis and timely intervention. In
the realm of diabetic retinopathy screening, DenseNet has shown remarkable
performance in detecting critical retinal abnormalities, including microaneurysms,
hemorrhages, and exudates—key indicators that signal the presence and
progression of DR. These features, although often subtle and challenging to
discern, are crucial for determining the various stages of the disease, from mild to
proliferative forms. The architecture's efficient parameter usage not only enables it
to maintain high levels of accuracy but also makes it less demanding in terms of
computational resources when compared to other popular architectures like
ResNet or VGG. This efficiency is particularly important in clinical settings where
computational power may be limited or where rapid analysis of images is
required.

2.2.8 Diabetic retinopathy identification using parallel convolutional neural

network based feature extractor and ELM classifier:

Recent advancements in the identification of diabetic retinopathy (DR) have

focused on the innovative combination of parallel Convolutional Neural Networks
(CNNs) for feature extraction and Extreme Learning Machine (ELM) classifiers to
significantly enhance classification performance. This hybrid approach leverages
the strengths of both methodologies to improve the accuracy and efficiency of DR
diagnosis. The use of parallel CNNs allows for the automatic extraction of multi-
30
level features from retinal fundus images, effectively capturing a broad range of

31
information. Low-level features such as textures and edges are vital for identifying
subtle abnormalities, while high-level features encapsulate more complex patterns
that may signify the presence of DR-related lesions, including microaneurysms
and hemorrhages. This dual-level feature extraction is crucial, as it enables the
model to recognize diverse manifestations of DR, which can vary significantly
among patients. The parallel structure of the CNNs facilitates simultaneous
processing of features at multiple scales, further enhancing the model's capability
to identify and classify a wide array of DR-related signs. This multi-scale analysis
is particularly beneficial in the context of fundus images, where variations in
lesion size and location can influence diagnosis. Following the feature extraction
phase, the ELM classifier is employed to categorize the images into various
severity levels of DR. ELM is renowned for its rapid learning speed and strong
generalization capabilities, which allows it to effectively utilize the features
extracted by the CNNs. Research has demonstrated that this hybrid model
consistently outperforms standalone CNN classifiers, providing not only improved
accuracy but also significantly faster training times. This efficiency is particularly
important in clinical settings, where timely diagnoses can lead to better patient
outcomes and more effective management of the disease.

2.2.9 A fast interpretable adaptive meta-learning enhanced deep learning

framework for diagnosis of diabetic retinopathy:

Recent research on fast, interpretable adaptive meta-learning frameworks for

diabetic retinopathy (DR) detection underscores the growing demand for medical
AI systems that not only achieve high accuracy but also offer transparency and
adaptability. Meta-learning, often referred to as "learning to learn," is particularly
valuable in medical contexts like DR screening, where large and diverse datasets
are difficult to obtain. This approach allows AI models to generalize effectively

32
across various retinal image datasets by learning from multiple related tasks, such

33
as distinguishing between different stages of DR or identifying other retinal
diseases. By enabling models to adapt quickly to new tasks with minimal data,
meta-learning frameworks offer a solution to the challenges of data scarcity and
heterogeneity in retinal imaging. One of the key advantages of these frameworks
is their emphasis on interpretability, which is vital for building trust in AI systems
within the medical community. In clinical settings, it is not enough for an AI
model to simply make a prediction; it must also provide clear, interpretable
explanations for its decisions, whether through visualizations (such as heatmaps)
or feature- based insights that highlight critical areas in the retinal images.
Techniques like Gradient-weighted Class Activation Mapping (Grad-CAM) are
often employed to make the model’s decision-making process more transparent,
allowing clinicians to understand why certain regions of the retina were flagged as
indicators of DR.

2.2.10 Optical Identification of Diabetic Retinopathy Using

Hyperspectral Imaging:

Recent research into the optical identification of diabetic retinopathy (DR) using
hyperspectral imaging (HSI) has highlighted significant breakthroughs in
improving early detection and diagnosis of the disease. Hyperspectral imaging is a
powerful technology that captures a broad range of the electromagnetic spectrum,
gathering detailed information about retinal tissues beyond what is possible with
traditional imaging methods like fundus photography. By capturing and analyzing
this vast array of wavelengths, HSI can detect subtle biochemical and structural
changes in the retina that are associated with the early stages of DR, such as
variations in oxygenation and blood flow. These subtle changes may be invisible
in conventional imaging techniques but are crucial for diagnosing DR at its
earliest stages, where intervention can prevent disease progression. Studies have
34
shown that HSI can differentiate between healthy and diseased retinal tissues by
identifying

35
specific spectral signatures that correlate with key retinal abnormalities, such as
microaneurysms, haemorrhages, and lipid exudates—important markers for DR
progression. The integration of hyperspectral imaging with advanced machine
learning techniques, particularly convolutional neural networks (CNNs), has
further enhanced the accuracy and efficiency of DR detection. CNNs are well-
suited for analysing the complex, high-dimensional data generated by HSI,
allowing the model to automatically learn and recognize patterns in spectral
information that correspond to early signs of DR. This combination of HSI and
deep learning has enabled the automated identification of microscopic features
that may precede visible symptoms of DR, such as microaneurysms and exudates,
allowing for earlier and more precise diagnosis. Moreover, research suggests that
hyperspectral data can be used to generate quantitative assessments of retinal
health, potentially providing new metrics for tailoring individualized treatment
plans based on a patient’s unique retinal profile.

36
 CHAPTER 3

SYSTEM REQUIREMENT SPECIFICATION

3.1 INTRODUCTION
A thorough understanding of the commissioning and operational requirements of
the proposed system is crucial, with a key focus on gaining a comprehensive grasp of
the hardware and software needs for the project.

3.2 SOFTWARE REQUIREMENT

3.2.1 JUPYTER NOTEBOOK
The Jupyter Notebook is an open-source web application that provides an
interactive computing environment, allowing users to create and share documents that
contain live code, equations, visualizations, and narrative text. It is widely used in data
science, machine learning, and scientific research for its flexibility and ease of use.
Users can write code in various programming languages, including Python, R, and Julia,
and execute it in real-time, making it ideal for data exploration and analysis. The ability
to combine code with rich text elements, such as Markdown, allows for the creation of
well- documented analyses and presentations. Additionally, Jupyter Notebook supports
the integration of various libraries and tools for data visualization, enabling users to
create dynamic graphs and charts within the notebook. Its interactive nature and support
for version control make it a popular choice for collaborative projects and educational
purposes, fostering an engaging learning experience for students and professionals alike.

3.2.2 FEATURES OF JUPYTER NOTEBOOK

 Interactive Code Execution: Jupyter Notebook allows users to run code in real-

time and view the output immediately, making it ideal for experimenting and
debugging.
37
 Support for Multiple Languages: While primarily used for Python, Jupyter also

supports other programming languages such as R, Julia, and SQL via different kernels.
 Rich Text Integration: With support for Markdown, users can include formatted

text, equations (via LaTeX), images, and links alongside code, making it useful for
creating comprehensive, well-documented reports.
 Data Visualization: Jupyter integrates with numerous libraries such as Matplotlib,

Seaborn, and Plotly to generate dynamic and interactive visualizations directly within
the notebook.

Fig 3.1: Jupyter Notebook

3.3 PYTHON

Python is a high-level, interpreted programming language known for its simplicity

and versatility. Designed to emphasize readability with its clean syntax, Python allows
developers to write code that is easy to understand and maintain. It supports multiple
38
programming paradigms, including procedural, object-oriented, and functional

39
programming, making it a flexible choice for a wide range of applications. Python is
extensively used in web development, data science, artificial intelligence, machine
learning, automation, and scientific computing due to its vast collection of libraries and
frameworks such as Django, Flask, Pandas, NumPy, and TensorFlow. Its cross-platform
compatibility and strong community support have contributed to Python's widespread
adoption, making it one of the most popular programming languages in the world today.

3.3.1 FEATURES OF PYTHON

Python provides many useful features which make it popular and valuable from
the other programming languages. It supports object-oriented programming, procedural
programming approaches and provides dynamic memory allocation. We have listed
below a few essential features.

1) Simple and easy to learn:

Python has a clean, readable syntax that makes it easier for beginners and developers
to write and understand code.

2) Interpreted language:

Python executes code line by line, allowing for immediate testing and debugging
without needing a compilation step.

3) Object-oriented programming (OOP):

Python supports OOP concepts like classes, inheritance, and encapsulation,

making it a powerful tool for designing complex applications.

40
4) Extensive standard library:

Python comes with a vast standard library that provides modules and functions
for tasks like file I/O, regular expressions, web development, and more.

5) Cross-platform:

Python runs on various platforms, including Windows, macOS, and Linux,

allowing for cross-platform development with minimal changes to the code.

6) Dynamic typing:

Python is dynamically typed, meaning you don’t need to declare variable types
explicitly. The interpreter determines the type at runtime.

7) Large community and support:

Python has a massive community, extensive documentation, and many tutorials,

making it easier to find help and learn.

8) Support for multiple programming paradigms:

Python supports various programming styles, including procedural, object-

oriented, and functional programming.

9) Third-party libraries and frameworks:

Python has a rich ecosystem of libraries and frameworks like NumPy, Pandas,
Django, Flask, and TensorFlow, which extend its capabilities.

41
10) Integration capabilities:

Python can be easily integrated with other languages (like C/C++/Java) and
systems, making it versatile for building different kinds of applications.

Fig 3.2: Python

3.3.2 NumPy

NumPy, short for Numerical Python, serves as the foundational library for
numerical computing in Python. Its primary offering is the `ndarray` object, an N-
dimensional array that allows for efficient storage and manipulation of homogeneous
data types. This array structure is designed to handle large datasets and perform
mathematical computations swiftly, making it indispensable for tasks involving
complex numerical calculations, particularly in fields like machine learning and image
processing.
One of the standout features of NumPy is its ability to perform element-wise
operations. This capability enables users to conduct mathematical operations on entire
arrays without needing explicit loops, significantly enhancing code readability and
performance.

42
3.3.3 Keras

Keras is an open-source high-level neural networks API that runs on top of

TensorFlow, making it easier for developers and researchers to build and train deep
learning models. Its user-friendly interface and modular design have made Keras a
popular choice for rapid prototyping and experimentation, particularly in complex
tasks like image recognition, which is at the core of your diabetic retinopathy
detection project.
One of Keras's standout features is its simplicity. With just a few lines of
code, users can define a neural network architecture, compile the model, and start
training. This ease of use is particularly beneficial when iterating on different model
designs, allowing you to focus on optimizing performance rather than getting
bogged down in technical details. For example, you can quickly switch between
different types of layers-such as convolutional, pooling, and dense layers-to
experiment with various network architectures that best capture the features of
retinal images.

3.3.4 PyTorch

PyTorch is an open-source machine learning framework that has gained

significant traction in the research and development community due to its dynamic
computation graph and user-friendly interface. Developed by Facebook’s AI
Research lab, PyTorch provides a flexible platform for building deep learning
models, making it especially suitable for complex tasks such as diabetic retinopathy
detection. One of the defining features of PyTorch is its dynamic computation
graph, also known as "define-by-run." This allows developers to build networks on-
the-fly, modifying the architecture and structure during runtime. This flexibility is
crucial for research- oriented projects where rapid experimentation is essential. In

43
your diabetic retinopathy detection project, you may need to try various model
architectures or modify hyperparameters quickly to achieve optimal performance.

44
3.3.5 NumPy

NumPy, short for Numerical Python, serves as the foundational library for
numerical computing in Python. Its primary offering is the `ndarray` object, an N-
dimensional array that allows for efficient storage and manipulation of
homogeneous data types. This array structure is designed to handle large datasets
and perform mathematical computations swiftly, making it indispensable for tasks
involving complex numerical calculations, particularly in fields like machine
learning and image processing.
One of the standout features of NumPy is its ability to perform element-wise
operations. This capability enables users to conduct mathematical operations on
entire arrays without needing explicit loops, significantly enhancing code readability
and performance. For example, when processing images for diabetic retinopathy
detection, you can easily manipulate pixel data, applying filters or transformations
uniformly across the dataset. Such operations are not only more intuitive but also
executed at speeds unattainable with Python’s built-in lists.

3.3.6 Pandas

Pandas is a powerful open-source library specifically designed for data

manipulation and analysis in Python. It provides flexible data structures and
functions tailored for working with structured data, making it a favorite among data
scientists, especially in the realm of machine learning and data analysis. Its core
components- Series and DataFrame - facilitate the handling of one-dimensional and
two- dimensional data, respectively, allowing for seamless integration of diverse
data types.
The DataFrame, in particular, is central to your project, as it enables the
organization of complex datasets that may include retinal images alongside patient

45
information. With its intuitive indexing and slicing capabilities, Pandas simplifies

46
data retrieval and manipulation, making it easy to filter datasets based on specific
criteria or to group data for statistical analysis. For instance, you can effortlessly
merge patient metadata with corresponding image data, creating a cohesive dataset
that supports multi-modal data fusion, a critical aspect of your proposed system.

3.3.7 Matplotlib

Matplotlib is a powerful and versatile plotting library for Python that provides
a comprehensive suite of tools for creating static, animated, and interactive
visualizations in a wide variety of formats. As one of the foundational libraries for
data visualization in the Python ecosystem, it is particularly essential for projects
like diabetic retinopathy detection, where understanding data visually can lead to
better insights and model performance.
At the core of Matplotlib is the ability to generate high-quality plots with
minimal code. The library's primary interface, `pyplot`, allows users to create a
range of plots—line graphs, scatter plots, histograms, and more—using simple
function calls. This simplicity makes it accessible for both beginners and advanced
users, allowing for rapid prototyping and exploration of data. For instance, in your
project, you can quickly visualize the distribution of pixel intensities in retinal
images or compare model performance metrics across different experiments.

3.3.8 Seaborn

Seaborn is a statistical data visualization library built on top of Matplotlib,

designed to simplify the creation of informative and attractive visual representations
of data. It provides a high-level interface for drawing attractive graphics and
enhances the capabilities of Matplotlib by integrating more advanced statistical
visualizations and better aesthetics. In the context of your diabetic retinopathy

47
detection project, Seaborn can play a vital role in exploring complex relationships
within your datasets

48
and presenting findings in a clear and engaging manner.
One of Seaborn’s key strengths is its ability to handle data frames directly,
which aligns seamlessly with Pandas. This integration allows users to create
complex visualizations with minimal code, making it easier to explore and analyze
the data. For instance, you can quickly generate scatter plots, heatmaps, and box
plots that illustrate the relationships between features in your dataset, such as pixel
intensity distributions and their correlation with diabetic retinopathy severity.

3.3.9 OpenCV

OpenCV (Open Source Computer Vision Library) is a highly efficient library

designed for real-time computer vision and image processing tasks. With its
comprehensive suite of tools, OpenCV enables developers to manipulate and
analyze images and video, making it an indispensable resource for projects like
diabetic retinopathy detection.
One of the key strengths of OpenCV is its extensive collection of algorithms
for image processing. These algorithms cover a wide range of functionalities,
including filtering, edge detection, image segmentation, and feature extraction. For
your project, these capabilities are essential for preprocessing retinal images, which
is a critical step in ensuring accurate model training and evaluation. For instance,
techniques such as histogram equalization can enhance image contrast, making it
easier for models to identify relevant features indicative of diabetic retinopathy.

3.3.10 Scikit-Image

scikit-image is an open-source image processing library for Python that

extends the capabilities of NumPy. Built on top of the SciPy ecosystem, scikit-
image provides a collection of algorithms for image processing and computer
vision, making it an invaluable resource for tasks that require sophisticated
49
image manipulation and

50
analysis. In the context of your diabetic retinopathy detection project, scikit-image
can play a crucial role in enhancing the quality of retinal images and extracting
meaningful features.
One of the standout features of scikit-image is its comprehensive set of image
processing algorithms, which include filtering, morphology, segmentation, feature
extraction, and more. These algorithms are designed to work seamlessly with
NumPy arrays, allowing for efficient processing of image data. For example, you
can apply Gaussian blurring to reduce noise in retinal images, or use edge detection
techniques to highlight the boundaries of features that may be indicative of diabetic
retinopathy.

51
 CHAPTER 4

PROJECT DESCRIPTION

4.1 EXISTING SYSTEM

The Convolutional Neural Networks (CNNs) are a powerful tool for analyzing
retinal images, especially when identifying features associated with diabetic
retinopathy, a common complication of diabetes that affects the eyes. CNNs work
by leveraging layers of filters to capture patterns and features from the retinal
images, such as blood vessel structures, exudates, and hemorrhages, which are
indicative of diabetic retinopathy. The multi-layer architecture of CNNs enables
the automatic extraction of low-level and high-level features, making them ideal
for image classification tasks. Given the complexity of retinal images and the
subtlety of diabetic retinopathy features, CNNs provide a robust solution for
automating this diagnostic process.

Transfer learning plays a key role in enhancing the performance of CNN

models. It involves using pre-trained models that have already learned features
from large datasets, often from tasks similar to image classification. These models
are fine- tuned to the specific task of diabetic retinopathy detection, which allows
for faster training and better accuracy even when the retinal image dataset is
limited. This approach reduces the need for training large, complex models from
scratch, saving computational resources and time, while leveraging the knowledge
the model has acquired from the source domain to improve performance in the
target domain.

Ensemble methods are employed to further boost classification accuracy by

combining the predictions from multiple models. The rationale behind ensemble
methods is that different models may capture different aspects of the data, and by

52
integrating these perspectives, the system becomes more resilient to overfitting and
variance. Techniques such as bagging or boosting can be used to average or weight

53
 predictions from various CNN architectures, reducing the risk of errors from
individual models. By aggregating predictions, ensemble models offer more stable
and reliable outputs, which are particularly useful in medical image analysis, where
minimizing false positives and negatives is crucial.

DISADVANTAGES

 CNNs require significant computational resources, including powerful GPUs and

large memory, making them expensive and time-consuming to train.
 CNNs need large, well-annotated datasets to perform well, and obtaining quality
retinal images with accurate labels can be challenging in medical fields.
 CNNs can overfit on small datasets and are often difficult to interpret, which is a
concern in medical diagnostics where transparency is crucial.
 CNN models are sensitive to variations in image quality, lighting, and device
differences, which can affect prediction accuracy and lead to biases across patient
groups.
 CNNs may struggle with unseen or rare conditions, reducing effectiveness in real-
world scenarios.
 Building and tuning CNN architectures can be complex, requiring expertise that
limits accessibility to non-experts.
 CNNs can reflect biases in training data, leading to unequal performance across
demographic groups.
 Predictions often need validation from medical professionals, reducing the benefits
of automation.

54
4.2 PROPOSED SYSTEM
The proposed system for the project aims to advance the current approach of
diagnosing diabetic retinopathy by integrating hybrid models, and Explainable AI
(XAI) technique. This integration helps capture multiple perspectives of the retinal
structure, improving diagnostic accuracy by leveraging the strengths of different
imaging modalities. By fusing these data sources, the system can identify subtle
patterns that might not be visible in a single type of image, leading to more precise
and robust diagnoses.
At the heart of the system is a hybrid model that combines Convolutional
Neural Networks (CNNs) with Transformer architectures. The CNN component
excels at extracting detailed local features from retinal images, such as small
lesions, microaneurysms, and abnormal blood vessel formations. Meanwhile, the
Transformer component is designed to recognize global patterns and relationships
across the entire image, such as large-scale structural changes in the retina. By
combining these two approaches, the hybrid CNN-Transformer model can capture
both fine-grained details and overall patterns, which enhances its ability to detect
diabetic retinopathy at various stages with high accuracy.
To ensure the system’s transparency and trustworthiness, Explainable AI (XAI)
techniques like Grad-CAM (Gradient-weighted Class Activation Mapping) is
employed. Grad-CAM helps highlight the regions of the image that the model
focuses on when making predictions, offering visual explanations of why certain
areas are considered indicative of DR.

4.2.1 EfficientNetB0

EfficientNetB0 is a lightweight and highly efficient convolutional neural

network (CNN) architecture that has shown promising results in medical image
classification tasks, including diabetic retinopathy detection. Developed by Google
55
 AI, EfficientNet introduces a novel compound scaling method that uniformly scales
network depth, width, and resolution using a set of fixed scaling coefficients.
EfficientNetB0, the baseline model in the family, strikes a perfect balance between
performance and computational cost, making it ideal for deployment in resource-
constrained environments such as mobile and edge devices.
In the context of diabetic retinopathy detection, EfficientNetB0 efficiently
captures fine-grained features such as microaneurysms, hemorrhages, and exudates
by leveraging its optimized architecture and depth-wise separable convolutions. The
model has been trained on large retinal image datasets like APTOS and EyePACS,
achieving high accuracy while maintaining a small model size. Its ability to
generalize well on limited data and its robustness to overfitting make it a preferred
choice for real-world DR screening applications, particularly when explainability
and deployment efficiency are essential.

4.2.2 ResNet50

ResNet50, a 50-layer deep convolutional neural network introduced by

Microsoft Research, has become one of the most widely adopted backbone
architectures in the field of medical imaging. It leverages residual learning through
skip connections, allowing the model to train deeper networks without suffering
from vanishing gradients. This capability is particularly beneficial for diabetic
retinopathy detection, where identifying subtle and complex retinal anomalies
requires learning deep and hierarchical representations.
In diabetic retinopathy classification tasks, ResNet50 demonstrates strong
performance in extracting meaningful features from high-resolution retinal fundus
images. Its deep architecture is capable of detecting intricate retinal lesions and
classifying DR into multiple stages with high precision. When fine-tuned on DR-
specific datasets, ResNet50 has shown competitive accuracy and recall rates,
56
making it a reliable choice for automated DR grading systems. Additionally, it
integrates well with explainable AI techniques like Grad-CAM, which helps
visualize regions of interest and supports clinical validation of the model's
predictions.

57
 CHAPTER 5
FUNCTIONAL DESIGNS
5.1 FUNCTIONAL DIAGARAM

A functional diagram is a graphical representation used to illustrate the

relationships between different components, functions, or processes within a system.
It breaks down the system into its core functions and shows how they interact, flow,
or depend on one another. Functional diagrams are widely used in engineering,
software development, and systems design to visually convey how a system
operates or how its components are connected.

Fig 5.1 Functional Diagram for Detection of Diabetic Retinopathy

58
5.1.1 Data Collection

In this stage, the system gathers retinal images and other relevant medical data.
The data collected typically includes:

 Retinal fundus images: These images capture detailed visual information

about the retina, where diabetic retinopathy lesions, such as microaneurysms,
haemorrhages, and exudates, are present.
 Patient history: Sometimes, other health metrics such as blood sugar levels,
age, and duration of diabetes might be integrated to enhance the prediction
accuracy.

5.1.2 Hybrid CNN-Transformer Model

The system employs a hybrid Convolutional Neural Network (CNN) and

Transformer model for feature extraction and pattern recognition.

 CNN: Convolutional Neural Networks are well-suited for analysing and

extracting local features from retinal images, such as microaneurysms,
haemorrhages, and exudates.
 Transformer: Transformers, typically used for sequence and attention-based
tasks, are integrated into the system to capture global patterns and context.
This combination allows the model to focus on detailed local features and
also understand the overall image structure.
 The hybrid architecture enhances interpretability and accuracy by taking
advantage of CNNs for image data and Transformers for understanding global
interactions in the features.

59
 5.1.3 Explainable AI with Grad-CAM

In this step, the system applies Explainable AI (XAI) techniques to provide

insights into the model's decisions. This is crucial in healthcare applications, where
understanding why a model makes a certain decision is important for clinical
acceptance and trust.

 Grad-CAM (Gradient-weighted Class Activation Mapping): Grad-CAM

generates heatmaps that highlight important regions in the retinal image,
helping clinicians understand which parts of the image the model focused on
for making a diagnosis.

5.2 FUNCTIONAL MODULES

A block diagram is a simplified graphical representation of a system, process, or

concept that uses blocks (usually rectangular shapes) to represent components or
functions. The blocks are connected by lines or arrows to show the relationships, data
flow, or interaction between them. Block diagrams are widely used in engineering,
electronics, software design, and business processes to illustrate how parts of a system
work together.

60
 Input Retinal Images

Image Preprocessing

Image Segmentation

Image Enhancement

Extraction of Retinal Parameters

Abnormal Feature Extraction

DR Disease Severity Analysis

No DR Detected Proliferate NPDR

Mild NPDR Moderate NPDR Severe NPDR

5.2 Block Diagram for Detection of Diabetic Retinopathy

5.2.1 Input retinal images

The process begins with the acquisition of retinal images. These images are
typically captured using fundus cameras or other imaging techniques such as Optical
Coherence Tomography (OCT). These images are vital for the detection of diabetic
retinopathy, as they allow the system to analyze the structural and vascular changes
61
in the retina caused by the disease.

62
5.2.2 Image processing

The acquired retinal images often contain noise, illumination variations, or

low contrast, which can hinder the detection of important features. In the
preprocessing stage, several techniques are applied to enhance image quality. These
may include:

 Noise removal: Filters are applied to remove random variations or distortions

in the image.
 Normalization: Adjusting the brightness or contrast to make important retinal
features stand out.
 Smoothing: Techniques to reduce pixel-level variations and enhance the
clarity of important structures like blood vessels.

5.2.3 Image Segmentation

This stage involves segmenting the image into different regions of interest.
For diabetic retinopathy detection, the primary focus is on segmenting retinal
structures such as:

 Blood vessels: To identify abnormal growth or leakage.

 Optic disc and macula: To detect swelling or other abnormalities.
Segmentation helps isolate these regions for more detailed analysis in the
subsequent stages.

5.2.4 Image Enhancement

After segmentation, image enhancement techniques are applied to further

clarify and highlight important retinal features. This could involve improving the
contrast of specific structures such as microaneurysms, exudates, hemorrhages, or
63
neovascularization, all of which are indicative of diabetic retinopathy. Enhancement

64
makes these features more easily detectable for analysis.

5.2.5 Extraction of Retinal Parameters

In this step, important parameters and features are extracted from the
enhanced and segmented image. These features may include:

 Blood vessel density and branching.

 Presence of microaneurysms, hemorrhages, and exudates.
 Texture and shape of retinal structures. These parameters are crucial for
identifying early and advanced stages of diabetic retinopathy.

5.2.6 Abnormal Feature Extraction

This stage focuses on identifying abnormal features that are directly linked to
diabetic retinopathy. Key abnormalities include:

 Microaneurysms: Small bulges in blood vessels that are one of the earliest
signs of DR.
 Hemorrhages: Blood leaking into the retina.
 Exudates: Fatty deposits from damaged blood vessels.
 Neovascularization: Growth of new, fragile blood vessels in later stages of
the disease. Extracting these features helps in determining the severity and
progression of the disease.

5.2.7 DR Disease Severity Analysis

Based on the extracted abnormal features, a disease severity analysis is

performed. The system evaluates the severity of diabetic retinopathy and classifies it
into one of the following categories:

 No DR Detected (Class 0): No visible signs of DR in the retinal images.

65
  Mild NPDR (Class 1): Presence of a few microaneurysms and possibly small
hemorrhages.
 Moderate NPDR (Class 2): Increased presence of hemorrhages and
microaneurysms, with possible early exudate formation.
 Severe NPDR (Class 3): Numerous hemorrhages, cotton wool spots, and
early neovascularization. The patient is at high risk of developing
proliferative DR.
 Proliferative DR (Class 4): Advanced stage of the disease where abnormal
blood vessels grow on the retina's surface, leading to the risk of retinal
detachment or vision loss.

5.2.8 Classification of DR Stages

The severity analysis leads to the final classification of the disease into different
stages:

 No DR (Class 0): The retina is healthy, with no signs of DR.

 Mild NPDR (Class 1): Early-stage DR with minimal abnormalities.
 Moderate NPDR (Class 2): Intermediate stage with more significant blood
vessel damage.
 Severe NPDR (Class 3): Advanced stage with a high risk of progression to
proliferative DR.
 Proliferative DR (Class 4): The most severe stage, characterized by the
growth of new blood vessels and the potential for vision loss.

66
 CHAPTER 6
IMPLEMENTATION AND RESULT

6.1 IMPLEMENTATION

This allows the user to select a file from their system. The dialog lists
directories, files, and their respective metadata (name, type, size, and modification
date). Once a file is selected, its path is typically processed by the code for further
actions such as loading or analyzing the file.

67
 The output displays the result of a diabetic retinopathy detection model. An input retinal
image was preprocessed and passed to a trained model (64x3-CNN.keras). The model
predicted the condition as "No DR" (No Diabetic Retinopathy), confirming the
absence of diabetic retinopathy in the provided image. The processed image is also
visualized.

6.2 RESULT ANALYSIS

The output shows a classification report evaluating a machine learning model's

performance. It includes precision, recall, F1-score, and support for each class. The
model achieved an overall accuracy of 94.1%, but performance varies across classes,
with some having low precision and recall.

68
69
 CHAPTER 7

CONCLUSION AND FUTURE WORK

7.1CONCLUSION

In conclusion, this project on the detection of diabetic retinopathy leverages

cutting-edge techniques such as Explainable AI (XAI) with Grad-CAM, combined
with multi-modal image fusion, to enhance both the accuracy and interpretability of
the diagnosis process. The multi-modal data fusion integrates diverse sources, such
as retinal images and patient medical history, providing a comprehensive analysis
that goes beyond just image-based features. The hybrid CNN-Transformer model
used in the system not only captures intricate local patterns, such as microaneurysms
or hemorrhages, but also considers the global context of the retinal image,
improving the robustness of predictions. Grad-CAM visually highlights the critical
regions of the retina contributing to a particular diagnosis, giving clinicians a clear
visual explanation. This explainable component is essential in building trust in AI-
driven systems, as they allow medical professionals to understand the "why" behind
each prediction. Furthermore, the system’s capability to deliver real-time inferences
on edge devices, such as smartphones, makes it highly practical for deployment in
various healthcare settings, especially in remote areas with limited access to medical
specialists. This blend of advanced AI techniques and practical deployment
solutions significantly improves early detection and diagnosis of diabetic
retinopathy, ultimately aiding in timely treatment and better patient outcomes.

70
7.2 FUTURE ENHANCEMENT
7.2.1 Integrating OCT scans, genetic information, and blood biomarkers
alongside retinal images offers a more holistic view of the patient’s
condition. This multimodal approach enhances the accuracy of diabetic
retinopathy detection and diagnosis.

7.2.2 Moving beyond Grad-CAM, methods like Layer-wise Relevance

Propagation (LRP) and attention-based visual explanations can provide
deeper insights into how models arrive at decisions, further boosting
interpretability.

7.2.3 Models can be customized for specific patient groups (age, ethnicity, or
diabetic history), leading to more accurate and individualized
predictions. This adaptation allows the AI to provide tailored diagnoses
based on patient-specific data.

7.2.4 By using federated learning, models can be trained on decentralized

devices without sharing sensitive patient data with a central server. This
ensures data privacy while enabling large-scale data usage for model
improvement.

7.2.5 Future systems can incorporate continuous learning, where models

dynamically update with new data and patterns. This would enable the
system to adapt to changing diagnostic criteria or new imaging
technologies over time.

7.2.6 Combining edge computing for real-time analysis and cloud resources
for heavy computations improves the system's performance on resource-
constrained devices like smartphones, making it more efficient and
scalable.
71
 APPENDIX

A. READ ME FILE

A.1 ABOUT THE PROJECT

The project "Detection of Diabetic Retinopathy Using Grad-CAM, Explainable AI,

and Multi-Modal Image Fusion" harnesses the power of deep learning and
advanced interpretability techniques to enhance the detection of diabetic
retinopathy from retinal images. By employing multi-modal image fusion, the
system integrates different data sources for a more comprehensive analysis,
improving diagnostic accuracy. Additionally, Explainable AI method like Grad-
CAM provides clear visual and interpretive insights into the model’s decision-
making process, helping medical professionals understand why a certain
diagnosis was made. This not only boosts confidence in the AI-generated results
but also ensures timely communication between doctors and patients, enabling
earlier detection and treatment of diabetic retinopathy. Overall, the system offers
a robust, transparent, and efficient approach to diagnosing this vision-
threatening condition.

A.2 DEVELOPER

AKASH VISHNU R [email protected]

HARI PRASANNA M [email protected]

RAVEE RAGHUL C [email protected]

72
 SOURCE CODE

import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, confusion_matrix, accuracy_score

import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.layers import Input, Dense, GlobalAveragePooling2D, Dropout,
BatchNormalization
from tensorflow.keras.models import Model
from tensorflow.keras.applications import ResNet50, EfficientNetB0
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau, ModelCheckpoint
import cv2
from ipywidgets import FileUpload, VBox, Label
from IPython.display import display

import tensorflow_addons as tfa

# Set dataset paths

train_csv_path = '/path_to_dataset/train_csv/train.csv' # Update this path
images_dir = '/path_to_dataset/train_images/' # Update this path

# Load the CSV file

df = pd.read_csv(train_csv_path)

# Mapping diagnosis labels to stages

stage_mapping = {
0: 'No DR',
1: 'Mild',
2: 'Moderate',
3: 'Severe',
4: 'Proliferative DR'
}
df['stage'] = df['diagnosis'].map(stage_mapping)
73
# Split into training and validation sets
train_df, val_df = train_test_split(df, test_size=0.2, stratify=df['diagnosis'], random_state=42)

print(f"Training samples: {len(train_df)}")

print(f"Validation samples: {len(val_df)}")

# Image size and parameters

IMAGE_SIZE = (224, 224)
BATCH_SIZE = 32
NUM_CLASSES = 5

# Training data generator with augmentation

train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=15,
width_shift_range=0.1,
height_shift_range=0.1,
zoom_range=0.1,
horizontal_flip=True,
fill_mode='nearest'
)

# Validation data generator (no augmentation)

val_datagen = ImageDataGenerator(rescale=1./255)

# Flow from dataframe

train_generator = train_datagen.flow_from_dataframe(
train_df,
directory=images_dir,
x_col='id_code',
y_col='diagnosis',
target_size=IMAGE_SIZE,
batch_size=BATCH_SIZE,
class_mode='categorical'
)

val_generator = val_datagen.flow_from_dataframe(
val_df,
directory=images_dir,
x_col='id_code',
74
 y_col='diagnosis',
target_size=IMAGE_SIZE,
batch_size=BATCH_SIZE,
class_mode='categorical',
shuffle=False
)

# ResNet50 base model

resnet_base = ResNet50(weights='imagenet', include_top=False, input_shape=(*IMAGE_SIZE,
3))
resnet_base.trainable = False # Freeze the base model

# Add custom layers

resnet_x = resnet_base.output
resnet_x = GlobalAveragePooling2D()(resnet_x)
resnet_x = Dense(512, activation='relu')(resnet_x)
resnet_x = BatchNormalization()(resnet_x)
resnet_x = Dropout(0.5)(resnet_x)
resnet_predictions = Dense(NUM_CLASSES, activation='softmax')(resnet_x)

resnet_model = Model(inputs=resnet_base.input, outputs=resnet_predictions)

# EfficientNetB0 base model

efficientnet_base = EfficientNetB0(weights='imagenet', include_top=False,
input_shape=(*IMAGE_SIZE, 3))
efficientnet_base.trainable = False # Freeze the base model

# Add custom layers

efficientnet_x = efficientnet_base.output
efficientnet_x = GlobalAveragePooling2D()(efficientnet_x)
efficientnet_x = Dense(512, activation='relu')(efficientnet_x)
efficientnet_x = BatchNormalization()(efficientnet_x)
efficientnet_x = Dropout(0.5)(efficientnet_x)
efficientnet_predictions = Dense(NUM_CLASSES, activation='softmax')(efficientnet_x)

efficientnet_model = Model(inputs=efficientnet_base.input, outputs=efficientnet_predictions)

# Combine the two models

combined_input = Input(shape=(*IMAGE_SIZE, 3))

# Pass through ResNet50

75
resnet_features = resnet_model(combined_input)

# Pass through EfficientNetB0

efficientnet_features = efficientnet_model(combined_input)

# Combine features
combined = tf.keras.layers.concatenate([resnet_features, efficientnet_features])

# Add further dense layers

combined = Dense(512, activation='relu')(combined)
combined = BatchNormalization()(combined)
combined = Dropout(0.5)(combined)
combined = Dense(256, activation='relu')(combined)
combined = BatchNormalization()(combined)
combined = Dropout(0.5)(combined)

# Final output
final_output = Dense(NUM_CLASSES, activation='softmax')(combined)

# Define the ensemble model

ensemble_model = Model(inputs=combined_input, outputs=final_output)

# Compile the ensemble model

ensemble_model.compile(
optimizer=Adam(lr=1e-3),
loss='categorical_crossentropy',
metrics=['accuracy']
)

# Set callbacks
early_stop = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=5, min_lr=1e-6)
checkpoint = ModelCheckpoint('ensemble_model.h5', monitor='val_loss', save_best_only=True)

# Calculate steps
steps_per_epoch = train_generator.samples // BATCH_SIZE
validation_steps = val_generator.samples // BATCH_SIZE

# Train the model

history = ensemble_model.fit(
train_generator,
76
 steps_per_epoch=steps_per_epoch,
epochs=100,
validation_data=val_generator,
validation_steps=validation_steps,
callbacks=[early_stop, reduce_lr, checkpoint]
)

# Load the best saved model

ensemble_model.load_weights('ensemble_model.h5')

# Evaluate on validation set

val_generator.reset()
preds = ensemble_model.predict(val_generator, steps=validation_steps, verbose=1)
pred_classes = np.argmax(preds, axis=1)
true_classes = val_generator.classes[:len(pred_classes)]

# Classification Report
print("Classification Report:")
print(classification_report(true_classes, pred_classes, target_names=stage_mapping.values()))

# Confusion Matrix
cm = confusion_matrix(true_classes, pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm, annot=True, fmt='d', xticklabels=stage_mapping.values(),
yticklabels=stage_mapping.values(), cmap='Blues')
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.title('Confusion Matrix')
plt.show()

# Overall Accuracy
accuracy = accuracy_score(true_classes, pred_classes)
print(f"Validation Accuracy: {accuracy * 100:.2f}%")

# Function to generate Grad-CAM

def get_gradcam(model, img_array, layer_name=None):
import tensorflow.keras.backend as K
from tensorflow.keras.models import Model

if layer_name is None:
# Find last convolutional layer
77
 for layer in reversed(model.layers):
if len(layer.output_shape) == 4:
layer_name = layer.name
break
if layer_name is None:
raise ValueError("No convolutional layer found in the model.")

grad_model = Model(
[model.inputs],
[model.get_layer(layer_name).output, model.output]
)

with tf.GradientTape() as tape:

conv_outputs, predictions = grad_model(img_array)
loss = predictions[:, np.argmax(predictions[0])]

grads = tape.gradient(loss, conv_outputs)

pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2))
conv_outputs = conv_outputs[0]

conv_outputs *= pooled_grads
heatmap = tf.reduce_sum(conv_outputs, axis=-1)

heatmap = np.maximum(heatmap, 0) / np.max(heatmap)

return heatmap.numpy()

# Function to display Grad-CAM

def display_gradcam(img_path, model, preprocess_input, layer_name=None):
img = cv2.imread(img_path)
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img_resized = cv2.resize(img, IMAGE_SIZE)
img_array = np.expand_dims(img_resized, axis=0) / 255.0

heatmap = get_gradcam(model, img_array, layer_name)

# Resize heatmap to image size

heatmap = cv2.resize(heatmap, IMAGE_SIZE)
heatmap = np.uint8(255 * heatmap)
heatmap = cv2.applyColorMap(heatmap, cv2.COLORMAP_JET)

superimposed_img = heatmap * 0.4 + img_resized

78
 plt.figure(figsize=(10,10))
plt.subplot(1,2,1)
plt.imshow(img_resized)
plt.title('Original Image')
plt.axis('off')
plt.subplot(1,2,2)
plt.imshow(superimposed_img.astype('uint8'))
plt.title('Grad-CAM')
plt.axis('off')
plt.show()

# Example usage of Grad-CAM

display_gradcam('/path_to_dataset/train_images/id_code.png', ensemble_model, None)

# Prediction function
def predict_stage(img_path, model):
img = Image.open(img_path)
img = img.resize(IMAGE_SIZE)
img_array = np.array(img) / 255.0
img_array = np.expand_dims(img_array, axis=0)

preds = model.predict(img_array)
pred_class = np.argmax(preds, axis=1)[0]
pred_prob = preds[0][pred_class]
stage = stage_mapping[pred_class]

return stage, pred_prob, img_array

# Display function
def display_prediction(file):
if file is not None:
# Read the image file
content = file['content']
img = Image.open(io.BytesIO(content))
img_rgb = img.convert('RGB')
img_resized = img_rgb.resize(IMAGE_SIZE)
img_array = np.array(img_resized) / 255.0
img_input = np.expand_dims(img_array, axis=0)

# Predict
preds = ensemble_model.predict(img_input)
79
 pred_class = np.argmax(preds, axis=1)[0]
pred_prob = preds[0][pred_class]
stage = stage_mapping[pred_class]

# Display the image

plt.figure(figsize=(6,6))
plt.imshow(img_resized)
plt.axis('off')
plt.title(f'Predicted Stage: {stage} ({pred_prob*100:.2f}%)')
plt.show()

# Display Grad-CAM
img_bytes = io.BytesIO()
img_rgb.save(img_bytes, format='PNG')
img_array_for_gradcam = np.expand_dims(np.array(img_rgb.resize(IMAGE_SIZE)),
axis=0) / 255.0
heatmap = get_gradcam(ensemble_model, img_array_for_gradcam)

# Resize heatmap to image size

heatmap = cv2.resize(heatmap, IMAGE_SIZE)
heatmap = np.uint8(255 * heatmap)
heatmap = cv2.applyColorMap(heatmap, cv2.COLORMAP_JET)

superimposed_img = heatmap * 0.4 + np.array(img_resized)

# Display Grad-CAM
plt.figure(figsize=(6,6))
plt.imshow(superimposed_img.astype('uint8'))
plt.axis('off')
plt.title('Grad-CAM')
plt.show()

# File upload widget

upload = FileUpload(
accept='image/*',
multiple=False
)

output = Label(value="")

def on_upload_change(change):
80
 for name, file in upload.value.items():
display_prediction(file)

upload.observe(on_upload_change, names='value')
display(VBox([Label("Upload an eye image for Diabetic Retinopathy Detection:"), upload]))

81
REFERENCES

2 3 4
[1] 1Inamullah, Saima Hassan, Nabil A. Alrajeh, Emad A. Mohammed,
5
Shafiullah Khan, Data Diversity in Convolutional Neural Network Based
Ensemble Model for Diabetic Retinopathy, Journal of Biomimetics, Vol 8, 2023

[2] 1Ghulam Ali, 2Aqsa Dastgir, 3Muhammad Waseem Iqbal, 4Muhammad Anwar,
5
Muhammad Faheem, A Hybrid Convolutional Neural Network Model for
Automatic Diabetic Retinopathy Classification From Fundus Images, IEEE Journal
of Translational Engineering in Health and Medicine, Vol 11, 2023

[3] 1Ashish Bora, 2Richa Tiwari, 3Pinal Bavishi, 4Sunny Virmani, 5Rayman
Huang, 6Ilana Traynis, 7Greg S. Corrado, 8Lily Peng, 9Dale R. Webster, 10Avinash
11 12 13
V. Varadarajan, Warisara Pattanapongpaiboon, Reena Chopra, Paisan
Ruamviboonsuk, Risk Stratification for Diabetic Retinopathy Screening Order
Using Deep Learning: A Multicenter Prospective Study, Journal of Translational
Vision Science and Technology, Vol 12, 2023

[4] 1Fangsheng Chen, 2Shaodong Ma, 3Jinkui Hao, 4Mengting Liu, 5Yuanyuan Gu,
6
Quanyong Yi, 7Jiong Zhang, 8Yitian Zhao, Dual-Path and Multi-Scale Enhanced
Attention Network for Retinal Diseases Classification Using Ultra-Wide-Field
Images, IEEE Access, Vol 11, 2023

[5] 1Sanli Yi, 2Lingxiang Zhou, 3Lei Ma, 4Dangguo Shao, MTRA-CNN: A Multi-
Scale Transfer Learning Framework for Glaucoma Classification in Retinal Fundus
Image, IEEE Access, Vol 11, 2023

82
[6] 1S. Rama Krishna, 2Naresh Cherukuri, 3Y. Dileep Kumar, 4R. Jayakarthik, 5B.
Nagarajan, 6Allam Balaram, 7G. Divya Jyothi, Convolutional Neural Networks for
Automated Diagnosis of Diabetic Retinopathy in Fundus Images, Journal of
Artificial Intelligence and Technology, Vol 3, 2023

2 3
[7] 1Sheena Christabel Pravin, Sindhu Priya Kanaga Sabapathy, Suganthi
Selvakumar, 4Saranya Jayaraman, 5Selvakumar Varadharajan Subramani, An
Efficient DenseNet for Diabetic Retinopathy Screening, International Journal of
Engineering and Technology Innovation, Vol 13, 2023

[8] 1Md. Nahiduzzaman, 2Md. Robiul Islam, 3Md. Omaer Faruq Goni, 4Md.
Shamim Anower, 5Mominul Ahsan, 6Julfikar Haider, 7Marcin Kowalski, Diabetic
retinopathy identification using parallel convolutional neural network based feature
extractor and ELM classifier, Elsevier - Expert Systems With Applications, Vol
217, 2023

[9] 1Maofa Wang, 2Qizhou Gong, 3Quan Wan, 4Zhixiong Leng, 5Yanlin Xu,
6
Bingchen Yan, 7He Zhang, 8Hongliang Huang, 9Shaohua Sun, A fast interpretable
adaptive meta-learning enhanced deep learning framework for diagnosis of diabetic
retinopathy, Elsevier - Expert Systems with Applications, Vol 244, 2023

[10] Ching-Yu Wang, 2Arvind Mukundan, 3Yu-Sin Liu, 4Yu-Ming Tsao, 5Fen-
1

Chi Lin, 6Wen-Shuang Fan, 7Hsiang-Chen Wang, Optical Identification of Diabetic

Retinopathy Using Hyperspectral Imaging, Journal of Personalized Medicine, Vol
13, 2023

83