Real-Time Object Recognition with Voice Feedback

for Visually Impaired Based on Raspberry Pi

D Vijendra Kumar Venkata Ramana Kammampati B.Tapasvi
Department of ECE Department of ECE Department of Electronics and
Godavari Institute of Engineering & Aditya College of Engineering And Communication Engineering
Technology Technology SRKR Engineering College
Rajahmundry, INDIA Surampalem, INDIA Bhimavaram, INDIA
[email protected] [email protected] [email protected]

K Indira Priyadarsini Eswara Prasad Konakalla Hima Kiran Killamsetti

Department of ECE Department of Physics & Electronics Department of Electronics and
DNR College of Engineering and B V Raju College Communication Engineering
Technology Andhra Pradesh, West Godavari, Vishnu Institute of Technology
Bhimavaram, INDIA Bhimavaram, INDIA Bhimavaram, INDIA
[email protected] [email protected] [email protected]

Abstract— Visually impaired individuals rely on real-time significantly impact their daily lives and overall well-being.
object recognition to overcome the challenges posed by their Some of the basic challenges include: limited access to
limited visual capabilities. Traditional navigation for the
information (may face challenges in accessing printed
visually impaired often involves the use of canes or guide dogs,
but these methods have limitations, particularly when it comes materials, visual content, and other information that is
to identifying specific objects, obstacles, or changes in the primarily presented in a visual format); mobility and
environment. Real-time object recognition technology utilizes navigation (can be challenging due to obstacles, uneven
cameras and advanced algorithms to identify and interpret surfaces, and a lack of clear signage); educational barriers
visual information in real-time, providing auditory or haptic (access to education can be hindered by the unavailability of
feedback to users. This capability enables visually impaired educational materials in accessible formats, as well as a lack
individuals to access instant information about their
surroundings, such as recognizing familiar objects, reading
of awareness and support for inclusive education);
signs, avoiding obstacles, and navigating through complex employment(challenges in workplace accessibility);
environments independently. This technology enhances their technological accessibility(lack of accessible digital content
spatial awareness and allows them to make informed decisions, and user interfaces can be a significant barrier); and
promoting greater autonomy and reducing reliance on external transportation (public transportation may pose challenges,
assistance. Hence, in this work an efficient approach to address including difficulties in locating stops, reading schedules, and
the challenges faced by visually impaired individuals through
ensuring a safe and accessible travel experience).
the development of a real-time object recognition system with
voice feedback has been designed. The proposed system However, the digital era has played a transformative role
leverages the capabilities of Raspberry Pi, integrating computer in enhancing the day-to-day lives of visually impaired
vision techniques to identify objects in the surroundings. In real- individuals, offering innovative solutions to address various
time, the system processes live video input from a camera, challenges they encounter [3]. The significance of technology
identifies objects, and provides instant auditory feedback to the in the lives of the visually impaired can be observed across
user through a voice interface. The proposed system accuracy several domains such as: assistive devices; voice command
and efficiency are evaluated through comprehensive testing, systems; navigation apps; object recognition; braille
demonstrating its potential as an assistive technology to enhance technology; educational tools; and wearable devices. The
the liberation and security of visually impaired users in home
significance of technology in the lives of visually impaired
environments.
individuals lies in its ability to break down barriers, promote
Keywords— Real-Time Object Recognition, Voice Feedback, independence, and enhance overall quality of life [4].
Visually Impaired, Raspberry Pi, Computer Vision, Assistive Despite advancements in technology, visually impaired
Technology, Accessibility, Object Identification, Auditory individuals may still face specific challenges in a home
Interface, Portable System. environment when performing their daily activities
(orientation and mobility, identifying objects and items, using
I. INTRODUCTION appliances and electronics, safety concerns etc). To address
In a world predominantly shaped by visual information and these challenges, a combination of assistive technologies,
cues, the experiences of visually impaired individuals offer a home modifications, and awareness about accessible design
unique perspective on navigating the intricate tapestry of principles is crucial [5]. Creating an environment that is
daily life. The challenges faced by individuals with visual organized, labeled, and equipped with tactile or auditory cues
impairments extend beyond the absence of sight; they can significantly enhance the independence and quality of life
encompass a complex interplay of societal, technological, for visually impaired individuals within their homes. Hence,
and environmental factors. There are an estimated 4.95 in this work a system has been developed which can identify
million blind people and 70 million people with vision the objects in a quick time and gives the voice output of
impairment living in India, according to [1]. These recognized object based on Raspberry Pi. The proposed
individuals face a range of challenges [2] that can system is best suitable at home environment.
 II. LITERATURE REVIEW B. Circuit Implementation
The integration of object recognition with vocal sound The circuit implementation of proposed system has been
response, specifically utilizing the raspberry pi platform, created and depicted in Fig.2.
represents a cutting-edge approach to addressing the unique
challenges faced by this user group. This literature review
explores key aspects of this technology, including computer
vision advancements, the role of raspberry pi in assistive
technology, and the significance of voice feedback systems
in enhancing the daily lives of visually impaired individuals.
The authors in [6], has developed an algorithm for accurate
object recognition. A system has been developed in [7], that
Fig. 2. Circuit implementation
is aiming to empower impaired individuals by supporting
them in identifying and navigating their surroundings. The C. Developed Hardware Prototype
aim of the authors in [8], is to transform the visual
environment into an auditory realm, providing blind The developed prototype (see Fig.3), has been tested for
individuals with notifications about objects in their path. variety of house hold needs as provided in the dataset.
The study in [9], employs the YOLO V4 algorithm in
conjunction with pyttsx3 to detect and vocalize object labels
in images or videos. The research in [10], introduced a novel
system aimed at creating a third-eye hand glove object
detection solution for individuals with visual impairments.
The objective of authors in [11], is to devise and deploy a
portable assisting system in perceiving their surroundings,
including objects and people, while accurately estimating
distances. In [12], a groundbreaking framework was
introduced, for enhancing assistive mobility. In [13], the
authors presented a pioneering approach for mobility issue. A
system for instantaneous object detection utilizing deep
learning techniques has been proposed in [14]. A tool has been
developed in [15], to assist surroundings. Hence, the proposed
work is motivated from the available literature, and developed Fig. 3. Develpoed hardware prototype
a system based on TTS (text-to-speech) synthesis and
Raspberry Pi in order to identify the objects with voice D. Dataset
feedback.
A dataset has been created for the training data in order to
III. SYSTEM DESIGN identify the objects. The created dataset consists of 66 types
of objects, including human objects. The proposed system
A. Proposed Block Diagram accurately identifies the daily need in the home with voice
The proposed block diagram for real time object recognition output for visually impaired. Table 1, summarized the list of
has been depicted in Fig.1. It mainly consists of Pi camera, objects present in the data base for reorganization.
Raspberry Pi, audio device. The programming has been done Table 1. Trained dataset
through open CV. Person Hair Drier Cow
Bicycle Handbag Elephant
Pi Raspberry Car Stop Sign Bear
Object Pi Airplane Bench Giraffe
Camera
Bus Bird Hat
Train Cat Backpack
Truck Dog Umbrella
Boat Horse Shoe
Open CV Tv Sheep Eye Glasses
Audio pyttsx3
Laptop Knife Cake
Output Text to
Mouse Spoon Chair
voice
Remote Bowl Couch
Keyboard Banana Toothbrush
Bottle Apple Bed
Object Plate Sandwich Mirror
Database Refrigerator Orange Toaster
Cup Broccoli Window
Fig. 1. Proposed syetem block diagram
Fork Carrot Desk
Blender Sink Vase
Book Pizza Scissors
Clock Donut Teddy Bear
E. Flow Diagram
A flow chart (see Fig.4) has been created for understanding
the detailed operation of proposed system.

Start

Start Video Frame

Acquire images using Pi Camera

Object Detection using

Pretrained Dataset

Fig. 5. Object identifed (Keyboard) with 76.05% accuracy.

Is object
detected
?

Classify Detected object

based on dataset

Text to Voice Conversion

Fig. 6. Object identifed (mouse) with 76.05% accuracy

It is observed form the Fig.7 and Fig. 8, the proposed system

identifies multiple objects at a time but the accuracy of
Audio Output recognition is decreased.

Stop

Fig. 4. Flow chart

IV. EXPERIMENTAL RESULTS

The programming has been done through open CV, and it
uses pyttsx3 voice library for text to speech conversion and
tensor flow for image training and detection. The proposed
system is accurately identifying the single object (see Fig.5,
Fig.6) and multiple objects (see Fig.7 and Fig.8) with voice
feedback to the visually impaired. It is clearly depicted form
Fig.5; the proposed system accuracy is 69.53 when the person
is identified. Whereas, 76.05 and 73.49 percent of accuracy
achieved when it recognizes the objects like mouse and
Fig. 7. Multple object recogniation (TV and Chair)
keyboard respectively.
 Once (YOLO) v4-Tiny Algorithm," in 2022 IEEE International
Conference on Artificial Intelligence in Engineering and Technology
(IICAIET), Kota Kinabalu, Malaysia, 2022, pp. 1-6.
[11] A. Aljarf, G. Almaghrabi, H. Albarakati, H. Ahmed, R. Alharbi, and S.
Aljuhani, "EBSAR: Detecting of Objects that Hinder Visually
Impaired in a Controlled Area Using Deep Learning," in 2022 Fifth
National Conference of Saudi Computers Colleges (NCCC), Makkah,
Saudi Arabia, 2022, pp. 19-25.
[12] V. K. Paswan and A. Choudhary, "Camera Based Indoor Object
Detection and Distance Estimation Framework for Assistive Mobility,"
in 2022 IEEE International Conference on Service Operations and
Logistics, and Informatics (SOLI), Delhi, India, 2022, pp. 1-6.
[13] L. Bougheloum, M. B. Salah, and M. Bettayeb, "Real-time obstacle
detection for visually impaired people using deep learning," in 2023
6th International Conference on Signal Processing and Information
Security (ICSPIS), Dubai, United Arab Emirates, 2023, pp. 51-56.
[14] K. Sharma and P. Syal, "Real Time Object Detection for Assisting
Fig. 8. Multiple objects recogniation (person with mobile). Visually Impaired People," in 2022 OPJU International Technology
Conference on Emerging Technologies for Sustainable Development
V. CONCLUSION (OTCON), Raigarh, Chhattisgarh, India, 2023, pp. 1-6.
[15] S. Alagarsamy, T. D. Rajkumar, K. P. L. Syamala, C. S. Niharika, D.
The proposed system leverages the capabilities of Raspberry U. Rani, and K. Balaji, "An Real-Time Object Detection Method for
Pi, integrating computer vision techniques to identify objects Visually Impaired Using Machine Learning," in 2023 International
in the surroundings. In real-time, the system processes live Conference on Computer Communication and Informatics (ICCCI),
video input from a camera, identifies objects, and provides Coimbatore, India, 2023, pp. 1-6.
instant auditory feedback to the user through a voice
interface. The proposed system accuracy and efficiency are
evaluated through comprehensive testing, demonstrating its
potential as an assistive technology to improve the liberation
and security of users in home environments. It is evidently
depicted form experimental results; the proposed system is
best suitable in indoor environments with high accuracy in
recognizing objects.

REFERENCES
[1] S. Mannava, R. R. Borah, and B. R. Shamanna, "Current estimates of
the economic burden of blindness and visual impairment in India: A
cost of illness study," Indian Journal of Ophthalmology, vol. 70, no. 6,
pp. 2141-2145, 2022.
[2] K. U. Panchal, D. C. Khara, T. J. Gari, and V. Chavan, "Companion:
Easy Navigation App for Visually Impaired Persons," in 2021
International Conference on Intelligent Technologies (CONIT), Hubli,
India, 2021, pp. 1-6.
[3] K. M. Mulyono, T. Budi Santoso, and R. W. Sudibyo, "Design and
Implementation of Real-time Object Detection for Blind using
Convolutional Neural Network," in 2022 International Electronics
Symposium (IES), Surabaya, Indonesia, 2022, pp. 554-558.
[4] S. Cherian and C. Singh, "Real Time Implementation of Object
Tracking Through Webcam," International Journal of Research in
Engineering and Technology, vol. 3, no.1, pp. 128-132, 2014.
[5] A. Bhandari et al., "Object detection and recognition: using deep
learning to assist the visually impaired," Disability and Rehabilitation:
Assistive Technology, vol. 16, no. 3, pp. 280-288, 2021.
[6] R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich Feature
Hierarchies for Accurate Object Detection and Semantic
Segmentation," in 2014 IEEE Conference on Computer Vision and
Pattern Recognition, Columbus, OH, USA, 2014, pp. 580-587.
[7] A. R. Jambhulkar, A. R. Gajera, C. M. Bhavsar, and S. Vatkar, "Real-
Time Object Detection and Audio Feedback for the Visually
Impaired," in 2023 3rd Asian Conference on Innovation in Technology
(ASIANCON), Ravet IN, India, 2023, pp. 1-5.
[8] S. Vaidya, N. Shah, N. Shah, and R. Shankarmani, "Real-Time Object
Detection for Visually Challenged People," in 2020 4th International
Conference on Intelligent Computing and Control Systems (ICICCS),
Madurai, India, 2020, pp. 311-316.
[9] T. Bindamrutha, A. Likhitha, S. A. Reddy, G. V. Reddy, and S. S.
Priya, "A Real-time Object Detection System for the Visually
Impaired," in 2023 2nd International Conference on Vision Towards
Emerging Trends in Communication and Networking Technologies
(ViTECoN), Vellore, India, 2023, pp. 1-5.
[10] J. P. Docto, A. I. Labininay, and J. F. Villaverde, "Third Eye Hand
Glove Object Detection for Visually Impaired using You Only Look