Table of Contents

1. Abstract....................................................................................3
2. Introduction and Background........................................................3
3. Literature Review......................................................................3
4. Problem Statement....................................................................4

1|Page
 Table of Content
1. Abstract 3
2. Introduction and Background 3
3. Literature Review 4
4. Problem Statement 4
5. Research Statement 5
6. Project Goal and Objectives 6
7. Scope 6
8. Limitations 7
9. Targeted Users 7
10. Phases of Project 7
11. Final Project Output 8
12. Project Deliverables 8
13. Block Level Diagram 9
14. Tools and Technologies 10
15. Methodology Justification 10
16. Timeline 11
17. References 12

2|Page
1. Abstract
SignLink is a proposed mobile application aimed at bridging the persistent communication gap
between the deaf/mute community and the hearing majority. The solution leverages advancements
in Artificial Intelligence (AI) and computer vision to develop a two-way translation system that is
practical, cost-effective, and widely accessible. The application enables deaf users to perform sign
language gestures in front of a standard smartphone camera, which are then translated into audible
speech. Conversely, spoken words are converted into text
displayed on the screen, facilitating seamless
communication between signers and non-signers.

Unlike existing hardware-dependent solutions such as

sensor gloves or motion-capture devices, which are often
Figure 1: Communication Bridge costly and impractical for everyday use (Biz4Group, 2024;
AI Competence Center, 2024), SignLink relies solely on ubiquitous smartphone technology,
ensuring affordability and accessibility for a wide range of users.

Beyond translation, the app is envisioned to include a learning module for both deaf and hearing
individuals as well as offline functionality, promoting independence, social inclusion, and
everyday usability (Arm Community, 2024; Wired UK, 2024).

2. Introduction and Background

Globally, more than 70 million people rely on sign language as their primary means of
communication, yet interaction with non-signers remains a major challenge, restricting access to
education, employment, healthcare, and social integration ([Link], 2024; Nimdzi, 2024).
Existing solutions often depend on human interpreters, which are costly, not always available, and
raise privacy concerns ([Link], 2024; Nimdzi, 2024). Similarly, many technological approaches
rely on specialized hardware such as depth sensors or wearables, which limits widespread adoption
(Biz4Group, 2024; Arm Community, 2024).

With the rapid advancement of high-performance smartphones equipped with advanced cameras
and processors, the integration of lightweight AI models directly on mobile devices has become
feasible. For example, recent efforts such as Silence Speaks are exploring AI-driven signing
avatars to translate text into British and American Sign Language in real time, even capturing
emotional nuance (Wired UK, 2024).

Building on this opportunity, SignLink aims to provide a portable, real-time, and reliable
communication bridge, empowering the deaf/mute community to actively participate in all aspects
of daily life.

3|Page
3. Literature Review
In recent years, Artificial Intelligence (AI) has significantly advanced the fields of image
processing and gesture recognition, notably through Convolutional Neural Networks (CNNs) and
Transfer Learning methods. Pre-trained models such as MobileNet and EfficientNet have enabled
high-accuracy recognition while maintaining efficiency on limited data and device resources
(Deshpande et al., 2022).

Several initiatives illustrate AI’s real-world potential in this domain. Google’s Project Relate
enables users to train personalized speech recognition models using their own voice samples, even
allowing offline transcription functionality for those with impaired speech (Google AI, 2024).

Despite these breakthroughs, most of these projects still rely on cloud-based processing, which
requires sustained internet access, or platform-specific implementations, limiting universal reach.
Furthermore, they often focus on major sign languages like ASL, while regional languages such
as PSL remain underrepresented.

In Pakistan, significant strides have recently been made. A deep learning–based web app using
MediaPipe achieved near-perfect accuracy for both static and dynamic Pakistani Sign Language
(PSL) gestures, demonstrating the feasibility of lightweight, highly accurate recognition models
deployable on mobile devices (Amjad et al., 2024). Another development in real-time PSL
recognition has been published in Engineering Applications of Artificial Intelligence (Ahmad et
al., 2024). The Pakistani government has also taken notice - by mid-2025, initiatives offering AI-
driven sign language interpretation tools were showcased in high-level policy discussions (MoITT,
2025).

The proposed SignLink system builds on these advances by emphasizing on-device (edge)
processing, ensuring privacy, low latency, and offline usability. Empowered by efficient models
and developed as a cross-platform Flutter mobile application, SignLink is designed to be cost-
effective, scalable, and uniquely tailored to cater to regional sign languages like PSL.

4. Problem Statement
The central challenge addressed by this project is the communication gap between the deaf/mute
community and individuals unfamiliar with sign language. This gap often results in the social,
economic, and personal isolation of millions of people worldwide who rely on sign language as
their primary mode of communication.
The consequences of this barrier are significant:

4|Page
  Social Exclusion: Deaf and mute individuals frequently face difficulties in building
relationships, participating in community events, and engaging in everyday interactions,
leading to feelings of isolation.
 Economic Disadvantage: Limited access to quality education and employment
opportunities restricts their economic growth and independence.
 Accessibility Issues: Essential services such as healthcare, emergency response, legal
systems, and government offices often remain inaccessible without interpreters.
 Dependence on Interpreters: Relying on human interpreters for basic communication not
only creates dependency but can also compromise privacy and spontaneity in
conversations.

5. Research Statement
This research project seeks to investigate the feasibility and effectiveness of developing a real-
time, two-way sign language translation mobile application for the Pakistani Sign Language (PSL)
context, using a purely software-based approach.

Core Research Question:

“Can a lightweight Convolutional Neural Network (CNN) model, deployed on a standard

smartphone via TensorFlow Lite, accurately recognize a curated set of static Pakistani Sign
Language gestures in real-time and facilitate seamless bidirectional communication between
deaf/mute and hearing individuals?”
Hypotheses:
H1: A CNN model (e.g., based on a MobileNetV2 architecture) can be trained to achieve a
classification accuracy of over 90% on a custom dataset of 20-30 static PSL signs.

H2: The computational latency of the optimized TFLite model on a mid-range Android device will
be low enough (e.g., <200ms per inference) to provide a real-time user experience without
requiring cloud processing.

H3: The integration of on-device speech recognition (Speech-to-Text) and synthesis (Text-to-
Speech) APIs can create a functional and practical communication bridge without external
hardware or constant internet connectivity.

5|Page
 Success Metrics:
The success of this research will be measured by:

 The model's accuracy on a validation set.

 The application's performance metrics (frames-per-second, latency).
 Qualitative feedback from initial user testing with target users.

6. Project Goal and Objectives

Overall Goal:
To develop an accessible mobile application that reduces the communication gap for the deaf and
mute community.

Specific Objectives:

 To design an intuitive and accessible UI/UX for both deaf and hearing users.
 To create a curated dataset of 20–30 PSL signs.
 To train and optimize a CNN model achieving >90% accuracy on live video feeds.
 To integrate the trained TFLite model into a Flutter application.
 To implement on-device Speech-to-Text and Text-to-Speech functionality.
 To conduct user testing and iterate based on feedback.

7. Scope
The initial scope of the SignLink project will be focused and achievable:

 Platform: A minimum viable product (MVP) will be developed for Android. The use of
Flutter will ensure the code base is ready for future iOS expansion.
 Sign Language: The first version will recognize 20–30 basic, static signs from Pakistani
Sign Language (e.g., Hello, Thank You, Help, Water, Yes, No, Eat, Drink).
 Core Features: The MVP will include the two main modes: Gesture-to-Speech and Speech-
to-Text.
 Technical Scope: The ML model will be trained on a custom-curated dataset for the
selected signs. The app will prioritize on-device, offline functionality for core features.

6|Page
8. Limitations
 Sign Complexity: Only static signs will be recognized in the initial version. Dynamic/moving
signs will be considered later.
 Vocabulary Size: Limited to a pre-trained set of 20–30 signs, not full sentences.
 Environmental Factors: Lighting, background clutter, and camera quality may affect
recognition.
 User Variability: Differences in hand shape, size, and signing style may impact accuracy.
 Regional Variations: Model trained on one dialect of PSL may not cover all regional variants.

9. Targeted Users
 Primary Users:
 Deaf and mute individuals using Pakistani Sign Language for daily communication.
 Secondary Users:
 Hearing individuals (family, friends, shopkeepers, healthcare workers).
 Students and enthusiasts interested in learning basic PSL.

10. Phases of Project

1. Research & Planning (2 Weeks): Finalize scope, wireframes, and environments.
2. Data Acquisition & Preprocessing (4 Weeks): Dataset creation (video recording, frame
extraction, labeling, augmentation).
3. Model Development & Training (4 Weeks): Train, evaluate, and convert CNN model to
TFLite.
4. Application Development (6 Weeks): Build Flutter app, integrate ML model, camera, and ML
Kit APIs.
5. Testing & Deployment (2 Weeks): User testing, bug fixing, APK generation.

7|Page
11. Final Project Output
A fully functional Android APK named SignLink featuring:

 Home screen with “Sign to Voice” and “Voice to Text” options.

 Real-time camera interface translating signs into speech.
 Microphone interface transcribing speech into text.
 Basic Practice Section for recognized signs.

Figure 2: App Home Screen /UI

12. Project Deliverables

 Project Proposal Document
 Source Code (Python for ML, Dart/Flutter for App) on GitHub
 Trained TFLite Model File
 Final signed APK for Android
 Complete Project Report and User Manual
 Demonstration Video

8|Page
13. Block Level Diagram

Figure 3: Workflow Design

9|Page
14. Tools and Technologies

Category Technology / Language Purpose

Core Programming Python Developing, training, and exporting the
ML model
Mobile App Flutter (Dart) Building the cross-platform mobile
Framework application
Machine Learning TensorFlow / Keras Building and training the CNN model
ML Deployment TensorFlow Lite Converting the model for mobile
(TFLite) deployment
Computer Vision OpenCV Processing video frames (e.g., cropping,
grayscale)
Speech APIs Google’s ML Kit (on- Offline Speech-to-Text and Text-to-
device) Speech
IDE Android Studio / VS Development environment for Flutter and
Code Python
Version Control Git / GitHub Source code management and
collaboration

15. Methodology Justification

The selection of technologies for SignLink is a deliberate strategy to maximize efficiency,
accessibility, and performance, justified by recent advancements in the field.

The cross-platform Flutter framework was selected over native development (Kotlin/Swift) to
achieve a single codebase for both Android and iOS. This approach significantly reduces
development time and resource requirements while ensuring future scalability and easier
maintenance (Flutter Team, 2023).

TensorFlow Lite (TFLite) is chosen for deploying the Convolutional Neural Network (CNN)
model to enable on-device, offline inference. This is critical for ensuring low-latency real-time
translation, maximizing user privacy by keeping data on the device, and guaranteeing functionality
in environments with limited or no internet connectivity (Abadi et al., 2016).

Employing transfer learning based on the pre-trained MobileNetV2 architecture is essential for
overcoming the constraint of a limited, custom-curated dataset. This methodology allows the
project to leverage features learned from a large-scale dataset (e.g., ImageNet), leading to higher
accuracy and significantly reduced training time and computational resources (Howard et al.,
2017).

10 | P a g e
 The use of Google's ML Kit's on-device Speech-to-Text and Text-to-Speech APIs is justified by
its cost-effectiveness, robustness, and offline capability. This choice aligns with the project's core
principle of accessibility, as it eliminates any recurring costs for users and ensures reliable
performance without dependency on network stability (Google AI, 2024).

16. Timeline
August, September & October: Foundation and Research

 Marvi: (AI Model): Research on MediaPipe Hands / Holistic and TensorFlow for gesture
recognition. Learn how to convert a trained model into TensorFlow Lite (TFLite) for
Flutter. Explore the speech_to_text and flutter_tts APIs.
 Qadeer: (Frontend – Flutter): Setup the Flutter project. Design application wireframes.
Explore necessary Flutter packages: camera, tflite_flutter, speech_to_text, and flutter_tts.
 Aaqib: (Data & Testing + Docs): Collect a dataset of sign language images and videos.
Preprocess the dataset (resizing, labeling). Start preparing the project documentation.

November, December & January: Core Development

 Marvi: Train and test the AI model. Convert the final model to TFLite. Implement Speech-
to-Text and Text-to-Speech functionality for local use.
 Qadeer: Build the application UI screens: Home screen, Sign-to-Text screen, Text-to-Sign
screen. Integrate the camera and model prediction pipeline.
 Aaqib: Begin testing the AI model's accuracy. Document progress with weekly notes.

February, March & April: Integration

 Marvi and Qadeer: Integrate the trained TFLite model with the Flutter app for real-
time sign detection. Connect the speech modules with the UI.
 Aaqib: Perform comprehensive functional and usability testing. Record bugs. Update
documentation.

11 | P a g e
 May: Finalization

 Marvi: Optimize the AI model for performance. Perform a final check.

 Qadeer: Polish the UI. Prepare the final application build.
 Aaqib: Finalize all documentation. Prepare the final Demo Presentation Slides and
Report.

17. References
1. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., ... & Zheng, X. (2016).
TensorFlow: A system for large-scale machine learning. In 12th USENIX Symposium on
Operating Systems Design and Implementation (OSDI 16) (pp. 265–283).
2. Ahmad, M., et al. (2024). Real-time Pakistani Sign Language recognition using a hybrid
deep learning model. Engineering Applications of Artificial Intelligence, 133(Part B),
108105. [Link]
3. Amjad, A., et al. (2024). PSLapp: A deep learning-based web application for static and
dynamic Pakistani Sign Language recognition. Software Impacts, 20, 100648.
[Link]
4. Arm Community. (2024). Breaking sound barriers: AI-powered sign language translation.
Retrieved from [Link]
blog/posts/breaking-sound-barriers
5. Biz4Group. (2024). Sign language recognition using AI: A complete guide. Retrieved from
[Link]
6. Deshpande, A., et al. (2022). A review of deep learning techniques for sign language
recognition. IEEE Transactions on Neural Networks and Learning Systems, 33(12), 7609-
7626.
7. Flutter Team. (2023). Flutter: Build apps for any screen. [Link]
8. Google AI. (2024). Project Relate: Helping people with non-standard speech be better
understood. [Link]
9. Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto,
M., & Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile
Vision Applications. arXiv preprint arXiv:1704.04861. Retrieved from
[Link]
10. Ministry of Information Technology and Telecommunication (MoITT), Pakistan. (2025).
National policy for the empowerment of persons with disabilities 2025 (Draft Report).
11. Nimdzi. (2024). The sign language interpreting market: Trends and insights. Retrieved
from [Link]
12. [Link]. (2024). The challenges of sign language interpretation. Retrieved from
[Link]
13. Wired UK. (2024). This AI-powered sign language interpreter is a quantum leap for the
deaf community. Retrieved from [Link]
interpreter

12 | P a g e