To build our project, we developed an iOS-based app using Swift, designed to assist visually impaired individuals in locating objects and navigating complex indoor environments. By leveraging advanced technologies like augmented reality, machine learning, spatial audio, and voice recognition, we created a highly sophisticated and technically challenging solution. The app uses Swift's ARKit and the iPhone's LiDAR Scanner to detect objects and map the surroundings accurately. This setup allows for precise detection of surfaces, including vertical, horizontal, and angled planes, enabling the app to guide users effectively through spaces. For object detection, we integrated the YOLOv3 model with CoreML, allowing the app to identify various items, such as people, bottles, bags, and phones, in real-time. The integration of YOLOv3 with Swift posed significant technical challenges, as we needed to ensure that the object detection was both fast and accurate, even in dynamic environments. A standout feature of the app is its use of spatial audio via AirPods Pro. This allows the app to emit directional sounds, guiding the user towards objects and helping them navigate by adjusting the volume based on proximity—the closer the obstacle or desired object, the louder the sound. This required precise calibration of audio cues with real-time object positions, ensuring accurate and intuitive feedback. One of the app's most complex aspects is its dynamic 3D mapping and object tracking. As the user moves, the app continuously calculates and updates the positions and distances of objects relative to the user, even if objects move out of view. This continuous mapping and tracking create a reliable reference for navigation, demanding efficient data handling and processing to maintain performance and accuracy. When a user searches for a specific object, the app places a virtual beacon on the detected item and provides proximity feedback through spatial audio cues, which become louder as the user approaches the object. The development of this app required the seamless integration of multiple advanced technologies, pushing the boundaries of iOS development. The challenges included real-time performance optimization, precise spatial audio calibration, and the dynamic adaptation of 3D mapping in constantly changing environments. By overcoming these hurdles, we created a powerful tool that significantly enhances the independence and mobility of visually impaired users.
Yash1361/SoundScape
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|