Inspiration
The inspiration for "SoundTrack" comes from the desire to create a personalized and immersive audio-visual experience that reflects the user's daily life. By integrating multimodal data such as images, biomedical signals, and environmental factors, we aim to generate a soundtrack that not only matches the user's mood and surroundings but also creates a unique and engaging narrative of their day. The project is inspired by the idea of turning everyday moments into a cinematic experience, blending technology and creativity to enhance the user's emotional connection with their environment.
What it does
SoundTrack is a system that utilizes biomedical data (EEG, ECG, and IMU) to ascertain a user's mood and emotions. It captures images and occasional video clips of the user's surroundings, analyzing them to determine the context. Using this information, it selects and generates music in real-time, adapting the soundtrack to match the user's emotional state and environment. The system can also generate short-form music videos showcasing a set of user-selected images in a video playground.
How we built it
SoundTrack was built using a combination of Python for signal processing and data analysis, and various APIs for music selection and video generation. Here's a breakdown of the key components and technologies used:
Data Capture:
- Images and video clips are captured periodically using the device's camera.
- Biomedical data (EEG, ECG, IMU) is collected using appropriate sensors.
Signal Processing:
- Biomedical data is processed to convert it to the frequency domain and obtain Power Spectral Density (PSD) data.
- PSD data and location data are attributed to specific emotions/moods using machine learning models.
Music Selection:
- Images are processed using Gemini to extract information about the scene.
- The extracted information, along with user song preferences and mood derived from biomedical data, is fed into Gemini 2.5 to generate song recommendations.
- Spotify API is used to fetch and play the recommended songs.
Video Generation:
- Python processing is used to stitch together pairs of images and audio.
- Basic transitions are used for video clips to create a seamless montage.
- Lyria is used to generate background music based on the overall theme of the video.
Backend:
- Built using Python and FastAPI, providing a robust and scalable framework for handling real-time data processing and API interactions.
- Data is ingested using Firebase to handle real-time and batch data.
- FFmpeg is used to stitch together the images and audio to create a cohesive video montage.
Other Key Technologies Leveraged:
- Developed using React and Next.js, providing a dynamic and responsive user interface.
- Used Raspberry Pi Zero 2W for image capture, ensuring a compact and efficient setup.
Challenges we ran into
- Hardware Limitations: The Raspberry Pi Zero 2W, which was a vital component of the project, had significant performance limitations. These were remedied by simplifying its use-case, resulting in the offloading of all image processing to Gemini 2.5 and necessitating the use of Firebase to store captured images. Server deployment also ran into bottlenecks, necessitating scaling up the infrastructure.
- API Deprecation: Spotify API deprecated features that were crucial for our implementation, such as obtaining song details. We had to quickly pivot to supplement these features with Gemini 2.5 to maintain functionality.
- Accurate Emotion Mapping: Precisely correlating physiological signals with emotional states was challenging. We had to experiment with different algorithms and models to achieve accurate results.
Accomplishments that we're proud of
Throughout the project, we learned the importance of cross-disciplinary collaboration, drawing insights from fields like neuroscience, signal processing, and multimedia art. The complexity of mood analysis taught us valuable lessons in data interpretation and algorithm development. We also gained experience in utilizing APIs effectively to enhance user interaction and content generation.
What we learned
- Managing a Full-Stack Project: We gained valuable experience in managing all aspects of a full-stack project, from data capture and processing to API integration and user interface design.
- Leveraging Existing LLMs: We learned how to effectively use existing large language models (LLMs) for prototyping and planning, which significantly accelerated our development process.
- Real-time Processing: We developed skills in optimizing algorithms for real-time processing, ensuring efficient data handling and synchronization.
- Adaptability: We learned the importance of quickly pivoting when things don't go as planned, such as when API features were deprecated or hardware limitations were encountered.
What's next for SoundTrack
- Enhancing Accuracy and Reliability: Improving the accuracy of mood and emotion detection through advanced signal processing and machine learning techniques.
- User Customization: Adding more user customization options, such as the ability to prioritize certain types of music or visual elements.
- Expanded Data Sources: Incorporating additional data sources, such as weather or calendar events, to further personalize the experience.
- Social Sharing: Developing features for users to share their soundtracks and videos, fostering a community around this unique form of personal expression.
Log in or sign up for Devpost to join the conversation.