theremin

Start you experience here!
Interface view

Have you ever wondered what music generation on your fingertips would be like? Well, look no further!

Inspiration

Throughout history, humanity has been fascinated by music. A symphony that began nearly 75,000 years ago with the discovery of ancient instruments, has evolved into a majestic crescendo. From the very first flutes and stringed instruments, we have made our way to the piano and electronic instruments. Inspired by the revolutionary original Theremin, we combine modern electronic music synthesis with computer vision and machine learning to reinvent musical performance.

What it does

Brings music to your finger tips, literally. Theremin allows unleashes the musician in you, where you are the embodiment of the music that you produce. With the power of AI, you become the conductor, and your movements shape the symphony. Theremin is a music generation software that produces drums, chords, and melodies by analyzing your hand gestures and movements.

Not only does is it able to recognize your hand poses, it also uses a machine learning model to generate great-sounding melodies based on the chords you are playing.

Your left hand controls the chords, your right hand the melody. Open and close your hands to activate their respective sounds. Make a Rockstar sign with your hand to activate the drums!

Hit record and enjoy creating music realtime!

How we built it

We used MediaPipe to get hand pose and gesture information and Magenta.js and Tone.js to generate melodies and audio using synths and samples. We also used Teoria.js for music theory.

Challenges we ran into

We initially had plans to include multi-player as well as body-pose data. However, we had compatibilty and optimization issues that forced us to reduce complexity. This ended up having a silver lining, in that it allowed us to be more creative with the degrees of freedom we had available.

Accomplishments that we're proud of

Something that we all celebrated was the generation of different chords based on the hand placement. This kicked off our project, because it was at this point that we were able to map the music with the gestures successfully. Ultimately, we wanted to parse pose data into music in a way that felt intuitive and natural. This required considerable effort, but I think we achieved it.

What we learned

We learned a lot more than some frame works this hackathon! We learned quite a bit of music theory to produce melodies based on chords, and we learned about the process of electronic music synthesis from the unique perspective of code. Learning about the gesture recognition API was really fun for all of us.

What's next for theremin

One of the first things that we would like to add to theremin is the possibility for more than one person to generate music. Secondly we would also like to add an option to produce more variation in the generated music by giving users an option to read in body movements too. Finally, we would like to add more instruments and expand our music generation capabilities. (Possibly enough for theremin to be a comprehensive music production software one day!)