Play with audiovisual instrument

by Chikashi Miyama Ph.D

Udacity VR High Immersion Capstone Project

Every musical instrument offers an interface for controlling sound, such as keyboards, strings, sliders etc. What if we can use the VR 3D space as an control surface of an electronic instrument?

Playing musical instrument is a JOYFUL experience for all of us but it usually requires years of practice.

PureData (http://puredata.info) is an environment for developing interactive audio signal processing algorithms that can be integrated in another software, using libPd (https://github.com/libpd). Using libPd created LibPd4UnityTools ( https://github.com/Magicolo/LibPd4UnityTools) in order to facilitate the connection between Pd and Unity.

Oculus Rift

1. Empathy Achievement (100 pts)

   • An instrument for non-musician

2. Physics Achievement (100 pts)

   • Sound synthesis (physical simulation of waves)

3. Animation Achievement (100 pts)

   • Realtime sound visualization

4. Scale Achievement (100 pts)

   • Big sphere for sound control

   • Visualization outside of the sphere

   • Small 3D buttons in front of the player

5. Lighting Achievement (100 pts)

   • 3D buttons are lighted properly

1. Diegetic UI achievement (250 pts)

   • The function of buttons can be understood without words

2. 3D Modeling achievement (250 pts)

   • The buttons are modeled with Blender

1. Compute Shader achievement (500 pts)

   • The FFT Centroid is calculated with the compute shader

    

Link: Achievements

Link: Descriptions