- Lab Windows/CVI 2017 Full Development System: version 17.0.0
- Ableton Live 10 Suite: version 10.1.1
- Unity: version 2017.4 (LTS release)
This is a rehabilitation system designed for finger exercises. It utilizes Unity to connect our self-made devices (desktop and handheld versions). The desktop version is compatible with Hololens for AR training. The handheld version can be connected to an iPad via Bluetooth for training, and it provides a specialized UI on the iPad to assist with the exercises. This is a demo held in Nara's hospital.
https://www.youtube.com/watch?v=Pf5aNwb4C14
Connect the cables of load cells to the DAQ device (National Instruments USB-6211) like the bottom image. If you use right hand, connect the thumb's cable to AI1 and AI9.
Start the project, double cllicking "PETS.exe" or Open "PETS.prj" in Lab Windows and click the run button.
Confirm that it matches the fingers on the monitor by pressing the sensors in order.
The thumb part in the window should react when pressed with the thumb.
If they don't react, check the cable connection (section 1.1.1). +5V or GND cables often miss the connection.
If they don't match, Change the cabele connections or the setting of finger correspondences (the part enclosed in red in the bottom right image, line 62-66) in "main.c".
Set the Max effect, "UDPtoUnity.amxd" under "Ableton/User Library/Presets/Audio Effects/Max Audio Effect/~" and set "playviaudp.amxd" under "Ableton/User Library/Presets/MIDI Effects/Max MIDI Effect/~" (In my case, I found the Ableton folder in "Users/(user name)/Music/~". I think the place of the folder changes according to the setting when you install Ableton Live).
Open the music project (.als file) in Music-PETS project floder.
And check if Ableton Live recognizes Max effects correctly. If it recognizes them, you can see them like the bottom images. If you see the piano roll in the bottom of the Abelton Live, press "Shift+Tab". you can see switching the display there.
If Ableton Live is missing the Max effects, Set them into the missed effects.
Postscript: The function, "OSC-route", used in the effect, "UDPtoUnity.amxd", is the plugin function. So you hove to download it so that it works well. Go to http://cnmat.berkeley.edu/downloads and download "CNMAT Max/MSP Externals for Mac and Windows". After unzip this, move it into the folder (Max8/Packages/). Reboot Ableton Live. Then "OSC-route" works.
Open Music-PETS folder by Unity2017. (I recommend you use Unity Hub. It can manage various versions of Unity.) This folder is the Unity project file as it is. Or import "Music-PETS.unitypackage" after make a new project.
If you play Music-PETS without HoloLens (play with desktop monitor), you can skip this section.
Click the tab, "Mixed Reality Toolkit" -> "Configure" -> "Apply UWP Capability Settings". And Check the check boxes other than "Internet Client".
Click the tab, "File" -> "Build Settings...". And Set like the bottom image.
After setting, Click Build button. And select build folder. If you don't have build folder or you want to remain previous builded app, make new folder.
Connect HoloLens to the PC via USB cable.
Open "HololensPETS-Unity.sln" in the build folder. And set the deploy setting like the bottom image.
After setting, click the run button. If you are asked the PIN code of HoloLens, input it (you can check PIN code in HoloLens, "Setting" -> "Update&Security" -> "For developer" -> "Pair").
If the app starts well, stop the run. the app is already listed in HoloLens applications. When you want to start HoloLens app, click "All apps" and the app you want to start.
Start the project, double cllicking "PETS.exe" or Open "PETS.prj" in Lab Windows and click the run button.
And press start button on the window.
If it work correctly, The waveform reacts as shown in the image below. Confirm that it matches the fingers on the monitor by pressing the sensors in order.
Open "PCControlPanelStartScene" in Unity. (It's in "Asset/PETSControlPanel/Scenes".) Click the run button, then Unity app starts.
After Unity app starts, calibrate base forces and max forces of fingers.
First, click PETS calibration button. When it calibrate, don't press sensors mounted on PETS.
Second, click each finger calibration button. When it calibrate, press the sensor as hard as possible. If you'll use only right hand or left hand, you don't need to calibrate both hands.
Once you calibrate, you can reuse this data by saving and loading it. If you want to save the data, enter the name and click the save button.
If you play Music-PETS without HoloLens (play with desktop monitor), you can skip this section.
Check the IP address of HoloLens and enter it in the textbox in the upper right of Unity window. You can check the IP address in "Settings" -> "Network & Internet" -> (connected Wi-Fi) -> "Advanced options" -> (scroll down) -> "IPv4 address".
Start the HoloLens app by click "All apps" and "HololensPETS-Unity".
When you start the app, you can see the text "State: Runnning" in the upper right. After long tap, you can see it change to "State: Calibtating". Then you calibrate the five sensor potision using euphoria image.
The xyz axis can be seen by placing the Euphoria image on the sensor like the image in below. And when you tap, the potision of the sensor is registered. And you can see a green cube in registered position. you repeat this five times from thumb to pinky. After finish five calibrations, you can see five green cube on the sensors like the image in below. Then tap and Check the state is "Running".
Open the project of the music you want to play. Then enter the IP address of Unity. Click the edit button enclosed in red in the bottom image and open the max effect, "UDPtoUnity". Doublr click the state enclosed in red in the bottom image and enter the IP address. (Second argument is the port. You don't need to change it.)
After editting, save it by pressing "Ctrl+s" key. And Finish Max software. When Max software work, the communication of Unity and Ableton Live don't work well.
Click "Music Play" toggle box in Unity app. Then you can see like the image in below.
The settings menu is open from the beginning. You can choose the music and set the difficulty in this window. You can open and close this menu by pressing "s" key.
First, enter the IP address of Ableton Live. If Unity and Ableton Live work on the same PC, enter "127.0.0.1".
Second, choose the music and track. You have to choose the music you opened in Abelton Live in previous chapter. Basically, choose the melody track because some track have few note for playing.
When the connection between Unity and Ableton Live works well, you can see the message from Unity like the image in below. When you select the track, you can see the light of the selected track turn off. The track turned off are muted so that the player play.
Third, set the difficulty. Frequency means the minimum interval of finger pressing. The longer this interval is, the easier the difficulty is. Set the number (1, 2, 4, 8, 16, 32) in it. 1/4 bar means the interval is quarter note.
Finger Select enables select of fingers for playing. If you want to play without thumb, remove the check on thumb toggle button.
After setting all, click "OK" button.
If you use HoloLens,
You can see the five music lanes on the sensors like the image in below. And when you press the sensors, you can see the circle under the lanes.
Check the following:
- You can listen the sound when you press the sensors.
If this is wrong, you might miss setting IP address. Set it correctly. - The music you selected in Unity and the music opened in Ableton Live are same.
If this is wrong, set it in the setting menu after open it by press "s" key. - The track you selected in Unity and the track muted in Ableton Live are same.
If this is wrong, set it in the setting menu after open it by press "s" key. - The start position in Ableton Live is set the begining of the music (the number to left of play button is "1.1.1").
If this is wrong, click the stop button in Ableton Live twice.
If you use HoloLens,
- You can see the music lanes in the correct position.
If this is wrong, HoloLens miss tracking. click the toggle button of the other game in Unity and click "Music Play" again. Then you might see them correctly.
press "space" key. Then the music starts. Press the sensors at the timing indicated by the score.
The music stops by pressing "space" key.
The setting menu opens by pressign "s" key.
I'll write this section in future. In this section, I write how to add new musics in this system.