This repository provides you code to replicate the AR experiences in the paper Adding Force Feedback to Mixed Reality Experiences and Games using Electrical Muscle Stimulation (CHI'18), which you can read here. Also here is the accompanying video.

This repository has the two core folders:

1. Assets: pretty much everything lives in here. See the following:

   1.1 Assets/Root/Server/Scenes/Server.unity the Unity3D app that run in the server (a laptop connected to the stimulator) is at

   1.2 Assets/Root/Applications/Scenes/ has four HoloLens applications you see in our video: labyrinth.unity (a marble balancing game to be played with a physical prop), Walkthrough.unity (a room planning application with a couch, lamp, button and thermostat), Escape.unity(and AR escape room!) and Userstudy.unity(if you want to replicate our study conditions).

2. ProjectSettings: Just settings.

This will require:

1. Unity 5.5.1f (other versions might work with modifications)
2. Vuforia Plugin for marker tracking (you can read here how to install Vuforia for Unity)
3. A working HoloLens installation with the HoloLens []toolchain](https://developer.microsoft.com/en-us/windows/mixed-reality/install_the_tools) (Windows 10, Visual Studio, etc)
4. A HoloLens Mixed Reality Headset connected to your Wifi router
5. Put the Unity Assets in the right place (AssetStoreTools folder).
   • Holotoolkit 5.5.0
   • Vuforia >6.2.10
   • In Vuforia's DefaultTrackableEventHandler please change OnTrackingFound(),OnTrackingLost() to protected virtual.

Optionally, we recommend:

• using our AR marker generator to quickly create a batch of markers (for hands, objects, etc) instead of manually creating every marker.
• Note: we removed the asset (3D object and scripts) of the catapult you see in our video because this is a commercial asset we purchased (feel free to do the same, it is available as trebuchet on the Unity Asset store!).

This project was built for a particular EMS device (Rehastim V1, which is off the market, sorry!). However, you can easily modify it to run on your custom EMS device (as long as it accepts USB commands) or using an open-source EMS controller such as openEMSstim. If you are modifying the EMS_server to run with another EMS device, read this first.

Since the HoloLens we used does not support a USB connection (for the muscle stimulator) we used an additional laptop running as a server (hence the Assets/Root/Server/Scenes/Server.unity that yoy must run on your laptop). See the paper for how to pack this is a convenient backpack form factor.

1. Clone this repository by running git clone https://github.com/PedroLopes/HoloLens-ElectricalMuscleStimulation.git (or you preferred visual tool)
2. Open the project in Visual Studio.
3. Connect your HoloLens, connect also your HoloLens to a Wifi
4. Select to deploy this project to the HoloLens (verify that the app is launched and running)
5. Open the project in your local machine (that will act as EMS_server), connect this machine to the same Wifi.
6. Setup the communication between both HoloLens <> EMS_Server. To do this you have to configure the EMS_server to receive requests from the HoloLens. <give the IP address of the EMS_server to the HoloLens app?>.
7. Deploy and Start both apps (once more double checked the IP addresses and Ports match).
8. Configure your EMS machine, calibrate it to work comfortably and Pain-free (see here for more details).

1. Now, launch the HoloLens app, put your hand out in the pointing gesture (to make sure it is tracked, or attach a Hand_marker to it -- review your code to make sure you supplied the right marker to Vuforia).
2. Reach your hand out and touch a virtual object (e.g., we suggest you try the couch scene first). You should feel EMS (verify that the EMS_Server received the message).
3. Hooray! (or read the FAQ below)

You can't simply copy someone else's EMS calibration since it varies slightly from user to user (refer to openEMSstim for details on how to calibrate or see our paper). Still, you can use these values on the table below to guide your haptic effect design:

p.s.: note that they might differ from the current code which is more of a demo-version.

This work was kindly supported by the Hasso Plattner Institute.

The code was authored by Sijing You (Hololens Apps, EMS Unity Server) and Pedro Lopes (EMS protocol, Marker Generator). You can contact them via github.

The paper Adding Force Feedback to Mixed Reality Experiences and Games using Electrical Muscle Stimulation (CHI'18), which you can read here was authored by Pedro Lopes, Sijing You, Alexandra Ion and Patrick Baudisch.

You can get or build an openEMSstim (a piece of hardware that allows you to easily control an off the shelf muscle stimulator). All the schematics and code are available.

1. Look at your EMS_Server, did it get the message from the HoloLens? If not

   1. re-check the IP addresses for both devices.
   2. Make sure they are on the same network
   3. Make sure they can ping each other (in other words, make sure this network is not aggressively blocking ports/messages)

2. If the message arrived at your EMS_Server but you did not feel anything:

   1. Recheck your EMS machine by calibrating it again.

Please refer to the liability waiver (in documentation/liability_waiver.md).