Demonstrate IT 2014 Project Report
for
Kinect Controlled Devices
Version 3.0
Team Demon
|
|
|
Table of Contents............................................................................................................. 1
Group Members................................................................................................................ 2
Project Timeline................................................................................................................ 2
1. Project Summary............................................................................................................ 3
1.1. Content Summary................................................................................................... 3
1.2. Lessons Learned..................................................................................................... 4
2. Reusability..................................................................................................................... 4
2.1. Materials Required.................................................................................................. 4
2.2. Setup....................................................................................................................... 5
3. How it works................................................................................................................. 6
4. Future directions............................................................................................................ 7
5. Appendix........................................................................................................................ 8
5.1. Lego Mindstorms.................................................................................................... 8
5.2. MindSqualls .Net Library........................................................................................ 8
5.3. Kinect SDK............................................................................................................ 8
5.4. Foundation of our project....................................................................................... 8
5.5. C# Programming.................................................................................................... 8
5.6. Others:.................................................................................................................... 8
|
Name |
Role |
Responsibilities |
|
Conor Fitzpatrick |
Software Developer, Concept Manager & Project Presenter |
Oversee project activities, supervise solution integration into existing infrastructure during updates. C# & XML development. Presenting Project at open day. |
|
Ollie Noonan |
Interface Design, Gesture Configuration & Software Developer |
Supervise hardware implementation. Design and implementation of UI. Power regulation control. C# development |
Project Timeline
|
Name |
Date |
Reason For Changes |
Version |
|
Conor Fitzpatrick Ollie Noonan |
10/03/2014 |
Initial Project plan: Robot design and development software chosen. |
0.1 |
|
Ollie Noonan Conor Fitzpatrick |
16/03/2014 |
Lego Mindstorm robot constructed. Test code for robot control build in Java, Research on Kinect programming. |
0.2 |
|
Conor Fitzpatrick Ollie Noonan |
29/03/2014 |
Initial Windows Application created for controlling the robot. Kinect development delayed as research continues. |
1.0 |
|
Ollie Noonan Conor Fitzpatrick |
31/03/2014 |
Kinect now working as basic controller but with poorly. |
1.5 |
|
Conor Fitzpatrick Ollie Noonan |
02/04/2014 |
Complete rewrite of software, Dropped use of java on robot. Device now completely controlled by Kinect over Bluetooth by keyboard simulation, Basic motion control. No code needed on robot |
2.0 |
|
Ollie Noonan |
04/04/2014 |
More advanced control through extra gesture recognition |
2.1 |
|
Conor Fitzpatrick |
10/04/2014 |
Simplified Gestures based on feedback from demonstration to Lecturers |
2.2 |
|
Ollie Noonan Conor Fitzpatrick |
11/04/2014 |
Final Version, Removed redundant code |
3.0 |
To boost interest in computer science at NUI Maynooth open days we decided our project should impress visitors and let them see the potential open to them if they were to pursue a degree in computer science.
As the expected age group of the open days are 2nd level leaving certificate students we felt that using existing technology that they would be familiar with combined with robotics may provide enough of a wow factor to increase interest in computer science.
Scope
This project demonstrates simulated keyboard input using the Kinect and a Lego Mindstorms Robot.
By utilizing the Microsoft Kinect’s motion tracking functionality we were able to set certain “key” gestures that when recognised by the master program would simulate a key-press.
To demonstrate this key-press simulation we mapped our keypresses to Lego Mindstorms motor control commands.
When a motion executed and recognised, a keypress in simulated and a motor command is sent to the robot which then starts to move (or stop depending on the command sent).
Schedule
This engagement began on April 10, 2014 and finished on April 28, 2014 with the submission of this report. The majority of this time was spent on research with approximately 13 hours of development.
Costs
All Hardware was provided by the Computer Science department.
All additional costs have been development, software and programming time.
Risks
|
Risk |
Impact |
Probability |
Mitigation |
|
The Project would require more time that we would have available as neither team member has experience with Kinect development or C# |
Moderate |
Moderate |
Continuous communication between all team members and scheduled meetings allowed for efficient utilization of time available. |
|
Our project may appear too similar to other projects being showcased by other students |
High |
Moderate |
As our project requires the user to use their entire body to perform gesture control we hope that this extra effort combined with mild embarrassment of the user on front of their friends may work to our advantage |
Communications Strategy
The team utilized several communication methods to collaborate with each other; including email, instant web meeting technology & face to face meetings. Online storage through Dropbox allowed for basic version control. Feedback provided by Lectures during presentation of the project to them was noted with high regard for improvements and changes.
Our primary lesson learned was to the consideration as to what initially enticed us to computer science. After careful consideration, we determined that the WOW factor was a strong motivator and as such we should attempt to wow potential computer science students by showing them how technology that was already impressive to them could be put under their control.
Another important lesson was to aim for small milestones and once they were complete to then add additional features. This allowed us to ensure a completed core project within the time available before any attempts were made to enhance features.
Below is a list of the materials required to both run and further develop this project. A copy of Mindsqualls is included in the Zip file for this project. Links to the software are provided in appendix #. Compatibility with future releases of any hardware or software listed in not guaranteed.
.
Hardware:
- Microsoft Kinect Device
- Lego MindStorms v2.0 Robot
· Bluetooth dongle or Bluetooth enabled computer
· Admin Access on a Lab machine running Windows 7 or later.
Software:
- .Net v4.5.1 or higher.
- Kinect SDK v1.8
- Mindsqualls .Net Library v2.2
- Windows Visual Studio 2010
The following guide should allow you to showcase the project.
It is assumed that a CS lab PC is being used and Microsoft Visual Studio 2010 is already installed and configured for general use.
|
Steps: |
|
1. Assemble a Lego Mindstorms Robot |
|
2. Log into lab machine using admin password |
|
3. Insert Bluetooth dongle (Installation may be required) |
|
4. Download and Install Kinect SDK |
|
5. Plug in Kinect to power supply and connect USB to lab machine |
|
6. Launch Visual Studio 2010 and open KinectWindowsForm.cspr |
|
7. Turn on robot and ensure Bluetooth is enabled in its settings |
|
8. Put the robot on the floor |
|
9. On the PC determine the COM port for the Robot. |
|
10. Run the KinectWindowsForm.cspr file from Visual Studio 2010 |
|
11. Choose the required COM port in the Device dropdown list or enter it in the text field and click Connect. * If the Kinect is correctly installed you should see it listed as Online in the status bar. * The Robot should also show as connected. |
|
12. Stand on front of the Kinect sensor |
|
13. Flap arms about like an idiot |
|
14. Success! |
At its basics, When the Kinect recognizes a gesture it compares it to the preconfigured gestures in the Gestures.XML file which contains the key the gesture is mapped to.
A virtual Keypress is then sent to the KeyboardController object which handles how to react to the key press.
The xml file defines both the gesture and the key press to simulate. The Description value is the text to be displayed in the output debug window when the gesture is recognized. The beginning and end positions of the two joints of the Kinect skeleton determines when this gesture is auctioned. The MaxExecutionTime is the time in milliseconds that the gesture should be made to be recognized allowing us to ignore false positives.
The MainForm.cs file requires the declaration of an object that implements the KinectControlledDevice interface. In our code this is a KeyboardController object.
The method void SensorSkeletonFrameReady(AllFramesReadyEventArgs e) in the MainForm is called by the Kinect to assess any potential gestures. It is within this method that a call to PerformKeyCodeAction(VirtualKeyCode keycode) should be made.
In our project we have key presses being used to drive the robot however as long as you implement the KeyboardController interface you can easily control any device or program that responds to key presses. For further proof of this run the software and open a blank notepad file to see the key presses getting typed out.
From the beginning we worked with the intention of allowing other students to easily develop this project further. Implementing the KinectControlledDevice interface should allow for some interesting projects. Without any knowledge of Kinect programming any future developer should be able to alter this project to control multiple external devices or control PC games etc.
Through trial and error a student can reconfigure the gestures file to map new key presses to further extend the control. Any student feeling overwhelmed by the amount of files and code in this project should take heart that the PerformKeyCodeAction method is the core method to develop.
Any students that are really feeling brave could consider advancing this project to track eye/head movements and control an electric wheel chair etc.
http://www.lego.com/en-us/mindstorms/?domainredir=mindstorms.lego.com
Contains guides for working with NXT.
See http://cache.lego.com/bigdownloads/buildinginstructions/4589648.pdf for a step by step guide to constructing the robot we used.
Also contains some very good sample code for controlling the Lego Mindstorm NXT through C# without needing to install any code on the NXT brick.
http://www.microsoft.com/en-us/kinectforwindowsdev/start.aspx
Also some very good video guides on Kinect development. Don’t forget to check out the samples in the SDK once downloaded.
http://code.msdn.microsoft.com/windowsdesktop/Simple-Gesture-Processing-097c5527
Excellent sample project, used as foundation for this project.
http://www.tutorialspoint.com/csharp/
Useful for getting to grips with C# programming.