A UNIQUE GESTURE-BASED USER INTERFACE: AN ACCESSIBLE ALTERNATIVE TO A KEYBOARD
 
I have been developing an idea to create a user-interface system for the modern PC in place of the generic keyboard-and-mouse setup. I believe that this idea has a lot of potential to make a computer more accessible to people with a disability such as loss of fine motor control due to a disease like Parkinson's, or limbs lost in war or accident. I have been using the Microsoft Kinect Sensor along with the Microsoft Kinect SDK to recognize different gestures and movements made by the user. 
My work is centered on creating a virtual keyboard that is designed using the Kinect SDK. It is a dynamic keyboard that reconfigures itself to adapt to the current context of the user's intentions. The keyboard – for which I have fondly coined the name "Bubble Keyboard" – involves selecting letters that are situated in little “bubbles” that surround a center bubble. There are two concentric rings of bubbles where the inner ring displays the most probable letter keys for the next keystroke, while the outside rings display all letters. My vision is to make a Keyboard that reconfigures itself in order to reduce the time it takes for a user to find a specific letter on the keyboard. It gets rid of the traditional QWERTY keyboard and creates a more natural typing system. If the Kinect Sensor were to be integrated with a regular PC, then this Keyboard can be used to type on the desktop for people with disabilities. Concerning other uses, for the standard user, this Keyboard is not nearly as fast as the traditional Keyboard, so it is not a good substitute for the QWERTY system. But if the user needs to type a single phrase while sitting in front of their television or if they need to run a search during a presentation, then this Keyboard is a better option than the traditional QWERTY system.
The problem  
Standard keyboard layouts can be frustrating to computer users who are challenged with using fine motor skills. Even a simple task such as entering a search string to find a movie title can be daunting. I started to consider a different layout that makes typing quicker and simpler. I wanted to come up with a simple intuitive layout with which the user can quickly learn and utilize.
I wanted to address the following requirements: 
1.	The user gets assistance in finding his or her next desired letter to type.
2.	The keyboard is simple to understand and start using.
3.	The layout of the letters is logical.
4.	The user is able to control the keyboard with whichever hand he or she considers dominant.
5.	The user is assisted in forming common words.
6.	The user is informed of misspellings or mistakes.
7.	The user is able to easily type words from a standard English lexicon.
My idea and solution
	My program, the Bubble Keyboard, meets all of the above requirements. It speeds up typing by offering the user the most probable letters that will succeed a currently typed sub-word. These most-probable letters are the easiest to access on this dynamic keyboard and are organized in a logical configuration, while all other letters are further away and are also organized in a logical second layer. The letters are placed in two concentric circles. All the letters are placed in the outside concentric circle, and the most-probable letters are pulled into the inside circle. Both sets of letters are arranged alphabetically around the circle. In order to type a word, a user quickly moves her hand from the center of typing activity to the letter that she wishes to select and back to the center again. After each letter is selected, a new set of most probable letters appear in the inner concentric circle. A snapshot of the keyboard can be seen in the figure. There is also a secondary keyboard layout that can be accessed with swipe-gestures, using the less-dominant second hand. A secondary keyboard layout has the set of numbers and special characters and symbols such as &, ^, *, etc. The secondary keyboard layout characters can be accessed in the same way as the primary keyboard layout.
Logic

The most-probable letters are statistically selected based on a dictionary of words. When the keyboard is initialized, the program creates a tree that organizes all the words in such a way that the most probable letters can be easily determined. The blue diagram below shows the tree for the words, APE, TREE, TREES, and TRAY with weights. The weight of any node is equal to the number of END nodes that succeed it along a path down the tree, or in short, the number of valid words that are possible if this letter is chosen. The green diagram shows the tree with the weights. Thus, for a given instance of the dictionary tree and any sub-word, we predict which valid words and how many valid words start with that substring. We also know what letters are most probable to be the next letter in the string of characters. When forming the tree, the program reads each word in a list and forms the corresponding node sequence. When it encounters the end of a word, it places an END node and sends an incrementing call up the tree all the way to the starting node to ensure that each node in the path to the end node has the appropriate weight.
Using a quicksort algorithm, I sort the most-likely letters in alphabetical order and place them around the center circle in a Bubble-like formation. If the user types in a word that is not in the current lexicon, then it is very easy to add the word to the tree. Additionally, I can store individual Dictionaries for each user of their non-standard Dictionary words. When the program is run, it can load multiple Dictionaries.
The user's other hand is used for commands such as finalizing a completely spelled word, backspacing, and switching back and forth between the keyboard layouts. When using the Kinect, a user will make natural gestures that people use to describe a situation or to physically move an object. When moving over a large distance, users make a large sweeping motion. I can categorize this motion based on the velocity of the person's movement. Using the vector coordinates returned by the Kinect of the joints, I define gestures based on these vectors over small time periods (1 – 1.5 seconds). Recognizing these gestures, I can make the Kinect experience more natural for the user. It is a practical alternative to the traditional point-and-click scheme.
Ideas for further additions and improvements
My first goal is to help the user easily find the next letter to type. In my current design, letters which are on a path that leads down the tree to words from the dictionary are colored differently. When the user has completely typed a word, the circle, which is the center of the typing activity changes color. However, different approaches can be taken. For example, there are certain variations where the inner circle of probable letters doesn’t exist. One such variation that I would like to experiment with is: As soon as a letter as chosen, the more-probable letters start to grow in size and come closer to the user. Thus, the user can easily track the next letter from its starting position in the outer circle. Secondly, the inner circle of choices could include word segments instead of letters. That way, users can finish typing a long word incredibly fast, saving them the time to choose each letter.
Lastly, the user needs to be able to quickly select his or her letter. Currently, the user moves the mouse in and out to the letter. But, there are other possible designs. For example, the user could rotate his or her hand in clockwise or counter-clockwise rotations to make a pointer travel around the two concentric circles in either direction. This would be very similar to dialing a number on an old telephone. These are some future possible directions that I am exploring.