ENGR 103 - Spring 2013
Freshman Engineering Design Lab
“Hands Free Computer Interface”
Project Design Proposal
Date Submitted: May 14, 2013
|
Submitted to:
|
D.J. Bucci, djb83@drexel.edu
|
Group Members:
|
Anthony Young, ajy36@drexel.edu
|
Cheryl Tang, ct482@drexel.edu
| |
Leandra Lipat, lfl32@drexel.edu
|
Abstract:
Many people with limited hand use cannot fully use their computers. There are systems that use different trackers for mouse use, but they do not help very much with ease of use and different commands, which are better suited for keyboards. The goal of this design project is to facilitate computer use for quadriplegics by integrating a voice command system which can allow users to speak various commands and to script a mouse that can be controlled by the facial muscles of the eyes and jaw. This will be a difficult system to implement, as the user will have to focus on how they move their eyes and jaw, and the integration of the different systems could have some problems with muscles that have different uses. In the end, the final product will include a set of electrodes and a microphone. The microphone will send signals to a MATLAB script, which will differentiate between different commands and will interface with the operating system of the computer in order to control the mouse and use different functions by voice command.
1 Introduction
Computers are constantly being updated for user-friendliness, but usually fail to take into account a certain audience, people with limited to no hand mobility. Controlling a mouse and navigating a computer is difficult for them. This audience includes quadriplegics, amputees, arthritis sufferers, and anyone who would prefer to give their hands a break.
We plan to create a hands-free way to interact with the computer. Our device will detect electrical signals produced from muscles and use them as a computer mouse. EMG signals from jaw muscles will control left/right clicking and EOG signals from the eye will control the movement of the cursor on-screen. To make navigating the computer even easier for the users, we will also implement a word recognition program that will record voice commands and allow access to programs and actions. This will include shortcuts to the Internet, zooming in/out, saving a document, etc. This program, along with the program for EOG and EMG signals, is coded in MATLAB. We hope to allow the users to quickly and easily navigate their computer with only their eyes, jaw, and voice.
1.1 Problem Overview
Computers have become deeply integrated into modern society. Everyone in today’s age will most likely need to use a computer at some point. So, computers are constantly being updated for user-friendliness, but usually fail to take into account a certain audience, people with limited to no hand mobility. Hand use is absolutely necessary for the standard computer. Hands are used to type and control a computer mouse. Therefore, quadriplegics have little to no ability to access an ordinary computer. Since computers have become so integral to everyday life, these individuals have difficulty performing normal tasks that involve the computer, such as communicating or searching the internet.
Individuals with disabilities that affect their arms (quadriplegics and amputees) will need to use a different part of their body to interact with a computer. The head, eyes, tongue, and voice have been used to control devices. But, the eyes and voice are most natural to the user. People would naturally gaze at what icon they would click on. Talking is also a widely used form of communication, so having voice commands to navigate the computer would also be fitting.
Design constraints for our device involve taking the user’s comfort into account. The electrodes need to comfortably stay on the face, without the electrode leads getting in the way. The word recognition program also needs to conform to the user. The program needs to recognize the incoming speech correctly, however as quadriplegics can have trouble talking, they are more likely to have different speech patterns than those that would be picked up by a general interface. This also helps as not every user will have a similar voice or speak the same way. The cursor movement and clicking also needs to be in sync with the user’s actions.
Aiding in the implementation of this project, the weight and size of our device is small. This allows user-friendliness and makes it easy to transport if necessary. The price of our device is around $60 on top of the price of laptop and of the data acquisition board.
1.2 Existing Solutions
Having quadriplegics easily interact with computers is not new. Several assistive technologies already exist today, allowing users with limited to no hand mobility to navigate a computer with ease.
The AssistiveWare company created the SwitchXS, a virtual scanning keyboard. This keyboard includes letters, characters, mouse controls, and specialized keys (one for choosing food). The computer displays an onscreen keyboard and cycles through each row of keys. This cycle acts like a cursor. Once the cycle reaches a desired row, the user can activate a trigger to select it. “The selection trigger can be a puff of air blown in a tube, a twitch of a cheek muscle, movement of a finger, an EEG response (e.g., Kaiser et al., 2001), or the blink of an eye (Bauby, 1997; see also Tavalaro and Tayson, 1997) [1].” The cursor then targets on this selected row and cycles through that row, until the user triggers a switch to activate a certain key.
Taking a closer look into using the eye as a trigger, eye wink control interfaces have also been developed. They allow users to control mechanical devices by the timing of their eye winks. This is possible with the use of a infra-red eyewink detector and the computerized signal
interpreter. The detector is put on normal eyeglasses. “When an intentional wink occurs, the detector unit passes the information directly to a micro-computer. The computer matches the sequence of winks with a pre-defined command set, and issues the corresponding signal to the
output device [4].” However, the number of commands the eyewink control interface can take is limited. For example, to control a wheelchair “The four possible inputs from the eyes: both lids open, both lids closed, right lid down, and left lid down, resulted in four predefined actions: no movement, move forward, right turn and left tum respectively [4].” Eye movement and position tracking has more input possibilities, limited only by how many options can be brought onto the monitor screen.
Word recognition systems have also been developed, allowing users to just speak to type or perform a command. Most of these systems require training, so users need to give voice samples. These samples are stored and, when the system is activated, are matched with the incoming speech of the user. “Isolated word” recognisers match one word at a time. “Connected word” recognisers match phrases without users pausing between words. We would implement an isolated word system, due to the nature of the command phrases (one to three words).
1.3 Project Objectives
Our device will allow those with no hand mobility to access computer mouse controls and commands. The onscreen cursor will follow the movement of the user’s eyes. Clenching of the left or right jaw would activate a left or right click. To streamline activity with the computer, we will implement voice commands that serve as shortcuts to programs and commands. Commands will include: “open/close *program,* “ (programs would be Internet explorer, microsoft word, windows media player, iTunes, and an option to add other desired programs) “save document,” “undo” or “redo,” “zoom in/out,” “Search for file,” and “Stop voice commands” to stop recording. With this, users with disabilities can easily and comfortably interact with a computer
For eye movement tracking, we will use EOG signals from eye muscles. The signals will be picked up by electrodes on the sides of the eye. The electrodes will be placed above, below, and on the sides of the eye. These areas have the extraocular muscles that turn the eye. EMG signals from jaw muscles will control the mouse clicking. These electrodes will be placed on the sides of the jaw, where the masseter muscles are. These muscles close the jaw.
Figure 1: Electrode Placement for EOG signals
A built-in microphone from a laptop will record the user’s spoken command. The word recognition program on MATLAB will then compare this incoming speech with stored speech samples. Once a match is found, the command associated with that phrase will be executed. The word recognition program will initially need to be trained. This involves repeating a single word into the microphone and capturing and storing that speech. Due to the differences in voices and word utterances, each user will have to separately train the program.
2 Technical Activities
2.1 Electronics
Electrooculograph (EOG) technology is based upon the small charges created by movement of the eye. The eye has a small net positive charge at the cornea and a negative one at the retina. As the eye moves, electrodes placed on the skin will be used to determine the amount of electrical activity of muscles in the face, which will be used to control the computer. The electrodes will be placed on the face to the side of the eye and bottom of the eye, and adjusted until the sensors are able to record data successfully.
The electrical signal will need to be amplified, as the base potential difference is on the order of microvolts, in order to increase the signal strength so that it will be useful for MATLAB to delineate between smaller changes in charge. This will be accomplished by using an instrumentation amplifier connected to the electrodes on the face and a reference electrode, which will be placed on the forehead, where there will be no detectable charge from movement of the eyes. After the signal is amplified, it will be connected to a data acquisition board compatible with MATLAB. The electrodes provide the signal input for the mouse code and allow the user to passively control the computer with the mouse and buttons as they would control the mouse.
The mouse buttons will be controlled by Electromyograph (EMG) technology. This is based on the same principles as the EOG, but it uses the electrical signal from movement of skeletal muscles. For this application, electrodes will be placed on the jaw muscles that control closing the jaw and opening the jaw on either side. These electrodes will also be connected to an amplifier, and will control the clicking of the mouse as well as the microphone activation for voice commands.
Figure 2: Electronics and Signal Acquisition
The instrumentation amplifier, shown below in figure 3, was constructed on a circuit board. The electrodes would connect to wires leading out of the board with alligator clips. Another wire will then connect to the DAQ to transfer the signal. Two 9V batteries will power the device, with the wires shown in red/black.
Figure 3: The constructed instrumentation amplifier circuit board.
As stated, the instrumentation amplifier is needed to amplify the electrical signal received by the electrodes, as the electrical potential generated by muscle contractions is too small to be read by a computer. With the instrumentation amplifier, the signal is scaled by a factor in order to be computed by the DAQ. This instrumentation amplifier was constructed following the guidelines shown in figure 4:
Figure 4: Instrumentation Amplifier circuit layout.
2.2 Mouse Coding
The MATLAB code will take the electrical signals from the muscle movement around the eyes and associate them to the movement of the cursor on the computer screen. Each button on the mouse will be coded to clenching the respective side of the jaw when the cursor is properly placed. The MATLAB coding will consist of taking a baseline reading for the user’s EOG levels, and then doing a test run asking the user to look to different directions and having the program check the signal input from the different electrodes with each action. Afterwards this will be checked against a new set of eye commands after the algorithm is trained to the signals from the user’s muscle movements. The signal levels of the electrodes placed on the jaw muscles will also be used to control the clicking action of the mouse and will need to be coded in and tested as well.
Figure 5: The above block diagram summarizes the initialization of the mouse script.
2.3 Voice Command Coding
The microphone will record user responses to different prompts for shortcuts or programs,
including:
● “open/close *program* “ including Internet explorer, microsoft word, windows media player,
iTunes, and will include an option to add other desired programs
● “save document”
● “Undo” or “redo”
● “Zoom in/out”
● “Stop voice commands”
● “Search for file”
These will allow the user to gain much of the functionality of the keyboard for shortcuts and improve the interface and ease of use. The responses to prompts will be stored and then compared to the different voice commands given by the user, and the command that is recognized by the speech recognition program will be activated through a section of code that can interact with the operating system.
There has been some previous work with this detailed by Mathworks, the company that created MATLAB, which will be referenced, but it has not been implemented in conjunction with a hands free mouse control system. This will be one of the more complicated parts to code, and therefore will require a lot of work in order to get the program to recognize the different commands given by the user. In order to properly accomplish this, testing must be done to collect data of the sensor readings of each command. From here, a normalized range for each will be catalogued for reference. Admittedly, this will require a relatively high level of MATLAB coding, and as such, the internet and other online resources will prove to be very useful.
Figure 6: Voice Command Recording and Implementation
2.4 Project Timeline
Week
| ||||||||||
Task
|
1
|
2
|
3
|
4
|
5
|
6
|
7
|
8
|
9
|
10
|
EMG/EOG and Speech study
|
x
|
x
| ||||||||
Mechanical design
|
X
|
x
|
x
|
X
|
x
| |||||
Electrical design
|
x
|
x
|
X
|
x
|
X
| |||||
Coding
|
x
|
X
|
x
| |||||||
System integration
|
X
|
x
| ||||||||
Testing for cursor/voice
|
X
|
x
|
x
| |||||||
Final report preparation
|
x
|
x
|
x
|
Figure 2: Freshman design project timeline.
2.5 Project Budget
Category
|
Projected Cost
|
Electrodes
|
$23.93
|
Amplifier Subcomponents
|
$32.85
|
Circuit Board (2)
|
$8.99
|
Data Acquisition Box
|
$0.00
|
TOTAL
|
$65.77
|
Figure 3: Freshman design project budget.
Amplifier Subcomponent
|
Projected Cost
|
The instrumentation amplifier (4)
|
$21.65
|
Stranded wire
|
$0.00
|
Alligator clips
|
$0.00
|
9V battery leads
|
$1.22
|
9V batteries
|
$9.98
|
TOTAL
|
$32.85
|
Figure 4: Amplifier budget
3 Results
The project is currently incomplete.
4 Future Work
There are a few obvious ways to improve this interface system, starting with adding more interface options, such as transcription of the speech of the user. For the purposes of this system, it does help for navigating the computer, but has limited functionality for using the computer for its intended purposes. In order to give the user better functionality for programs and the possibility to have specific programs which could help with writing emails and sending text to others. In terms of sensitivity and effectiveness for the calibration of the interface, different settings could be created for use in different settings, where either the program or the user could adjust presets to their environment, if it was getting loud and they wanted to continue using the program, or if the program could adjust to different interfering sounds while the user was talking.
Another use for this system could be providing a set of mental exercises and games for patients who want to keep sharp. Having such limited mobility and depending on another person for much of their well being limits the user to basic forms of communication, and allowing them to have an added option to do puzzles or play games would entail a better quality of life when they are not accompanied by a caretaker. This would also enable them to keep busy when much of their time is limited to taking care of their body, but little of it helps their mind. The possibility to find another way to interact with friends and family through the kind of social games that are popular right now would hopefully give them another way to communicate and have fun with other people.
Once there is a platform and a market for people who have mouse movement and voice control as their interface tools, it would be possible to develop a way of getting more programs that could help patients interact, receive care, and pass time in much the same way as computers are used by the general public. Once the functionality of the program reaches a level where it could greatly help people with these limitations, it would be very easy to expand this system to other people with varying levels of ease with using computers, and allow for more people to interact and use computers.
5 References
[1] G. Francis. (2011, Jul.). “Speed–accuracy tradeoffs in specialized keyboards.” International Journal of Human-Computer Studies. [Online]. 69(7-8), pp. 526-538. Available: http://www.sciencedirect.com/science/article/pii/S1071581911000486
[2] J. J. Struijk. (2006, Dec.). “An Inductive Tongue Computer Interface for Control of Computers and Assistive Devices.” Biomedical Engineering, IEEE Transactions on. [Online]. 53(12), pp. 2594-2597. Available: http://ieeexplore.ieee.org/xpls/icp.jsp?arnumber=4015621
[3] J.M. Noyes, R. Haigh, and A.F. Starr. (1989, Dec.). “Automatic speech recognition for disabled people.” Applied Ergonomics. [Online]. 20(4), pp. 293-298. Available: http://www.sciencedirect.com/science/article/pii/0003687089901932
[4] R. Shaw, E. Crisman, A. Loomis, Z. Laszewski, “The eye wink control interface: using the computer to provide the severely disabled with increased flexibility and comfort,” Proceedings of Third Annual IEEE Symposium on Computer-Based Medical Systems, Chapel Hill, NC, 1990, pp. 105- 111
No comments:
Post a Comment