Researchers at Institute for Computing Research of the University of Alicante have developed a multisensor system for rehabilitation and interaction of people with motor and cognitive disabilities. The system enables to perform different therapies using multiple modes of interaction (pose and gestures of the body and hands, voice, touch and gaze position) depending on the type and degree of disability. Through a training process, the system can be customized enabling the definition of patients’ own gestures for each sensor. The system is integrated with a range of applications for rehabilitation through virtual reality applications and 3D interfaces. Examples of these applications could be the realization of puzzles, mazes and writing with predictive text. The system also provides a flexible and modular framework for the development of new applications oriented to new therapies. Entities and companies interested in commercial agreement, technical assistance or technical cooperation or a combination of some of these services are sought.
Main advantages and innovative aspects
The new system:
- Enables interaction and rehabilitation therapies performed to disabled with both motor and cognitive impairment.
- Enables multiple modes of interaction (gestures, poses / hand movement, voice, look, and touch) for people with different disabilities offering significant advantages over other systems.
- Enables customization of patient own gestures for each sensor providing a natural interaction experience with the system.
- Provides biometric identification (facial recognition) patients adapting the interaction (sensors and gestures) depending on the user disability level.
- Combines the data obtained from the sensors with 3D interfaces efficiently. The system provides a more realistic way of rehabilitation through the use of advanced virtual reality techniques.
- It provides a flexible and modular development workspace of new applications oriented to new therapies based on the different needs of patients.
- Comprises sensors and devices available on the market, therefore, can be modified, adapted and replicated easily at a reasonable cost depending on the type of patient, disabilities and therapies to implement.
According to WHO there are more than one billion people with some kind of disability in the world and about 200 million experience considerable difficulties. In the coming years, an increased number of disabled is expected causing of major concern. Mainly, this is due to the aging population, the increased risk of disability among the elderly and the global increase in chronic diseases such as diabetes, cardiovascular disease, cancer and disorders of mental health.
Among others, devices and assistive technologies increase mobility, hearing, vision and communication possibilities of disabled people. Therefore, using these technologies, they can improve their skills being able to live autonomously and participate in society.
Nowadays, there is a wide variety of devices to carry out the rehabilitation of disabled people using new technologies. For example, there are gloves based devices that enable an accurate representation of the hand movements or gestures. However, these systems are invasive, preventing natural interaction with patient free movements.
Moreover, there are other less invasive systems based on using 3D sensors to obtain the pose of patients and used for injury rehabilitation. However, these systems restrict device ways of interaction to high level gestures (arm, stooping, etc ...) limiting therefore its application to rehabilitation therapies for patients with minor disabilities or very limited mobility. We can also find systems capable of tracking the position of the eyes, despite their use in rehabilitation therapy is still very limited by requiring a very restricted position of the patient in terms of distance and position with respect to the sensor.
There are also other systems based on the use of multiple sensors but they are still unable to solve a set of problems that the system presented solves.
Researchers at Institute for Computing Research of the University of Alicante have developed a new system capable of providing a natural interaction to disabled performing motor and cognitive rehabilitation therapies. In this way, the system offers a new approach making an efficient use of new technologies and man-machine interaction sensors.
The system combines the use of various sensors. A long range sensor (such a 3D camera) that enables user identification and interaction with the system through gestures or patient own pose. This sensor also provides depth information of the scene, enabling to locate precisely the patient operation environment.
The data obtained by the long-range sensor can be combined with the use of another short-range 3D sensor that provides a precise virtual representation of users’ hands. This information is used and combined with information from other sensors to perform rehabilitation therapies requiring precise movements of hands, a pose or certain fingers gesture. This sensor is located on a swivel arm, enabling its use when needed.
Also, the system incorporates a gaze tracking sensor capable of obtaining the patient’s gaze position (eye tracking). This sensor is calibrated for each patient to find out the position of the patient's eyes regarding user interfaces displayed on the tablet screen.
The sensors mentioned are combined with a microphone array, enabling to capture users’ voice commands, and sound information of the environment.
Finally, all sensors are connected to a digital tablet that hosts the information processing and offers other means of additional interaction: interaction by contact. This mode of interaction also enables rehabilitation therapies or communication based on patient contact with the interfaces displayed on the tablet screen.
All components could be mounted on a mobile platform forming a complete system of natural interaction prepared to perform therapies with patients with different disabilities.
Figure 1.- Frontal view of prototipe
Regarding the mode of operation, the system has a working environment for developing applications that enables rehabilitation or user communication through various human-machine interfaces.
Initially, the system identifies a user in the environment leaving settings adjusted to patient’s requirements and characteristics according to their disability. This task can be carried out with a therapist or caregiver assistance who registers the user and trains the system for that patient.
The system receives gestural data as input and translates it to basic commands (up, down, left, right, etc.). In addition, the system enables patient’s own gestures customization for each patient and sensor. Thereby, the creation of a different set of poses, gestures with the body, arms, head, hand, eye movement and precise locally hand movements (finger movement) or voice commands are possible to carry out the same actions on the system. Through these actions the user can navigate the environment, options, applications, etc., to perform rehabilitation or specific communication with a specific application.
Each patient can be recognized in the system using biometric identification (facial recognition) and can also define a set of sensors and a set of custom gestures to be used depending on the type of disability. Through this customization and the proper sensors calibration, the system ensures the best user adaptation offering the most natural interface.
The system includes a set of applications enabling both interaction and patient rehabilitation. In particular, several applications taking the basic commands as input have been developed: predictive text application (convert text to speech); application motor rehabilitation (through virtual object manipulation) and cognitive rehabilitation (through guided drawing). However, this initial set of applications is extensible and adaptable according to the different needs of patients. In this way, the system provides a flexible and modular workspace for new therapies development.
The proposed system can be used to perform rehabilitation therapies for people with motor and cognitive impairments. The system also facilitates communication with others through a customized and natural interaction between patients and devices (sensors, tablet, etc.).
- Companies for disabled support systems development and trading.
- Associations or public/private entities aimed at providing support services to disabled.
- Regional or national governments in the Social Welfare and Disability areas.
Intellectual property status
This technology is protected by patent:
- Application number: 201531430
- Application date: 05/10/2015
Current development status
The system has been developed and successfully tested at laboratory. A prototype (see Figure 2) with the following characteristics is available:
- The prototype includes sensors to capture information from hand, eyes and body (head, arms, etc.) movements. The system is open and is ready to incorporate new sensors. Currently, the research group is performing new tests to include bracelet-type biorhythms sensors. They have also fully developed the platform where the sensors and tablet are placed.
- The group has developed an application of cognitive rehabilitation based on image cards. The user must select one of the displayed images. Another application developed is the search for the exit in a maze.
- The user management part of the system (identification by biometrics, selection of devices adapted to disability, and training and recognition of user gestures) is already fully developed.
Desired business relationship
Companies and organizations interested in technology use and / or commercial exploitation are sought. Possible agreements:
- License agreements (use, manufacture or marketing).
- R & D project agreement (technical cooperation) for system use and / or adaptation to other sectors.
- Subcontracting agreement (technical assistance, turnkey solutions, training, etc.)