It is possible to interface commercial off-the-shelf (COTS) EEG equipment with 3D virtual environments. In a student project I am supervising, we use the Emotiv Epoc EEG headset and interface it with the 3D game engine Unity. Now that we have a live communication channel established by means of UDP, there are virtually unlimited experiments that can be designed inside Unity. The EEG signals can be used to control objects in a 3D world (e.g., move a character), to affect screenplay based on emotion (e.g., when a user becomes bored as determined from EEG readings, give him a scare!), and so on. Both EEG and the 3D virtual environment may be further linked to real physical devices, such as a robot arm. Rehabilitation and training of both healthy and non-healthy participants is possible. Suitable experiments can be designed to aid in reverse-engineering the human brain and work out how the brain does things, such as sensing, planning, and control.
What do you think should be investigated?
http://www.emotiv.com
http://unity3d.com
Adaptive gameplay is one immediate use that comes to mind, both as fun factor in games and as a professional tool for training personnel.
In 3D computer games, the storyline can adapt to the player's emotions and attention. Certain states of mind can be use to trigger events in the game. The storyline becomes nonlinear and unpredictable for the player.
In training personnel, EEG signals can be used in 3D simulators, such as those for advanced marine operations that we use here at AAUC, to trigger certain events depending on the participants brain activity. For example, if we are able to determine that a participant is drowsy or less focused from his EEG, then we can induce an event such a simulated accident and observe how the participant react. Such results can then be compared with other participants with different states of mind. Not only can this improve the actual training process, it can also tell us more about limitations of the human brain when it comes to stressful situations, and how they can be avoided (in time), for example through cockpit design, redundancy in crew and safety measures, and so on.
Dear Prof. Ilankovic, thanks for your reply! Given your expertise in the field, do not hesitate if you have any advice or suggestions for experiments!
Anders' idea can of course be generalised to any kind of control where anti-tremor stabilisation is needed. 8-12 Hz physiological tremor is inherent in all of us and likely central in origin. Tremor cancellation algorithms or automatic triggering can likely benefit from real-time EEG signals and a virtual 3D environment can be used to setup at number if different situations to test the solutions.
We are planing to do a research on triggering facial expressions of avatars of CVEs based on EEG signal-based emotions.
Hello Robin, this post is quite 'old' but I'm curious about the progress of your research. As you mentioned there are virtually unlimited experiments...one application that would make sense to me is to analyse in a virtual learning environment whether the student is in the state of flow (a good sign of engagement). Another idea, as I think someone else already mentioned above, is to personalize the virtual environment to user's reactions. Please share the research progress so far. Thanks.
Hi Tito, you can check out our two papers at ECMS '16 on usage of EEG for stroke patient rehabilitation and for electric wheelchair control.
- Badges
- Science topic