Reality-based brain-computer interaction

Proc. CHI 2009 Workshop on …, 2009

We have developed a system where the combination of functional brain imaging (fMRI) and Virtual Reality (VR) can be used to study and evaluate user experience based on brain activation and models of cognitive neuroscience. The ability to study the brain during natural interaction with an (ecologically valid) environment has great potential for several areas of research and development, including evaluation of Reality-Based Interaction (RBI). The RBI concept of tradeoffs is of particular interest since we want to further explore the relation between how the brain works with an accepted reality and what happens when this reality is disrupted. We present the system with an overview of conducted studies to illustrate capabilities and feasibility. In particular, feasibility is supported by the fact that the brain activations seen in these studies match expectations based on existing literature. Further discussion elaborates on the relation to RBI and evaluation; and finally some possible future work is presented.

WSEAS TRANSACTIONS on INFORMATION SCIENCE and APPLICATIONS, 2023

Brain-computer interfaces (BCIs) can use data from non-invasive electroencephalogram (EEG) to transform different brain signals into binary code, often aiming to gain control utility of an end-effector (e.g mouse cursor). In the past several years, advances in wearable and immersive technologies have made it possible to integrate EEG with virtual reality (VR) headsets. These advances have enabled a new generation of user studies that help researchers improve understanding of various issues in current VR design (e.g. cybersickness and locomotion). The main challenge for integrating EEG-based BCIs into VR environments is to develop communication architectures that deliver robust, reliable and lossless data flows. Furthermore, user comfort and near real-time interactivity create additional challenges. We conducted two experiments in which a consumer-grade EEG headband (Muse2) was utilized to assess the feasibility of an EEG-based BCI in virtual environments. We first conducted a pilot experiment that consisted of a simple task of object re-scaling inside the VR space using focus values generated from the user's EEG. The subsequent study experiment consisted of two groups (control and experimental) performing two tasks: telekinesis and teleportation. Our user research study shows the viability of EEG for real-time interactions in non-serious applications such as games. We further suggest that a simplified way of calculating the mean EEG values is adequate for this type of use. We , in addition, discuss the findings to help improve the design of user research studies that deploy similar EEGbased BCIs in VR environments.

2019 9th International IEEE/EMBS Conference on Neural Engineering (NER)

We study the performance of brain computer interface (BCI) system in a virtual reality (VR) environment and compare it to 2D regular displays. First, we design a headset that consists of three components: a wearable electroencephalography (EEG) device, a VR headset and an interface. Recordings of brain and behavior from human subjects, performing a wide variety of tasks using our device are collected. The tasks consist of object rotation or scaling in VR using either mental commands or facial expression (smile and eyebrow movement). Subjects are asked to repeat similar tasks on regular 2D monitor screens. The performance in 3-D virtual reality environment is considerably higher compared to the to the 2D screen. Particularly, the median number of success rate across trials for VR setting is double of that for the 2D setting (8 successful command in VR setting compared to 4 successful command in 2D screen in 1 minute trials). Our results suggest that the design of future BCI systems can remarkably benefit from the VR setting.

Proceedings of the …, 2010

The combination of virtual reality (VR) and brain measurements is a promising development of HCI, but the maturation of this paradigm requires more knowledge about how brain activity is influenced by parameters of VR applications. To this end we investigate the influence of two prominent VR parameters, 3d-motion and interactivity, while brain activity is measured for a mental rotation task, using functional MRI (fMRI). A mental rotation network of brain areas is identified, matching previous results. The addition of interactivity increases the activation in core areas of this network, with more profound effects in frontal and preparatory motor areas. The increases from 3d-motion are restricted to primarily visual areas. We relate these effects to emerging theories of cognition and potential applications for brain-computer interfaces (BCIs). Our results demonstrate one way to provoke increased activity in task-relevant areas, making it easier to detect and use for adaptation and development of HCI.

Brain Sciences

The synergy of perceptual psychology, technology, and neuroscience can be used to comprehend how virtual reality affects cognition of human brain. Numerous studies have used neuroimaging modalities to assess the cognitive state and response of the brain with various external stimulations. The virtual reality-based devices are well known to incur visual, auditory, and haptic induced perceptions. Neurophysiological recordings together with virtual stimulations can assist in correlating humans’ physiological perception with response in the environment designed virtually. The effective combination of these two has been utilized to study human behavior, spatial navigation performance, and spatial presence, to name a few. Moreover, virtual reality-based devices can be evaluated for the neurophysiological correlates of cognition through neurophysiological recordings. Challenges exist in the integration of real-time neuronal signals with virtual reality-based devices, and enhancing the expe...

Human-Computer Interaction, 2010

We have integrated the Graz brain-computer interface (BCI) system with a highly-immersive virtual reality (VR) Cave-like system. This setting allows for a new type of experience, whereby participants can control a virtual world using imagination of movement. However, current BCI systems still have many limitations. In this paper we present two experiments exploring the different constraints posed by current BCI systems when used in VR.

2008

Functional MR imaging (fMRI) enables to detect different activated brain areas according to the performed tasks. However, data are usually evaluated after the experiment, which prohibits intra-experiment optimization or more sophisticated applications such as biofeedback experiments. Using a human-brain-interface (HBI), subjects are able to communicate with external programs, e.g. to navigate through virtual scenes, 1 or to experience and modify their own brain activation. 2 These applications require the real-time analysis and classification of activated brain areas. Our paper presents first results of different strategies for real-time pattern analysis and classification realized within a flexible experiment control system that enables the volunteers to move through a 3D virtual scene in real-time using finger tapping tasks, and alternatively only thought-based tasks.

Journal of Neuroscience Methods, 2012

We develop informatical concepts, which allow the creation of own VR-fMRI paradigms. We develop neuroinformatical techniques which provide real-time VR-fMRI studies. We embed an easy-to-handle integration concept for virtual environment files. We validate the application in a real-time VR-fMRI study with spatial memory topic. Subjects indicate higher interaction and more attention than in common fMRI studies.

Computer, 2000