Proceedings of the 20th International Conference on Intelligent User Interfaces - IUI '15, 2015
Peripheral physiological signals, as obtained using electrodermal activity and facial electromyog... more Peripheral physiological signals, as obtained using electrodermal activity and facial electromyography over the corrugator supercilii muscle, are explored as indicators of perceived relevance in information retrieval tasks. An experiment with 40 participants is reported, in which these physiological signals are recorded while participants perform information retrieval tasks. Appropriate feature engineering is defined, and the feature space is explored. The results indicate that features in the window of 4 to 6 seconds after the relevance judgment for electrodermal activity, and from 1 second before to 2 seconds after the relevance judgment for corrugator supercilii activity, are associated with the users' perceived relevance of information items. A classifier verified the predictive power of the features and showed up to 14% improvement predicting relevance. Our research can help the design of intelligent user interfaces for information retrieval that can detect the user's perceived relevance from physiological signals and complement or replace conventional relevance feedback.
We report research on player modeling using psychophysiology and machine learning, conducted thro... more We report research on player modeling using psychophysiology and machine learning, conducted through interdisciplinary collaboration between researchers of computer science, psychology, and game design at Aalto University, Helsinki. First, we propose the Play Patterns And eXperience (PPAX) framework to connect three levels of game experience that previously had remained largely unconnected: game design patterns, the interplay of game context with player personality or tendencies, and state-of-the-art measures of experience (both subjective and non-subjective). Second, we describe our methodology for using machine learning to categorize game events to reveal corresponding patterns, culminating in an example experiment. We discuss the relation between automatically detected event clusters and game design patterns, and provide indications on how to incorporate personality profiles of players in the analysis. This novel interdisciplinary collaboration combines basic psychophysiology research with game design patterns and machine learning, and generates new knowledge about the interplay between game experience and design.
EMOListen is a multiuser mobile biosignal sonification platform. Biosignal adaptation has gained ... more EMOListen is a multiuser mobile biosignal sonification platform. Biosignal adaptation has gained a lot of recognition in recent years both in academia and within industry. However, experimentations rarely involve multiuser systems, and they do not consider audio as the adaptation target. Biosignal sonification is a relevant domain, because audio can be used in adaptation with high temporal accuracy, can be associated to several biosignals, and can be used as a secondary stimulus, enabling long term and out-of-laboratory experimentations. Multiuser mobile sonification system can be used for creating sense of telepresence and mediated body language.
Abstract. This paper presents work conducted towards the automatic recogni-tion of negative emoti... more Abstract. This paper presents work conducted towards the automatic recogni-tion of negative emotions like boredom and frustration, induced due to the sub-ject's loss of interest during HCI. Focus was on the basic pre-requisite for the future development of systems utilizing ...
This position paper describes the work done and future research focus of Helsinki Institute for I... more This position paper describes the work done and future research focus of Helsinki Institute for Information Technology (HIIT) in Fun of Gaming (FUGA) project related to emotionally adaptive gaming. Whereas, the workshop is specifically targeted on BCI gaming application, in our ...
A game of poker is a typical example of a situation involving inperfect information: players have... more A game of poker is a typical example of a situation involving inperfect information: players have to make decisions under uncertainty. This uncertainty can evoke emotional arousal and lead the player to make irrational decisions. In this paper, we introduce the EmoPoker system, which aims at making the player aware of the arousal level by providing biofeedback. With the EmoPoker system, we expect that a poker player becomes able to control their own arousal, consequently improving their gaming performance. EmoPoker presents itself as an augmented reality application, and its design is based on the traditional game concept. In this paper we also introduce other possible use cases of biofeedback training.
Journal of Neuroscience, Psychology, and Economics, 2013
ABSTRACT Playing the poker game variant No Limit Texas Hold'em (NLHE) requires constant a... more ABSTRACT Playing the poker game variant No Limit Texas Hold'em (NLHE) requires constant and rapid decision making. Players need to continually assess the profitability of different actions with poker hands of varying strengths. Currently, little is known about how the decision-making processes required in NLHE are reflected in emotion-related physiological responses. Electrodermal activity (EDA) was measured from 29 healthy volunteers during playing a modified NLHE poker game on a computer. Predecision EDA was distinctly associated with various actions, increasing in the order of folding, calling and betting/raising. An association with poker hand strength and EDA was found. Actions taken with strong and weak hands elicited higher EDA compared with actions taken with hands of medium/uncertain strength. EDA is suggested to index the anticipated utility of poker game decisions, with anticipated utility increasing in the order of folding (no anticipated utility), calling (some anticipated utility) and betting/raising (high anticipated utility) actions.
This paper describes an affect-responsive interactive photo-frame application that offers its use... more This paper describes an affect-responsive interactive photo-frame application that offers its user a different experience with every use. It relies on visual analysis of activity levels and facial expressions of its users to select responses from a database of short video segments. This ever-growing database is automatically prepared by an offline analysis of user-uploaded videos. The resulting system matches its user's affect along dimensions of valence and arousal, and gradually adapts its response to each specific user. In an extended mode, two such systems are coupled and feed each other with visual content. The strengths This paper is based on [9]. and weaknesses of the system are assessed through a usability study, where a Wizard-of-Oz response logic is contrasted with the fully automatic system that uses affective and activity-based features, either alone, or in tandem.
This paper describes an affect-responsive interactive photo-frame application that offers its use... more This paper describes an affect-responsive interactive photo-frame application that offers its user a different experience with every use. It relies on visual analysis of activity levels and facial expressions of its users to select responses from a database of short video segments. This ever-growing database is automatically prepared by an offline analysis of user-uploaded videos. The resulting system matches its user's affect along dimensions of valence and arousal, and gradually adapts its response to each specific user. In an extended mode, two such systems are coupled and feed each other with visual content. The strengths This paper is based on [9]. and weaknesses of the system are assessed through a usability study, where a Wizard-of-Oz response logic is contrasted with the fully automatic system that uses affective and activity-based features, either alone, or in tandem.
Abstract—We develop an interactive photo-frame system in which a series of videos of a single per... more Abstract—We develop an interactive photo-frame system in which a series of videos of a single person are automatically segmented and a response logic is derived to interact with the user in real-time. The system is composed of five modules. The first module analyzes ...
Proceedings of the 22nd ACM international conference on Conference on information & knowledge management - CIKM '13, 2013
ABSTRACT We introduce interactive intent modeling, where the user directs exploratory search by p... more ABSTRACT We introduce interactive intent modeling, where the user directs exploratory search by providing feedback for estimates of search intents. The estimated intents are visualized for interaction on an Intent Radar, a novel visual interface that organizes intents onto a radial layout where relevant intents are close to the center of the visualization and similar intents have similar angles. The user can give feedback on the visualized intents, from which the system learns and visualizes improved intent estimates. We systematically evaluated the effect of the interactive intent modeling in a mixed-method task-based information seeking setting with 30 users, where we compared two interface variants for interactive intent modeling, namely intent radar and a simpler list-based interface, to a conventional search system. The results show that interactive intent modeling significantly improves users' task performance and the quality of retrieved information.
Proceedings of the 28th international conference on Human factors in computing systems - CHI '10, 2010
ABSTRACT To understand how implicit and explicit biofeedback work in games, we developed a first-... more ABSTRACT To understand how implicit and explicit biofeedback work in games, we developed a first-person shooter (FPS) game to experiment with different biofeedback techniques. While this area has seen plenty of discussion, there is little rigorous experimentation addressing ...
This paper discusses a specific customization technology – Psychological Customization - which en... more This paper discusses a specific customization technology – Psychological Customization - which enables the customization of information presented on a computer-based system in real-time and its application to manipulating emotions when playing computer games. The possibilities of customizing different elements of games to manipulate emotions are presented and a definition of emotionally adaptive games is given. A psychophysiologically adaptive game
In daily life, we often copy the gestures and expressions of those we communicate with, but recen... more In daily life, we often copy the gestures and expressions of those we communicate with, but recent evidence shows that such mimicry has a physiological counterpart: interaction elicits linkage, which is a concordance between the biological signals of those involved. To find out how the type of social interaction affects linkage, pairs of participants played a turn-based computer game in which the level of competition was systematically varied between cooperation and competition. Linkage in the beta and gamma frequency bands was observed in the EEG, especially when the participants played directly against each other. Emotional expression, measured using facial EMG, reflected this pattern, with the most competitive condition showing enhanced linkage over the facial muscle-regions involved in smiling. These effects were found to be related to self-reported social presence: linkage in positive emotional expression was associated with self-reported shared negative feelings. The observed effects confirmed the hypothesis that the social context affected the degree to which participants had similar reactions to their environment and consequently showed similar patterns of brain activity. We discuss the functional resemblance between linkage, as an indicator of a shared physiology and affect, and the well-known mirror neuron system, and how they relate to social functions like empathy.
Proceedings of the 2012 ACM Conference on Ubiquitous Computing - UbiComp '12, 2012
Abstract Advanced interactive visualization such as in virtual environments and ubiquitous intera... more Abstract Advanced interactive visualization such as in virtual environments and ubiquitous interaction paradigms pose new challenges and opportunities in considering real-time responses to subliminal cues. In this paper, we propose a synthetic reality platform that, combined with psychophysiological recordings, enables us to study in realtime the effects of various subliminal cues. We endeavor to integrate various aspects known to be relevant to implicit perception. The context is of consumer experience and choice of an artifact where ...
Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces - ITS '13, 2013
ABSTRACT Large screens are populating a variety of settings motivating research on appropriate in... more ABSTRACT Large screens are populating a variety of settings motivating research on appropriate interaction techniques. While gesture is popularized by depth cameras we contribute with a comparison study showing how eye pointing is a valuable substitute to gesture pointing in dragging tasks. We compare eye pointing combined with gesture selection to gesture pointing and selection. Results clearly show that eye pointing combined with a selection gesture allows more accurate and faster dragging.
Abstract Advanced interactive visualization such as in virtual environments and ubiquitous intera... more Abstract Advanced interactive visualization such as in virtual environments and ubiquitous interaction paradigms pose new challenges and opportunities in considering real-time responses to subliminal cues. In this paper, we propose a synthetic reality platform that, combined with psychophysiological recordings, enables us to study in realtime the effects of various subliminal cues. We endeavor to integrate various aspects known to be relevant to implicit perception. The context is of consumer experience and choice of an artifact where ...
Proceedings of the 20th International Conference on Intelligent User Interfaces - IUI '15, 2015
Peripheral physiological signals, as obtained using electrodermal activity and facial electromyog... more Peripheral physiological signals, as obtained using electrodermal activity and facial electromyography over the corrugator supercilii muscle, are explored as indicators of perceived relevance in information retrieval tasks. An experiment with 40 participants is reported, in which these physiological signals are recorded while participants perform information retrieval tasks. Appropriate feature engineering is defined, and the feature space is explored. The results indicate that features in the window of 4 to 6 seconds after the relevance judgment for electrodermal activity, and from 1 second before to 2 seconds after the relevance judgment for corrugator supercilii activity, are associated with the users' perceived relevance of information items. A classifier verified the predictive power of the features and showed up to 14% improvement predicting relevance. Our research can help the design of intelligent user interfaces for information retrieval that can detect the user's perceived relevance from physiological signals and complement or replace conventional relevance feedback.
We report research on player modeling using psychophysiology and machine learning, conducted thro... more We report research on player modeling using psychophysiology and machine learning, conducted through interdisciplinary collaboration between researchers of computer science, psychology, and game design at Aalto University, Helsinki. First, we propose the Play Patterns And eXperience (PPAX) framework to connect three levels of game experience that previously had remained largely unconnected: game design patterns, the interplay of game context with player personality or tendencies, and state-of-the-art measures of experience (both subjective and non-subjective). Second, we describe our methodology for using machine learning to categorize game events to reveal corresponding patterns, culminating in an example experiment. We discuss the relation between automatically detected event clusters and game design patterns, and provide indications on how to incorporate personality profiles of players in the analysis. This novel interdisciplinary collaboration combines basic psychophysiology research with game design patterns and machine learning, and generates new knowledge about the interplay between game experience and design.
EMOListen is a multiuser mobile biosignal sonification platform. Biosignal adaptation has gained ... more EMOListen is a multiuser mobile biosignal sonification platform. Biosignal adaptation has gained a lot of recognition in recent years both in academia and within industry. However, experimentations rarely involve multiuser systems, and they do not consider audio as the adaptation target. Biosignal sonification is a relevant domain, because audio can be used in adaptation with high temporal accuracy, can be associated to several biosignals, and can be used as a secondary stimulus, enabling long term and out-of-laboratory experimentations. Multiuser mobile sonification system can be used for creating sense of telepresence and mediated body language.
Abstract. This paper presents work conducted towards the automatic recogni-tion of negative emoti... more Abstract. This paper presents work conducted towards the automatic recogni-tion of negative emotions like boredom and frustration, induced due to the sub-ject's loss of interest during HCI. Focus was on the basic pre-requisite for the future development of systems utilizing ...
This position paper describes the work done and future research focus of Helsinki Institute for I... more This position paper describes the work done and future research focus of Helsinki Institute for Information Technology (HIIT) in Fun of Gaming (FUGA) project related to emotionally adaptive gaming. Whereas, the workshop is specifically targeted on BCI gaming application, in our ...
A game of poker is a typical example of a situation involving inperfect information: players have... more A game of poker is a typical example of a situation involving inperfect information: players have to make decisions under uncertainty. This uncertainty can evoke emotional arousal and lead the player to make irrational decisions. In this paper, we introduce the EmoPoker system, which aims at making the player aware of the arousal level by providing biofeedback. With the EmoPoker system, we expect that a poker player becomes able to control their own arousal, consequently improving their gaming performance. EmoPoker presents itself as an augmented reality application, and its design is based on the traditional game concept. In this paper we also introduce other possible use cases of biofeedback training.
Journal of Neuroscience, Psychology, and Economics, 2013
ABSTRACT Playing the poker game variant No Limit Texas Hold'em (NLHE) requires constant a... more ABSTRACT Playing the poker game variant No Limit Texas Hold'em (NLHE) requires constant and rapid decision making. Players need to continually assess the profitability of different actions with poker hands of varying strengths. Currently, little is known about how the decision-making processes required in NLHE are reflected in emotion-related physiological responses. Electrodermal activity (EDA) was measured from 29 healthy volunteers during playing a modified NLHE poker game on a computer. Predecision EDA was distinctly associated with various actions, increasing in the order of folding, calling and betting/raising. An association with poker hand strength and EDA was found. Actions taken with strong and weak hands elicited higher EDA compared with actions taken with hands of medium/uncertain strength. EDA is suggested to index the anticipated utility of poker game decisions, with anticipated utility increasing in the order of folding (no anticipated utility), calling (some anticipated utility) and betting/raising (high anticipated utility) actions.
This paper describes an affect-responsive interactive photo-frame application that offers its use... more This paper describes an affect-responsive interactive photo-frame application that offers its user a different experience with every use. It relies on visual analysis of activity levels and facial expressions of its users to select responses from a database of short video segments. This ever-growing database is automatically prepared by an offline analysis of user-uploaded videos. The resulting system matches its user's affect along dimensions of valence and arousal, and gradually adapts its response to each specific user. In an extended mode, two such systems are coupled and feed each other with visual content. The strengths This paper is based on [9]. and weaknesses of the system are assessed through a usability study, where a Wizard-of-Oz response logic is contrasted with the fully automatic system that uses affective and activity-based features, either alone, or in tandem.
This paper describes an affect-responsive interactive photo-frame application that offers its use... more This paper describes an affect-responsive interactive photo-frame application that offers its user a different experience with every use. It relies on visual analysis of activity levels and facial expressions of its users to select responses from a database of short video segments. This ever-growing database is automatically prepared by an offline analysis of user-uploaded videos. The resulting system matches its user's affect along dimensions of valence and arousal, and gradually adapts its response to each specific user. In an extended mode, two such systems are coupled and feed each other with visual content. The strengths This paper is based on [9]. and weaknesses of the system are assessed through a usability study, where a Wizard-of-Oz response logic is contrasted with the fully automatic system that uses affective and activity-based features, either alone, or in tandem.
Abstract—We develop an interactive photo-frame system in which a series of videos of a single per... more Abstract—We develop an interactive photo-frame system in which a series of videos of a single person are automatically segmented and a response logic is derived to interact with the user in real-time. The system is composed of five modules. The first module analyzes ...
Proceedings of the 22nd ACM international conference on Conference on information & knowledge management - CIKM '13, 2013
ABSTRACT We introduce interactive intent modeling, where the user directs exploratory search by p... more ABSTRACT We introduce interactive intent modeling, where the user directs exploratory search by providing feedback for estimates of search intents. The estimated intents are visualized for interaction on an Intent Radar, a novel visual interface that organizes intents onto a radial layout where relevant intents are close to the center of the visualization and similar intents have similar angles. The user can give feedback on the visualized intents, from which the system learns and visualizes improved intent estimates. We systematically evaluated the effect of the interactive intent modeling in a mixed-method task-based information seeking setting with 30 users, where we compared two interface variants for interactive intent modeling, namely intent radar and a simpler list-based interface, to a conventional search system. The results show that interactive intent modeling significantly improves users' task performance and the quality of retrieved information.
Proceedings of the 28th international conference on Human factors in computing systems - CHI '10, 2010
ABSTRACT To understand how implicit and explicit biofeedback work in games, we developed a first-... more ABSTRACT To understand how implicit and explicit biofeedback work in games, we developed a first-person shooter (FPS) game to experiment with different biofeedback techniques. While this area has seen plenty of discussion, there is little rigorous experimentation addressing ...
This paper discusses a specific customization technology – Psychological Customization - which en... more This paper discusses a specific customization technology – Psychological Customization - which enables the customization of information presented on a computer-based system in real-time and its application to manipulating emotions when playing computer games. The possibilities of customizing different elements of games to manipulate emotions are presented and a definition of emotionally adaptive games is given. A psychophysiologically adaptive game
In daily life, we often copy the gestures and expressions of those we communicate with, but recen... more In daily life, we often copy the gestures and expressions of those we communicate with, but recent evidence shows that such mimicry has a physiological counterpart: interaction elicits linkage, which is a concordance between the biological signals of those involved. To find out how the type of social interaction affects linkage, pairs of participants played a turn-based computer game in which the level of competition was systematically varied between cooperation and competition. Linkage in the beta and gamma frequency bands was observed in the EEG, especially when the participants played directly against each other. Emotional expression, measured using facial EMG, reflected this pattern, with the most competitive condition showing enhanced linkage over the facial muscle-regions involved in smiling. These effects were found to be related to self-reported social presence: linkage in positive emotional expression was associated with self-reported shared negative feelings. The observed effects confirmed the hypothesis that the social context affected the degree to which participants had similar reactions to their environment and consequently showed similar patterns of brain activity. We discuss the functional resemblance between linkage, as an indicator of a shared physiology and affect, and the well-known mirror neuron system, and how they relate to social functions like empathy.
Proceedings of the 2012 ACM Conference on Ubiquitous Computing - UbiComp '12, 2012
Abstract Advanced interactive visualization such as in virtual environments and ubiquitous intera... more Abstract Advanced interactive visualization such as in virtual environments and ubiquitous interaction paradigms pose new challenges and opportunities in considering real-time responses to subliminal cues. In this paper, we propose a synthetic reality platform that, combined with psychophysiological recordings, enables us to study in realtime the effects of various subliminal cues. We endeavor to integrate various aspects known to be relevant to implicit perception. The context is of consumer experience and choice of an artifact where ...
Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces - ITS '13, 2013
ABSTRACT Large screens are populating a variety of settings motivating research on appropriate in... more ABSTRACT Large screens are populating a variety of settings motivating research on appropriate interaction techniques. While gesture is popularized by depth cameras we contribute with a comparison study showing how eye pointing is a valuable substitute to gesture pointing in dragging tasks. We compare eye pointing combined with gesture selection to gesture pointing and selection. Results clearly show that eye pointing combined with a selection gesture allows more accurate and faster dragging.
Abstract Advanced interactive visualization such as in virtual environments and ubiquitous intera... more Abstract Advanced interactive visualization such as in virtual environments and ubiquitous interaction paradigms pose new challenges and opportunities in considering real-time responses to subliminal cues. In this paper, we propose a synthetic reality platform that, combined with psychophysiological recordings, enables us to study in realtime the effects of various subliminal cues. We endeavor to integrate various aspects known to be relevant to implicit perception. The context is of consumer experience and choice of an artifact where ...
Uploads
Papers by Ilkka Kosunen