Background: Patients who receive rehabilitation after hip replacement surgery are shown to have i... more Background: Patients who receive rehabilitation after hip replacement surgery are shown to have increased muscle strength and better functional performance. However, traditional physiotherapy is often tedious and leads to poor adherence. Exercise games, provide ways for increasing the engagement of elderly patients and increase the uptake of rehabilitation exercises. Objective: The objective of this study was to evaluate Fietsgame (Dutch for cycling game), which translates existing rehabilitation exercises into fun exercise games. The system connects exercise games with a patient's personal record and a therapist interface by an Internet of Things server. Thus, both the patient and physiotherapist can monitor the patient's medical status. Methods: This paper describes a pilot study that evaluates the usability of the Fietsgame. The study was conducted in a rehabilitation center with 9 participants, including 2 physiotherapists and 7 patients. The patients were asked to play 6 exercise games, each lasting about 5 min, under the guidance of a physiotherapist. The mean age of the patients was 74.57 years (standard deviation [SD] 8.28); all the patients were in the recovery process after hip surgery. Surveys were developed to quantitatively measure the usability factors, including presence, enjoyment, pain, exertion, and technology acceptance. Comments on advantages and suggested improvements of our game system provided by the physiotherapists and patients were summarized and their implications were discussed. Results: The results showed that after successfully playing the games, 75% to 100% of the patients experienced high levels of enjoyment in all the games except the squats game. Patients reported the highest level of exertion in squats when compared with other exercise games. Lunges resulted in the highest dropout rate (43%) due to interference with the Kinect v2 from support chairs. All the patients (100%) found the game system useful and easy to use, felt that it would be a useful tool in their further rehabilitation, and expressed that they would like to use the game in the future. The therapists indicated that the exercise games highly meet the criteria of motor rehabilitation, and they intend to continue using the game as part of their rehabilitation treatment of patients. Comments from the patients and physiotherapists suggest that real-time corrective feedback when patients perform the exercises wrongly and a more personalized user interface with options for increasing or decreasing cognitive load are needed. Conclusions: The results suggest that Fietsgame can be used as an alternative tool to traditional motor rehabilitation for patients with hip surgery. Lunges and squats are found to be more beneficial for patients who have relatively better balance skills. A follow-up randomized controlled study will be conducted to test the effectiveness of the Fietsgame to investigate how motivating it is over a longer period of time.
This article provides a comprehensive overview of artificial intelligence (AI) for serious games.... more This article provides a comprehensive overview of artificial intelligence (AI) for serious games. Reporting about the work of a European flagship project on serious game technologies, it presents a set of advanced game AI components that enable pedagogical affordances and that can be easily reused across a wide diversity of game engines and game platforms. Serious game AI functionalities include player modelling (realtime facial emotion recognition, automated difficulty adaptation, stealth assessment), natural language processing (sentiment analysis and essay scoring on free texts), and believable non-playing characters (emotional and socio-cultural, non-verbal bodily motion, and lip-synchronised speech), respectively. The reuse of these components enables game developers to develop high quality serious games at reduced costs and in shorter periods of time. All these components are open source software and can be freely downloaded from the newly launched portal at gamecomponents.eu. The components come with detailed installation manuals and tutorial videos. All components have been applied and validated in serious games that were tested with real end-users.
We present a procedural audio-driven speech animation method for interactive virtual characters. ... more We present a procedural audio-driven speech animation method for interactive virtual characters. Given any audio with its respective speech transcript, we automatically generate lip-synchronized speech animation that could drive any three-dimensional virtual character. The realism of the animation is enhanced by studying the emotional features of the audio signal and its effect on mouth movements. We also propose a coarticulation model that takes into account various linguistic rules. The generated animation is configurable by the user by modifying the control parameters, such as viseme types, intensities, and coarticulation curves. We compare our approach against two lip-synchronized speech animation generators. Our results show that our method surpasses them in terms of user preference. KEYWORDS audio-driven speech animation, emotional speech, procedural animation This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial, and no modifications or adaptations are made.
Proceedings of the 19th Acm Symposium, Oct 6, 2013
3D virtual humans and physical human-like robots can be used to interact with people in a remote ... more 3D virtual humans and physical human-like robots can be used to interact with people in a remote location in order to increase the feeling of presence. In a tele-presence setup their behaviors are driven by real participants. We envision that in the absence of the real users, when they have to leave or they do not want to do a repetitive task, the control of the robots can be handed to an artificial intelligence component to sustain the ongoing interaction. The point when human-mediated interaction is required again, the control can be given back to the real users. One of the main challenges in tele-presence research is the adaptation of 3D
Constraint programming is an approach for solving mostly combinatorial problems by declaratively ... more Constraint programming is an approach for solving mostly combinatorial problems by declaratively describing the problem and using special solving algorithms. Restrictions called constraints are stated over the problem variables that reduces the values each variable can take. In some cases it is not possible to satisfy all constraints or the user can state some preferences on them. One of the
Proceedings of the 19th ACM Symposium on Virtual Reality Software and Technology - VRST '13, 2013
3D virtual humans and physical human-like robots can be used to interact with people in a remote ... more 3D virtual humans and physical human-like robots can be used to interact with people in a remote location in order to increase the feeling of presence. In a tele-presence setup their behaviors are driven by real participants. We envision that in the absence of the real users, when they have to leave or they do not want to do a repetitive task, the control of the robots can be handed to an artificial intelligence component to sustain the ongoing interaction. The point when human-mediated interaction is required again, the control can be given back to the real users. One of the main challenges in tele-presence research is the adaptation of 3D
SIGGRAPH Asia 2014 Autonomous Virtual Humans and Social Robot for Telepresence on - SIGGRAPH ASIA '14, 2014
To give human-like capabilities to artificial characters, we should equip them with the ability o... more To give human-like capabilities to artificial characters, we should equip them with the ability of inferring user states. These artificial characters should understand the users' behaviors through various sensors and respond back using multimodal output. Besides natural multimodal interaction, they should also be able to communicate with multiple users and among each other in multiparty interactions. Previous work on interactive virtual humans and social robots mainly focuses on one-to-one interactions. In this paper, we study tracking and fusion aspects of multiparty interactions. We first give a general overview of our proposed multiparty interaction system and mention how it is different from previous work. Then, we provide the details of the tracking and fusion component including speaker identification, addressee detection and a dynamic user entrance/leave mechanism based on user re-identification using a Kinect sensor. Finally, we present a case study with the system and provide a discussion on the current capabilities, limitations and future work.
Presence: Teleoperators and Virtual Environments, 2014
3D virtual humans and physical human-like robots can be used to interact with people in a remote ... more 3D virtual humans and physical human-like robots can be used to interact with people in a remote location in order to increase the feeling of presence. In a telepresence setup, their behaviors are driven by real participants. We envision that in the absence of the real users, when they have to leave or they do not want to do a repetitive task, the control of the robots can be handed to an artificial intelligence component to sustain the ongoing interaction. At the point when human-mediated interaction is required again, control can be returned to the real users. One of the main challenges in telepresence research is the adaptation of 3D position and orientation of the remote participants to the actual physical environment to have appropriate eye contact and gesture awareness in a group conversation. In case the human behind the robot and/or virtual human leaves, multi-party interaction should be handed to an artificial intelligence component. In this paper, we discuss the challenges...
Personality and emotion modeling is being used recently in order to create variation in the behav... more Personality and emotion modeling is being used recently in order to create variation in the behaviour of virtual characters and social robots and to make them socially intelligent for more engaged interaction with people. For these characters to be everyday social actors, it is also important that they can remember and maintain their feelings and establish unique emotional interpersonal relationships
Constraint programming is an approach for solving mostly combinatorial problems by declaratively ... more Constraint programming is an approach for solving mostly combinatorial problems by declaratively describing the problem and using special solving algorithms. Restrictions called constraints are stated over the problem variables that reduces the values each variable can take. In some cases it is not possible to satisfy all constraints or the user can state some preferences on them. One of the
The cost of health care is expected to grow enormously in the coming years. To keep these costs l... more The cost of health care is expected to grow enormously in the coming years. To keep these costs limited, we need better technological tools for self-monitoring and independent aging, and to put a stronger emphasis on wellness, defined as a balanced and healthy lifestyle that avoids diseases at every stage of life. In contrast to traditional medical science that examines patients in laboratories, wellness is determined by our activities, which have so far been difficult to measure objectively. This situation is changing due to the availability of wearable and pervasive sensors, smartphones and other body-worn devices that can capture and infer users activities. Technological advances in this area are rapidly pushing forward. The range of methods as well as the health domains where sensors have been used for monitoring is proliferating. This has created a unique opportunity to understand wellness in an integrative and balanced framework. The goal of this survey is to present the current technological capabilities in sensor technology and activitiy detection methods with the purpose of guiding researchers that are interested in developing wellness technology.
2009 International Conference on CyberWorlds, 2009
Computers and video games are more and more used for serious purposes such as education and healt... more Computers and video games are more and more used for serious purposes such as education and health care. Virtual humans are the inhabitants of the simulated worlds in these games and the challenge is not only to make them look like humans but also to make them behave as humans. Thus, they should be equipped with properties such as social
With the recent advances, today people are able to communicate with embodied (virtual/robotic) en... more With the recent advances, today people are able to communicate with embodied (virtual/robotic) entities using natural ways of communication. In order to use them in our daily lives, they need to be intelligent enough to make long-term relationships with us and this is highly challenging. Previous work on long-term interaction frequently reported that after the novelty effect disappeared, users' interest into the interaction decreased with time. Our primary goal in this study was to develop a system that can still keep the attention of the users after the first interaction. Incorporating the notion of time, we think that the key to long-term interaction is the recall of past memories during current conversation. For this purpose, we developed a longterm interaction framework with remembering and dialogue planning capability. In order to see the effect of remembering on users, we designed a tutoring application and measured the changes in social presence and task engagement levels according to the existence of memory. Different from previous work, users' interest in our system did not decrease with time with the important contributions of remembering to the engagement level of users.
Background: Patients who receive rehabilitation after hip replacement surgery are shown to have i... more Background: Patients who receive rehabilitation after hip replacement surgery are shown to have increased muscle strength and better functional performance. However, traditional physiotherapy is often tedious and leads to poor adherence. Exercise games, provide ways for increasing the engagement of elderly patients and increase the uptake of rehabilitation exercises. Objective: The objective of this study was to evaluate Fietsgame (Dutch for cycling game), which translates existing rehabilitation exercises into fun exercise games. The system connects exercise games with a patient's personal record and a therapist interface by an Internet of Things server. Thus, both the patient and physiotherapist can monitor the patient's medical status. Methods: This paper describes a pilot study that evaluates the usability of the Fietsgame. The study was conducted in a rehabilitation center with 9 participants, including 2 physiotherapists and 7 patients. The patients were asked to play 6 exercise games, each lasting about 5 min, under the guidance of a physiotherapist. The mean age of the patients was 74.57 years (standard deviation [SD] 8.28); all the patients were in the recovery process after hip surgery. Surveys were developed to quantitatively measure the usability factors, including presence, enjoyment, pain, exertion, and technology acceptance. Comments on advantages and suggested improvements of our game system provided by the physiotherapists and patients were summarized and their implications were discussed. Results: The results showed that after successfully playing the games, 75% to 100% of the patients experienced high levels of enjoyment in all the games except the squats game. Patients reported the highest level of exertion in squats when compared with other exercise games. Lunges resulted in the highest dropout rate (43%) due to interference with the Kinect v2 from support chairs. All the patients (100%) found the game system useful and easy to use, felt that it would be a useful tool in their further rehabilitation, and expressed that they would like to use the game in the future. The therapists indicated that the exercise games highly meet the criteria of motor rehabilitation, and they intend to continue using the game as part of their rehabilitation treatment of patients. Comments from the patients and physiotherapists suggest that real-time corrective feedback when patients perform the exercises wrongly and a more personalized user interface with options for increasing or decreasing cognitive load are needed. Conclusions: The results suggest that Fietsgame can be used as an alternative tool to traditional motor rehabilitation for patients with hip surgery. Lunges and squats are found to be more beneficial for patients who have relatively better balance skills. A follow-up randomized controlled study will be conducted to test the effectiveness of the Fietsgame to investigate how motivating it is over a longer period of time.
This article provides a comprehensive overview of artificial intelligence (AI) for serious games.... more This article provides a comprehensive overview of artificial intelligence (AI) for serious games. Reporting about the work of a European flagship project on serious game technologies, it presents a set of advanced game AI components that enable pedagogical affordances and that can be easily reused across a wide diversity of game engines and game platforms. Serious game AI functionalities include player modelling (realtime facial emotion recognition, automated difficulty adaptation, stealth assessment), natural language processing (sentiment analysis and essay scoring on free texts), and believable non-playing characters (emotional and socio-cultural, non-verbal bodily motion, and lip-synchronised speech), respectively. The reuse of these components enables game developers to develop high quality serious games at reduced costs and in shorter periods of time. All these components are open source software and can be freely downloaded from the newly launched portal at gamecomponents.eu. The components come with detailed installation manuals and tutorial videos. All components have been applied and validated in serious games that were tested with real end-users.
We present a procedural audio-driven speech animation method for interactive virtual characters. ... more We present a procedural audio-driven speech animation method for interactive virtual characters. Given any audio with its respective speech transcript, we automatically generate lip-synchronized speech animation that could drive any three-dimensional virtual character. The realism of the animation is enhanced by studying the emotional features of the audio signal and its effect on mouth movements. We also propose a coarticulation model that takes into account various linguistic rules. The generated animation is configurable by the user by modifying the control parameters, such as viseme types, intensities, and coarticulation curves. We compare our approach against two lip-synchronized speech animation generators. Our results show that our method surpasses them in terms of user preference. KEYWORDS audio-driven speech animation, emotional speech, procedural animation This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial, and no modifications or adaptations are made.
Proceedings of the 19th Acm Symposium, Oct 6, 2013
3D virtual humans and physical human-like robots can be used to interact with people in a remote ... more 3D virtual humans and physical human-like robots can be used to interact with people in a remote location in order to increase the feeling of presence. In a tele-presence setup their behaviors are driven by real participants. We envision that in the absence of the real users, when they have to leave or they do not want to do a repetitive task, the control of the robots can be handed to an artificial intelligence component to sustain the ongoing interaction. The point when human-mediated interaction is required again, the control can be given back to the real users. One of the main challenges in tele-presence research is the adaptation of 3D
Constraint programming is an approach for solving mostly combinatorial problems by declaratively ... more Constraint programming is an approach for solving mostly combinatorial problems by declaratively describing the problem and using special solving algorithms. Restrictions called constraints are stated over the problem variables that reduces the values each variable can take. In some cases it is not possible to satisfy all constraints or the user can state some preferences on them. One of the
Proceedings of the 19th ACM Symposium on Virtual Reality Software and Technology - VRST '13, 2013
3D virtual humans and physical human-like robots can be used to interact with people in a remote ... more 3D virtual humans and physical human-like robots can be used to interact with people in a remote location in order to increase the feeling of presence. In a tele-presence setup their behaviors are driven by real participants. We envision that in the absence of the real users, when they have to leave or they do not want to do a repetitive task, the control of the robots can be handed to an artificial intelligence component to sustain the ongoing interaction. The point when human-mediated interaction is required again, the control can be given back to the real users. One of the main challenges in tele-presence research is the adaptation of 3D
SIGGRAPH Asia 2014 Autonomous Virtual Humans and Social Robot for Telepresence on - SIGGRAPH ASIA '14, 2014
To give human-like capabilities to artificial characters, we should equip them with the ability o... more To give human-like capabilities to artificial characters, we should equip them with the ability of inferring user states. These artificial characters should understand the users' behaviors through various sensors and respond back using multimodal output. Besides natural multimodal interaction, they should also be able to communicate with multiple users and among each other in multiparty interactions. Previous work on interactive virtual humans and social robots mainly focuses on one-to-one interactions. In this paper, we study tracking and fusion aspects of multiparty interactions. We first give a general overview of our proposed multiparty interaction system and mention how it is different from previous work. Then, we provide the details of the tracking and fusion component including speaker identification, addressee detection and a dynamic user entrance/leave mechanism based on user re-identification using a Kinect sensor. Finally, we present a case study with the system and provide a discussion on the current capabilities, limitations and future work.
Presence: Teleoperators and Virtual Environments, 2014
3D virtual humans and physical human-like robots can be used to interact with people in a remote ... more 3D virtual humans and physical human-like robots can be used to interact with people in a remote location in order to increase the feeling of presence. In a telepresence setup, their behaviors are driven by real participants. We envision that in the absence of the real users, when they have to leave or they do not want to do a repetitive task, the control of the robots can be handed to an artificial intelligence component to sustain the ongoing interaction. At the point when human-mediated interaction is required again, control can be returned to the real users. One of the main challenges in telepresence research is the adaptation of 3D position and orientation of the remote participants to the actual physical environment to have appropriate eye contact and gesture awareness in a group conversation. In case the human behind the robot and/or virtual human leaves, multi-party interaction should be handed to an artificial intelligence component. In this paper, we discuss the challenges...
Personality and emotion modeling is being used recently in order to create variation in the behav... more Personality and emotion modeling is being used recently in order to create variation in the behaviour of virtual characters and social robots and to make them socially intelligent for more engaged interaction with people. For these characters to be everyday social actors, it is also important that they can remember and maintain their feelings and establish unique emotional interpersonal relationships
Constraint programming is an approach for solving mostly combinatorial problems by declaratively ... more Constraint programming is an approach for solving mostly combinatorial problems by declaratively describing the problem and using special solving algorithms. Restrictions called constraints are stated over the problem variables that reduces the values each variable can take. In some cases it is not possible to satisfy all constraints or the user can state some preferences on them. One of the
The cost of health care is expected to grow enormously in the coming years. To keep these costs l... more The cost of health care is expected to grow enormously in the coming years. To keep these costs limited, we need better technological tools for self-monitoring and independent aging, and to put a stronger emphasis on wellness, defined as a balanced and healthy lifestyle that avoids diseases at every stage of life. In contrast to traditional medical science that examines patients in laboratories, wellness is determined by our activities, which have so far been difficult to measure objectively. This situation is changing due to the availability of wearable and pervasive sensors, smartphones and other body-worn devices that can capture and infer users activities. Technological advances in this area are rapidly pushing forward. The range of methods as well as the health domains where sensors have been used for monitoring is proliferating. This has created a unique opportunity to understand wellness in an integrative and balanced framework. The goal of this survey is to present the current technological capabilities in sensor technology and activitiy detection methods with the purpose of guiding researchers that are interested in developing wellness technology.
2009 International Conference on CyberWorlds, 2009
Computers and video games are more and more used for serious purposes such as education and healt... more Computers and video games are more and more used for serious purposes such as education and health care. Virtual humans are the inhabitants of the simulated worlds in these games and the challenge is not only to make them look like humans but also to make them behave as humans. Thus, they should be equipped with properties such as social
With the recent advances, today people are able to communicate with embodied (virtual/robotic) en... more With the recent advances, today people are able to communicate with embodied (virtual/robotic) entities using natural ways of communication. In order to use them in our daily lives, they need to be intelligent enough to make long-term relationships with us and this is highly challenging. Previous work on long-term interaction frequently reported that after the novelty effect disappeared, users' interest into the interaction decreased with time. Our primary goal in this study was to develop a system that can still keep the attention of the users after the first interaction. Incorporating the notion of time, we think that the key to long-term interaction is the recall of past memories during current conversation. For this purpose, we developed a longterm interaction framework with remembering and dialogue planning capability. In order to see the effect of remembering on users, we designed a tutoring application and measured the changes in social presence and task engagement levels according to the existence of memory. Different from previous work, users' interest in our system did not decrease with time with the important contributions of remembering to the engagement level of users.
Uploads
Papers by Zerrin Yumak