2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018
We explore new aspects of assistive living on smart human-robot interaction (HRI) that involve au... more We explore new aspects of assistive living on smart human-robot interaction (HRI) that involve automatic recognition and online validation of speech and gestures in a natural interface, providing social features for HRI. We introduce a whole framework and resources of a real-life scenario for elderly subjects supported by an assistive bathing robot, addressing health and hygiene care issues. We contribute a new dataset and a suite of tools used for data acquisition and a state-of-the-art pipeline for multimodal learning within the framework of the I-Support bathing robot, with emphasis on audio and RGB-D visual streams. We consider privacy issues by evaluating the depth visual stream along with the RGB, using Kinect sensors. The audio-gestural recognition task on this new dataset yields up to 84.5%, while the online validation of the I-Support system on elderly users accomplishes up to 84% when the two modalities are fused together. The results are promising enough to support further research in the area of multimodal recognition for assistive social HRI, considering the difficulties of the specific task. Upon acceptance of the paper part of the data will be publicly available.
2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), 2021
Over the past few years, deep learning methods have shown remarkable results in many face-related... more Over the past few years, deep learning methods have shown remarkable results in many face-related tasks including automatic facial expression recognition (FER) in-thewild. Meanwhile, numerous models describing the human emotional states have been proposed by the psychology community. However, we have no clear evidence as to which representation is more appropriate and the majority of FER systems use either the categorical or the dimensional model of affect. Inspired by recent work in multi-label classification, this paper proposes a novel multi-task learning (MTL) framework that exploits the dependencies between these two models using a Graph Convolutional Network (GCN) to recognize facial expressions in-the-wild. Specifically, a shared feature representation is learned for both discrete and continuous recognition in a MTL setting. Moreover, the facial expression classifiers and the valence-arousal regressors are learned through a GCN that explicitly captures the dependencies between them. To evaluate the performance of our method under real-world conditions we perform extensive experiments on the AffectNet and Aff-Wild2 datasets. The results of our experiments show that our method is capable of improving the performance across different datasets and backbone architectures. Finally, we also surpass the previous state-of-the-art methods on the categorical model of AffectNet.
2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018
Human-robot interaction (HRI) is a research area of growing interest with a multitude of applicat... more Human-robot interaction (HRI) is a research area of growing interest with a multitude of applications for both children and adult user groups, as, for example, in edutainment and social robotics. Crucial, however, to its wider adoption remains the robust perception of HRI scenes in natural, untethered, and multi-party interaction scenarios, across user groups. Towards this goal, we investigate three focal HRI perception modules operating on data from multiple audiovisual sensors that observe the HRI scene from the far-field, thus bypassing limitations and platform-dependency of contemporary robotic sensing. In particular, the developed modules fuse intra-and /or intermodality data streams to perform: (i) audiovisual speaker localization; (ii) distant speech recognition; and (iii) visual recognition of hand-gestures. Emphasis is also placed on ensuring high speech and gesture recognition rates for both children and adults. Development and objective evaluation of the three modules is conducted on a corpus of both user groups, collected by our far-field multisensory setup, for an interaction scenario of a question-answering "guess-the-object" collaborative HRI game with a "Furhat" robot. In addition, evaluation of the game incorporating the three developed modules is reported. Our results demonstrate robust far-field audiovisual perception of the multi-party HRI scene.
2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017
Elderly people have particular needs in performing bathing activities, since these tasks require ... more Elderly people have particular needs in performing bathing activities, since these tasks require body flexibility. Our aim is to build an assistive robotic bath system, in order to increase the independence and safety of this procedure. Towards this end, the expertise of professional carers for bathing sequences and appropriate motions has to be adopted, in order to achieve natural, physical human-robot interaction. In this paper, a real-time end-effector motion planning method for an assistive bath robot, using on-line Point-Cloud information, is proposed. The visual feedback obtained from Kinect depth sensor is employed to adapt suitable washing paths to the user's body part motion and deformable surface. We make use of a navigation function-based controller, with guarantied globally uniformly asymptotic stability, and bijective transformations for the adaptation of the paths. Experiments were conducted with a rigid rectangular object for validation purposes, while a female subject took part to the experiment in order to evaluate and demonstrate the basic concepts of the proposed methodology.
2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019
In this work we tackle the problem of child engagement estimation while children freely interact ... more In this work we tackle the problem of child engagement estimation while children freely interact with a robot in a friendly, room-like environment. We propose a deep-based multi-view solution that takes advantage of recent developments in human pose detection. We extract the child's pose from different RGB-D cameras placed regularly in the room, fuse the results and feed them to a deep neural network trained for classifying engagement levels. The deep network contains a recurrent layer, in order to exploit the rich temporal information contained in the pose data. The resulting method outperforms a number of baseline classifiers, and provides a promising tool for better automatic understanding of a child's attitude, interest and attention while cooperating with a robot. The goal is to integrate this model in next generation social robots as an attention monitoring tool during various Child Robot Interaction (CRI) tasks both for Typically Developed (TD) children and children affected by autism (ASD).
We present the i-Walk system1, a novel framework for intelligent mobility assistance applications... more We present the i-Walk system1, a novel framework for intelligent mobility assistance applications. The proposed system is capable of automatically understanding human activity, assessing mobility and rehabilitation progress, recognizing human intentions and communicating with the patients by giving meaningful feedback. To this end, multiple sensors, i.e. cameras, microphones, lasers, provide multimodal data in order to allow for user monitoring, while state-of-theart and beyond algorithms have been developed and integrated into the system to enable recognition, interaction and assessment. More specifically, i-Walk performs in real-time and consists of four main sub-modules that interact automatically to provide speech understanding, activity recognition, mobility analysis and multimodal communication for seamless HRI. The i-Walk assessment system is evaluated on a database of healthy subjects and patients, who participated in carefully designed experimental scenarios that cover esse...
In this paper we present a Greek speech database with real multi-modal data in a smart home two-r... more In this paper we present a Greek speech database with real multi-modal data in a smart home two-room environment. In total, 20 speakers were recorded in 240 one-minute long sessions. The recordings include utterances of activation keywords and commands for home automation control, but also phonetically rich sentences and conversational speech. Audio, speaker movements and gestures were captured by 20 condenser microphones installed on the walls and ceiling, 6 MEMS microphones, 2 close-talk microphones and one Kinect camera. The new publicly available database exhibits adverse noise conditions because of background noises and acoustic events performed during the recordings to better approximate a realistic everyday home scenario. Thus, it is suitable for experimentation on voice activity and event detection, source localization, speech enhancement and far-field speech recognition. We present the details of the corpus as well as baseline results on multi-channel voice activity detection and spoken command recognition.
2018 IEEE International Conference on Robotics and Automation (ICRA), 2018
Child-robot interaction is an interdisciplinary research area that has been attracting growing in... more Child-robot interaction is an interdisciplinary research area that has been attracting growing interest, primarily focusing on edutainment applications. A crucial factor to the successful deployment and wide adoption of such applications remains the robust perception of the child's multimodal actions, when interacting with the robot in a natural and untethered fashion. Since robotic sensory and perception capabilities are platform-dependent and most often rather limited, we propose a multiple Kinect-based system to perceive the child-robot interaction scene that is robot-independent and suitable for indoors interaction scenarios. The audiovisual input from the Kinect sensors is fed into speech, gesture, and action recognition modules, appropriately developed in this paper to address the challenging nature of child-robot interaction. For this purpose, data from multiple children are collected and used for module training or adaptation. Further, information from the multiple sensors is fused to enhance module performance. The perception system is integrated in a modular multi-robot architecture demonstrating its flexibility and scalability with different robotic platforms. The whole system, called Multi3, is evaluated, both objectively at the module level and subjectively in its entirety, under appropriate child-robot interaction scenarios containing several carefully designed games between children and robots.
In this paper we address the problem of multi-cue affect recognition in challenging scenarios suc... more In this paper we address the problem of multi-cue affect recognition in challenging scenarios such as child-robot interaction. Towards this goal we propose a method for automatic recognition of affect that leverages body expressions alongside facial ones, as opposed to traditional methods that typically focus only on the latter. Our deep-learning based method uses hierarchical multi-label annotations and multi-stage losses, can be trained both jointly and separately, and offers us computational models for both individual modalities, as well as for the whole body emotion. We evaluate our method on a challenging childrobot interaction database of emotional expressions collected by us, as well as on the GEMEP public database of acted emotions by adults, and show that the proposed method achieves significantly better results than facial-only expression baselines.
Index Terms-Human detection and tracking, human-centered robotics, physically assistive devices, ... more Index Terms-Human detection and tracking, human-centered robotics, physically assistive devices, automation in life sciences: biotechnology, pharmaceutical and health care, medical robots and systems.
2016 IEEE Symposium Series on Computational Intelligence (SSCI), 2016
In this paper we discuss the integration of a communication model in the MOBOT assistive robotic ... more In this paper we discuss the integration of a communication model in the MOBOT assistive robotic platform and its evaluation by target users. The MOBOT platform envisions the development of cognitive robotic assistant prototypes that act proactively, adaptively and interactively with respect to elderly humans with slight walking and cognitive impairments. The respective multimodal action recognition system has been developed to monitor, analyze and predict user actions with a high level of accuracy and detail. The robotic platform incorporates a human-robot communication model that has been defined with semantics of human actions in interaction, their capture and their representation in terms of behavioral patterns, to achieve an effective, natural interaction, aiming to support elderly users of slight walking and cognitive inability. The platform has been evaluated in a series of validation experiments with end users, the procedure and results of which are also presented in this paper.
We explore new directions for automatic human gesture recognition and human joint angle estimatio... more We explore new directions for automatic human gesture recognition and human joint angle estimation as applied for human-robot interaction in the context of an actual challenging task of assistive living for real-life elderly subjects. Our contributions include state-of-the-art approaches for both low-and mid-level vision, as well as for higher level action and gesture recognition. The first direction investigates a deep learning based framework for the challenging task of human joint angle estimation on noisy real world RGB-D images. The second direction includes the employment of dense trajectory features for online processing of videos for automatic gesture recognition with real-time performance. Our approaches are evaluated both qualitative and quantitatively on a newly acquired dataset that is constructed on a challenging real-life scenario on assistive living for elderly subjects.
2016 IEEE International Conference on Image Processing (ICIP), 2016
Motivated by the recent advances in human-robot interaction we present a new dataset, a suite of ... more Motivated by the recent advances in human-robot interaction we present a new dataset, a suite of tools to handle it and state-of-theart work on visual gestures and audio commands recognition. The dataset has been collected with an integrated annotation and acquisition web-interface that facilitates on-the-way temporal ground-truths for fast acquisition. The dataset includes gesture instances in which the subjects are not in strict setup positions, and contains multiple scenarios, not restricted to a single static configuration. We accompany it by a valuable suite of tools as the practical interface to acquire audiovisual data in the robotic operating system, a state-of-the-art learning pipeline to train visual gesture and audio command models, and an online gesture recognition system. Finally, we include a rich evaluation of the dataset providing rich and insightfull experimental recognition results.
2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016
Within the context of assistive robotics we develop an intelligent interface that provides multim... more Within the context of assistive robotics we develop an intelligent interface that provides multimodal sensory processing capabilities for human action recognition. Human action is considered in multimodal terms, containing inputs such as audio from microphone arrays, and visual inputs from high definition and depth cameras. Exploring state-of-the-art approaches from automatic speech recognition, and visual action recognition, we multimodally recognize actions and commands. By fusing the unimodal information streams, we obtain the optimum multimodal hypothesis which is to be further exploited by the active mobility assistance robot in the framework of the MOBOT EU research project. Evidence from recognition experiments shows that by integrating multiple sensors and modalities, we increase multimodal recognition performance in the newly acquired challenging dataset involving elderly people while interacting with the assistive robot.
2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015
In this work, we investigate the efficacy of Micro Electro-Mechanical System (MEMS) microphones, ... more In this work, we investigate the efficacy of Micro Electro-Mechanical System (MEMS) microphones, a newly developed technology of very compact sensors, for multichannel speech enhancement. Experiments are conducted on real speech data collected using a MEMS microphone array. First, the effectiveness of the array geometry for noise suppression is explored, using a new corpus containing speech recorded in diffuse and localized noise fields with a MEMS microphone array configured in linear and hexagonal array geometries. Our results indicate superior performance of the hexagonal geometry. Then, MEMS microphones are compared to Electret Condenser Microphones (ECMs), using the ATHENA database, which contains speech recorded in realistic smart home noise conditions with hexagonal-type arrays of both microphone types. MEMS microphones exhibit performance similar to ECMs. Good performance, versatility in placement, small size, and low cost, make MEMS microphones attractive for multichannel speech processing.
2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014
The paper presents our approach to speech-controlled home automation. We are focusing on the dete... more The paper presents our approach to speech-controlled home automation. We are focusing on the detection and recognition of spoken commands preceded by a key-phrase as recorded in a voice-enabled apartment by a set of multiple microphones installed in the rooms. For both problems we investigate robust modeling, environmental adaptation and multichannel processing to cope with a) insufficient training data and b) the far-field effects and noise in the apartment. The proposed integrated scheme is evaluated in a challenging and highly realistic corpus of simulated audio recordings and achieves F-measure close to 0.70 for key-phrase spotting and word accuracy close to 98% for the command recognition task.
Robotic rollators can play a significant role as assistive devices for people with impaired movem... more Robotic rollators can play a significant role as assistive devices for people with impaired movement and mild cognitive deficit. This paper presents an overview of the i-Walk concept; an intelligent robotic rollator offering cognitive and ambulatory assistance to people with light to moderate movement impairment, such as the elderly. We discuss the two robotic prototypes being developed, their various novel functionalities, system architecture, modules and function scope, and present preliminary experimental results with actual users.
2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018
We explore new aspects of assistive living on smart human-robot interaction (HRI) that involve au... more We explore new aspects of assistive living on smart human-robot interaction (HRI) that involve automatic recognition and online validation of speech and gestures in a natural interface, providing social features for HRI. We introduce a whole framework and resources of a real-life scenario for elderly subjects supported by an assistive bathing robot, addressing health and hygiene care issues. We contribute a new dataset and a suite of tools used for data acquisition and a state-of-the-art pipeline for multimodal learning within the framework of the I-Support bathing robot, with emphasis on audio and RGB-D visual streams. We consider privacy issues by evaluating the depth visual stream along with the RGB, using Kinect sensors. The audio-gestural recognition task on this new dataset yields up to 84.5%, while the online validation of the I-Support system on elderly users accomplishes up to 84% when the two modalities are fused together. The results are promising enough to support further research in the area of multimodal recognition for assistive social HRI, considering the difficulties of the specific task. Upon acceptance of the paper part of the data will be publicly available.
2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), 2021
Over the past few years, deep learning methods have shown remarkable results in many face-related... more Over the past few years, deep learning methods have shown remarkable results in many face-related tasks including automatic facial expression recognition (FER) in-thewild. Meanwhile, numerous models describing the human emotional states have been proposed by the psychology community. However, we have no clear evidence as to which representation is more appropriate and the majority of FER systems use either the categorical or the dimensional model of affect. Inspired by recent work in multi-label classification, this paper proposes a novel multi-task learning (MTL) framework that exploits the dependencies between these two models using a Graph Convolutional Network (GCN) to recognize facial expressions in-the-wild. Specifically, a shared feature representation is learned for both discrete and continuous recognition in a MTL setting. Moreover, the facial expression classifiers and the valence-arousal regressors are learned through a GCN that explicitly captures the dependencies between them. To evaluate the performance of our method under real-world conditions we perform extensive experiments on the AffectNet and Aff-Wild2 datasets. The results of our experiments show that our method is capable of improving the performance across different datasets and backbone architectures. Finally, we also surpass the previous state-of-the-art methods on the categorical model of AffectNet.
2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018
Human-robot interaction (HRI) is a research area of growing interest with a multitude of applicat... more Human-robot interaction (HRI) is a research area of growing interest with a multitude of applications for both children and adult user groups, as, for example, in edutainment and social robotics. Crucial, however, to its wider adoption remains the robust perception of HRI scenes in natural, untethered, and multi-party interaction scenarios, across user groups. Towards this goal, we investigate three focal HRI perception modules operating on data from multiple audiovisual sensors that observe the HRI scene from the far-field, thus bypassing limitations and platform-dependency of contemporary robotic sensing. In particular, the developed modules fuse intra-and /or intermodality data streams to perform: (i) audiovisual speaker localization; (ii) distant speech recognition; and (iii) visual recognition of hand-gestures. Emphasis is also placed on ensuring high speech and gesture recognition rates for both children and adults. Development and objective evaluation of the three modules is conducted on a corpus of both user groups, collected by our far-field multisensory setup, for an interaction scenario of a question-answering "guess-the-object" collaborative HRI game with a "Furhat" robot. In addition, evaluation of the game incorporating the three developed modules is reported. Our results demonstrate robust far-field audiovisual perception of the multi-party HRI scene.
2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017
Elderly people have particular needs in performing bathing activities, since these tasks require ... more Elderly people have particular needs in performing bathing activities, since these tasks require body flexibility. Our aim is to build an assistive robotic bath system, in order to increase the independence and safety of this procedure. Towards this end, the expertise of professional carers for bathing sequences and appropriate motions has to be adopted, in order to achieve natural, physical human-robot interaction. In this paper, a real-time end-effector motion planning method for an assistive bath robot, using on-line Point-Cloud information, is proposed. The visual feedback obtained from Kinect depth sensor is employed to adapt suitable washing paths to the user's body part motion and deformable surface. We make use of a navigation function-based controller, with guarantied globally uniformly asymptotic stability, and bijective transformations for the adaptation of the paths. Experiments were conducted with a rigid rectangular object for validation purposes, while a female subject took part to the experiment in order to evaluate and demonstrate the basic concepts of the proposed methodology.
2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019
In this work we tackle the problem of child engagement estimation while children freely interact ... more In this work we tackle the problem of child engagement estimation while children freely interact with a robot in a friendly, room-like environment. We propose a deep-based multi-view solution that takes advantage of recent developments in human pose detection. We extract the child's pose from different RGB-D cameras placed regularly in the room, fuse the results and feed them to a deep neural network trained for classifying engagement levels. The deep network contains a recurrent layer, in order to exploit the rich temporal information contained in the pose data. The resulting method outperforms a number of baseline classifiers, and provides a promising tool for better automatic understanding of a child's attitude, interest and attention while cooperating with a robot. The goal is to integrate this model in next generation social robots as an attention monitoring tool during various Child Robot Interaction (CRI) tasks both for Typically Developed (TD) children and children affected by autism (ASD).
We present the i-Walk system1, a novel framework for intelligent mobility assistance applications... more We present the i-Walk system1, a novel framework for intelligent mobility assistance applications. The proposed system is capable of automatically understanding human activity, assessing mobility and rehabilitation progress, recognizing human intentions and communicating with the patients by giving meaningful feedback. To this end, multiple sensors, i.e. cameras, microphones, lasers, provide multimodal data in order to allow for user monitoring, while state-of-theart and beyond algorithms have been developed and integrated into the system to enable recognition, interaction and assessment. More specifically, i-Walk performs in real-time and consists of four main sub-modules that interact automatically to provide speech understanding, activity recognition, mobility analysis and multimodal communication for seamless HRI. The i-Walk assessment system is evaluated on a database of healthy subjects and patients, who participated in carefully designed experimental scenarios that cover esse...
In this paper we present a Greek speech database with real multi-modal data in a smart home two-r... more In this paper we present a Greek speech database with real multi-modal data in a smart home two-room environment. In total, 20 speakers were recorded in 240 one-minute long sessions. The recordings include utterances of activation keywords and commands for home automation control, but also phonetically rich sentences and conversational speech. Audio, speaker movements and gestures were captured by 20 condenser microphones installed on the walls and ceiling, 6 MEMS microphones, 2 close-talk microphones and one Kinect camera. The new publicly available database exhibits adverse noise conditions because of background noises and acoustic events performed during the recordings to better approximate a realistic everyday home scenario. Thus, it is suitable for experimentation on voice activity and event detection, source localization, speech enhancement and far-field speech recognition. We present the details of the corpus as well as baseline results on multi-channel voice activity detection and spoken command recognition.
2018 IEEE International Conference on Robotics and Automation (ICRA), 2018
Child-robot interaction is an interdisciplinary research area that has been attracting growing in... more Child-robot interaction is an interdisciplinary research area that has been attracting growing interest, primarily focusing on edutainment applications. A crucial factor to the successful deployment and wide adoption of such applications remains the robust perception of the child's multimodal actions, when interacting with the robot in a natural and untethered fashion. Since robotic sensory and perception capabilities are platform-dependent and most often rather limited, we propose a multiple Kinect-based system to perceive the child-robot interaction scene that is robot-independent and suitable for indoors interaction scenarios. The audiovisual input from the Kinect sensors is fed into speech, gesture, and action recognition modules, appropriately developed in this paper to address the challenging nature of child-robot interaction. For this purpose, data from multiple children are collected and used for module training or adaptation. Further, information from the multiple sensors is fused to enhance module performance. The perception system is integrated in a modular multi-robot architecture demonstrating its flexibility and scalability with different robotic platforms. The whole system, called Multi3, is evaluated, both objectively at the module level and subjectively in its entirety, under appropriate child-robot interaction scenarios containing several carefully designed games between children and robots.
In this paper we address the problem of multi-cue affect recognition in challenging scenarios suc... more In this paper we address the problem of multi-cue affect recognition in challenging scenarios such as child-robot interaction. Towards this goal we propose a method for automatic recognition of affect that leverages body expressions alongside facial ones, as opposed to traditional methods that typically focus only on the latter. Our deep-learning based method uses hierarchical multi-label annotations and multi-stage losses, can be trained both jointly and separately, and offers us computational models for both individual modalities, as well as for the whole body emotion. We evaluate our method on a challenging childrobot interaction database of emotional expressions collected by us, as well as on the GEMEP public database of acted emotions by adults, and show that the proposed method achieves significantly better results than facial-only expression baselines.
Index Terms-Human detection and tracking, human-centered robotics, physically assistive devices, ... more Index Terms-Human detection and tracking, human-centered robotics, physically assistive devices, automation in life sciences: biotechnology, pharmaceutical and health care, medical robots and systems.
2016 IEEE Symposium Series on Computational Intelligence (SSCI), 2016
In this paper we discuss the integration of a communication model in the MOBOT assistive robotic ... more In this paper we discuss the integration of a communication model in the MOBOT assistive robotic platform and its evaluation by target users. The MOBOT platform envisions the development of cognitive robotic assistant prototypes that act proactively, adaptively and interactively with respect to elderly humans with slight walking and cognitive impairments. The respective multimodal action recognition system has been developed to monitor, analyze and predict user actions with a high level of accuracy and detail. The robotic platform incorporates a human-robot communication model that has been defined with semantics of human actions in interaction, their capture and their representation in terms of behavioral patterns, to achieve an effective, natural interaction, aiming to support elderly users of slight walking and cognitive inability. The platform has been evaluated in a series of validation experiments with end users, the procedure and results of which are also presented in this paper.
We explore new directions for automatic human gesture recognition and human joint angle estimatio... more We explore new directions for automatic human gesture recognition and human joint angle estimation as applied for human-robot interaction in the context of an actual challenging task of assistive living for real-life elderly subjects. Our contributions include state-of-the-art approaches for both low-and mid-level vision, as well as for higher level action and gesture recognition. The first direction investigates a deep learning based framework for the challenging task of human joint angle estimation on noisy real world RGB-D images. The second direction includes the employment of dense trajectory features for online processing of videos for automatic gesture recognition with real-time performance. Our approaches are evaluated both qualitative and quantitatively on a newly acquired dataset that is constructed on a challenging real-life scenario on assistive living for elderly subjects.
2016 IEEE International Conference on Image Processing (ICIP), 2016
Motivated by the recent advances in human-robot interaction we present a new dataset, a suite of ... more Motivated by the recent advances in human-robot interaction we present a new dataset, a suite of tools to handle it and state-of-theart work on visual gestures and audio commands recognition. The dataset has been collected with an integrated annotation and acquisition web-interface that facilitates on-the-way temporal ground-truths for fast acquisition. The dataset includes gesture instances in which the subjects are not in strict setup positions, and contains multiple scenarios, not restricted to a single static configuration. We accompany it by a valuable suite of tools as the practical interface to acquire audiovisual data in the robotic operating system, a state-of-the-art learning pipeline to train visual gesture and audio command models, and an online gesture recognition system. Finally, we include a rich evaluation of the dataset providing rich and insightfull experimental recognition results.
2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016
Within the context of assistive robotics we develop an intelligent interface that provides multim... more Within the context of assistive robotics we develop an intelligent interface that provides multimodal sensory processing capabilities for human action recognition. Human action is considered in multimodal terms, containing inputs such as audio from microphone arrays, and visual inputs from high definition and depth cameras. Exploring state-of-the-art approaches from automatic speech recognition, and visual action recognition, we multimodally recognize actions and commands. By fusing the unimodal information streams, we obtain the optimum multimodal hypothesis which is to be further exploited by the active mobility assistance robot in the framework of the MOBOT EU research project. Evidence from recognition experiments shows that by integrating multiple sensors and modalities, we increase multimodal recognition performance in the newly acquired challenging dataset involving elderly people while interacting with the assistive robot.
2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015
In this work, we investigate the efficacy of Micro Electro-Mechanical System (MEMS) microphones, ... more In this work, we investigate the efficacy of Micro Electro-Mechanical System (MEMS) microphones, a newly developed technology of very compact sensors, for multichannel speech enhancement. Experiments are conducted on real speech data collected using a MEMS microphone array. First, the effectiveness of the array geometry for noise suppression is explored, using a new corpus containing speech recorded in diffuse and localized noise fields with a MEMS microphone array configured in linear and hexagonal array geometries. Our results indicate superior performance of the hexagonal geometry. Then, MEMS microphones are compared to Electret Condenser Microphones (ECMs), using the ATHENA database, which contains speech recorded in realistic smart home noise conditions with hexagonal-type arrays of both microphone types. MEMS microphones exhibit performance similar to ECMs. Good performance, versatility in placement, small size, and low cost, make MEMS microphones attractive for multichannel speech processing.
2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014
The paper presents our approach to speech-controlled home automation. We are focusing on the dete... more The paper presents our approach to speech-controlled home automation. We are focusing on the detection and recognition of spoken commands preceded by a key-phrase as recorded in a voice-enabled apartment by a set of multiple microphones installed in the rooms. For both problems we investigate robust modeling, environmental adaptation and multichannel processing to cope with a) insufficient training data and b) the far-field effects and noise in the apartment. The proposed integrated scheme is evaluated in a challenging and highly realistic corpus of simulated audio recordings and achieves F-measure close to 0.70 for key-phrase spotting and word accuracy close to 98% for the command recognition task.
Robotic rollators can play a significant role as assistive devices for people with impaired movem... more Robotic rollators can play a significant role as assistive devices for people with impaired movement and mild cognitive deficit. This paper presents an overview of the i-Walk concept; an intelligent robotic rollator offering cognitive and ambulatory assistance to people with light to moderate movement impairment, such as the elderly. We discuss the two robotic prototypes being developed, their various novel functionalities, system architecture, modules and function scope, and present preliminary experimental results with actual users.
Uploads
Papers by Petros Maragos