Skip to main content

Masashi Inoue

Tohoku Institute of Technology, Information and Communication Engineering, Faculty Member

Followers

13

Following

11

Co-author

1

Public Views

My research interest lies in the understanding human cognition and communication from information processing perspective.

less

University of California, Santa Barbara

Javier Díaz Noci

Pompeu Fabra University

Scott R. Stroud

The University of Texas at Austin

Radhika Gajjala

Bowling Green State University

Kansas State University

Aswani Kumar Cherukuri

VIT University

Armando Marques-Guedes

UNL - New University of Lisbon

Ucla

Università di Bologna

Macquarie University

InterestsView All (8)

Uploads

Papers by Masashi Inoue

Retrieving lightly annotated images using image similarities

Proceedings of the 2005 ACM symposium on Applied computing, 2005

Users' search needs are often represented by words and images are retrieved according to such tex... more Users' search needs are often represented by words and images are retrieved according to such textual queries. Annotation words assigned to the stored images are most useful to connect queries to the images. However, due to annotation cost, quite limited amount of annotation words are available in many cases. When annotations are not given at all, there needs to be some techniques that assign annotations automatically. When only a few annotation words are given to each image (lightly annotated), there need to be some enhancement techniques that best use the available annotations. We address the later problem by estimating word associations to fill in the lexical gap between queries and annotations. The model of word associations can be learned from the data. However, since images are only lightly annotated, their sparseness in computing word associations becomes crucial. To compensate the sparseness, we propose a novel data exploration technique in which image similarities contribute to the estimation of word associations on the assumption that similar images have similar semantic concepts. We experimentally show the potential benefit of our approach.

Human Interpretation of Inter-robot Communication

Lecture Notes in Computer Science, 2022

Statistical learning from multiple information sources

Nara Institute of Science and Technology, Mar 24, 2004

In intelligent information processing tasks such as pattern recognition and information retrieval... more In intelligent information processing tasks such as pattern recognition and information retrieval (IR), probabilistic models are now widely used because they can represent ambiguities of observed data and are robust against noises. Parameters of probabilistic models are statistically estimated (learned) from given training data. However, when the training data contain an insufficient amount of information, the learned model becomes unreliable and its performance severely deteriorates. This thesis proposes two novel learning algorithms that use multiple information sources to mitigate this information scarcity problem in the following two applications. The first application is solving classification problems in which optimal class labels are automatically assigned to observations whose class labels are unknown. Among various types of classification problems, this paper considers classification of sequences that consist of sequential observation points. As a classifier, we focus on the hidden Markov model (HMM), which has been widely used for the classification of sequences. Generally, an HMM is trained on labeled data that consist of observed feature values and class labels. However, due to the high labeling cost, the amount of labeled training data is often small. In this thesis, we propose a learning scheme called semi-supervised learning to improve the classification performance even if the amount of labeled training data is small. The proposed scheme uses both of a small amount of labeled data, and unlabeled data that are not usually used for learning. First, we design a suitable HMM structure for using the unlabeled sequences. Then, we formally derive a semisupervised learning algorithm in which the convergence property is theoretically guaranteed. Next, we apply the proposed method to two types of time series sequences: those acquired from sign language sign data and those acquired from speech phoneme data. Experimental results show that the proposed method outperforms conventional methods. The second application is solving IR problems, especially cross-media IR in which queries and corresponding target data belong to different media. More specifically, this thesis focuses on situations where queries are represented as text and target data are images. When textual annotations explaining the contents of images are provided, such cross-media image retrieval can be regarded as ordinary textual IR. In text retrieval, associations between words can be learned from data and used to relate queries and text in the documents. However, in image retrieval, the number of annotations is too small to learn the word relationships. Using the fact that annotations and images are related, we regard images as the paired data of annotations.

Exploitation of Unlabeled Sequences in Hidden Markov Models

This paper presents a method for effectively using unlabeled sequential data in the learning of h... more This paper presents a method for effectively using unlabeled sequential data in the learning of hidden Markov models (HMMs). With the conventional approach, class labels for unlabeled data are assigned deterministically by HMMs learned from labeled data. Such labeling often becomes unreliable when the number of labeled data is small. We propose an extended Baum-Welch (EBW) algorithm in which the labeling is undertaken probabilistically and iteratively so that the labeled and unlabeled data likelihoods are improved. Unlike the conventional approach, the EBW algorithm guarantees convergence to a local maximum of the likelihood. Experimental results on gesture data and speech data show that when labeled training data are scarce, by using unlabeled data, the EBW algorithm improves the classification performance of HMMs more robustly than the conventional naive labeling (NL) approach.

Observational and Accelerometer Analysis of Head Movement Patterns in Psychotherapeutic Dialogue †

Sensors (Basel, Switzerland), 2021

Psychotherapists, who use their communicative skills to assist people, review their dialogue prac... more Psychotherapists, who use their communicative skills to assist people, review their dialogue practices and improve their skills from their experiences. However, technology has not been fully exploited for this purpose. In this study, we analyze the use of head movements during actual psychotherapeutic dialogues between two participants—therapist and client—using video recordings and head-mounted accelerometers. Accelerometers have been utilized in the mental health domain but not for analyzing mental health related communications. We examined the relationship between the state of the interaction and temporally varying head nod and movement patterns in psychological counseling sessions. Head nods were manually annotated and the head movements were measured using accelerometers. Head nod counts were analyzed based on annotations taken from video data. We conducted cross-correlation analysis of the head movements of the two participants using the accelerometer data. The results of two ...

Web Resource Selection for Dialogue System Generating Natural Responses

Communications in Computer and Information Science, 2011

Using Web information in example-based dialogue systems is considered to be a good way to increas... more Using Web information in example-based dialogue systems is considered to be a good way to increase the topical relevance of system responses. However, Web content is mostly written documents, not colloquial utterances. To alleviate the discrepancy in style between written and spoken language, we suggest that the corpus should be selected subsets of the Web, viz., online bulletin boards. The naturalness provided by casual utterances is especially important when the goal of a dialogue is chatting rather than formal question answering. By appropriately selecting the information source, our textbased dialogue system can generate friendly responses to users. These characteristics were evaluated with a questionnaire.

Using Visual Linkages for Multilingual Image Retrieval

Lecture Notes in Computer Science, 2007

The use of visual features in text-based ad-hoc image retrieval is challenging. Visual and textua... more The use of visual features in text-based ad-hoc image retrieval is challenging. Visual and textual information have so far been treated equally in terms of their properties and combined with weighting mechanisms for balancing their contributions to the ranking. The use of visual and textual information in a single retrieval system sometimes limits its applicability due to the lack of modularity. In this paper, we propose an image retrieval method that separates the usage of visual information from that of textual information. Visual clustering establishes linkages between images and the relationships are later used for the reranking. By applying clustering on visual features prior to the ranking, the main retrieval process becomes purely text-based. Experimental results on the ImageCLEFphoto ad-hoc task show this scheme is suitable for querying multilingual collections on some search topics.

Mining Visual Knowledge for Multi-Lingual Image Retrieval

21st International Conference on Advanced Information Networking and Applications Workshops (AINAW'07), 2007

Users commonly rely just on scarce textual annotation when their searches for images are semantic... more Users commonly rely just on scarce textual annotation when their searches for images are semantic or conceptual based. Rich visual information is often thrown away in basic annotation-based image retrieval because its relationship to the semantic content is not always clear. To ensure that appropriate visual information is included, we propose using visual clustering within pre-processing and post-processing steps of text-based retrieval. A clustering algorithm finds pairs of images that are nearly identical and are, therefore, presumed semantically similar. The output from basic retrieval systems is a ranked list of images based only on lexical term matching. The obtained cluster knowledge is then used to modify the ranking result during the post-processing step. Low ranked images considered nearly identical to more highly ranked images are then pulled up. The modularity of this architecture allows us to integrate a data mining process without having to change core information retrieval systems. Evaluation on a cross-language image retrieval test collection showed that this method improved retrieval performance for certain queries in multilingual settings.

Image retrieval: Research and use in the information explosion

Progress in Informatics, 2009

We surveyed the current research on the retrieval and utilization of images. This paper provides ... more We surveyed the current research on the retrieval and utilization of images. This paper provides an overview of the difficulties and possibilities of technological or technology-related research topics resulting from the huge amount of images currently available and highlights some of the important research topics. We looked at the ongoing research activities and analyzed them from four aspects, information access and organization technology, the computing infrastructure that enables access to large-scale image resources, issues in human-system interaction and human factors related to using images, and the social aspect of image media. On the technical side, we noticed that as the number of digital images increases, so does the importance of the accuracy and scalability in relation to the image retrieval. The accuracy and scalability are in fact needed to cope with the current explosion in digital images. On the social side, not so long ago, image retrieval technologies were only experimental tools or used by experts within a limited domain. However, now the general public has access to a wide range of digital images, which means that image retrieval technologies are being used by various users in a large diversity of social contexts. Thus, as we show in this paper, the accuracy and scalability are not the only important factors in the era of information explosion, but that researchers must also be concerned with the social aspects of these technologies.

Utility of Gestural Cues in Indexing Semantic Miscommunication

2010 5th International Conference on Future Information Technology, 2010

In multimedia data analysis, automated indexing of conversational video is an emerging topic. One... more In multimedia data analysis, automated indexing of conversational video is an emerging topic. One challenging problem in this topic is the recognition of higher-level concepts, such as miscommunications in conversations. While detecting miscommunications is generally easy for speakers as well as observers, it is not currently understood which cues contribute to their detection and to what extent. To make use of the knowledge on gestural cues in multimedia systems, the applicability of machine learning is investigated as a means of detecting miscommunication from gestural patterns observed in psychotherapeutic face-to-face conversations. Various features are taken from gesture data, and both simple and complex classifiers are constructed using these features. Both short-term and long-term effects are tested using different time window sizes. Also, two types of gestures, communicative and non-communicative, are considered. The experimental results suggest that there is no single gestural feature that can explain the occurrence of semantic miscommunication. Another interesting finding is that gestural cues correlate more with long-term gestural patterns than with short-term ones.

Interaction and user experiences with multimedia technologies

Proceeding of the 3rd ACM international workshop on Human-centered computing - HCC '08, 2008

In this paper, we investigate future topics and challenges of interaction and user experience in ... more In this paper, we investigate future topics and challenges of interaction and user experience in multimedia. We bring together different perspectives from overlapping fields of research such as multimedia, human-computer interaction, information retrieval, networked multimedia, and creative arts. Based on potential intersections, we define three application domains to be investigated further, as they create high demand and good prospect for long-lasting developments in the future. These application domains are: media working environments, media enter-/edutainment, and social media engagement. Each application domain is analyzed along five dimensions, namely: information quality, presentation quality, ambience, interactivity, and user expectations. Based on this analysis, we identify the most pressing research questions and key challenges for each area. Finally, we advocate a user-centered approach to tackle these challenges and questions in order to develop relevant multimedia applications that best meet the users' expectations.

User-Model-Based Evaluation for Interactive Image Retrieval

2010 IEEE/ACIS 9th International Conference on Computer and Information Science, 2010

User-system interaction is sometimes a cumbersome element of non-textual information access. Imag... more User-system interaction is sometimes a cumbersome element of non-textual information access. Image retrieval systems now incorporate various interaction mechanisms. However, there is no standard or established way of interacting that has been proven to be most effective. Therefore, many system designers are introducing or eliminating various functionalities. The bottleneck in the development effort is finding an effective and fair way to evaluate these systems. For static information access, several test collections have commonly been used. For interactive information access, expensive laboratory testing is needed. This paper presents a user semi-automatic user-modelbased evaluation method that is designed to reduce cost while maintaining objectivity.

Query Types and Visual Concept-Based Post-retrieval Clustering

Lecture Notes in Computer Science, 2009

In the photo retrieval task of ImageCLEF 2008, we examined the influences of image representation... more In the photo retrieval task of ImageCLEF 2008, we examined the influences of image representations and clustering methods for enhancing instance recall when they are used in post-retrieval clustering. Two types of visual concepts and hierarchical and partitioning clustering methods were compared. We used the title fields in the search topics, and either only the title fields or both the title and description fields of the annotations were in English. The experimental results showed that one type of visual concept representation dominated the other except for in one condition. Also, it was found that hierarchical clustering can enhance instance recall while preserving the precision when the threshold parameters are appropriately set. In contrast, partitioning clustering degraded the results. We also categorized the queries based on the clustering criteria, geographical and non-geographical, and found that the geographical queries are relatively easy and post-retrieval clustering works better for them.

Manual and Accelerometer Analysis of Head Nodding Patterns in Goal-oriented Dialogues

Lecture Notes in Computer Science, 2011

We studied communication patterns in face-to-face dialogues between people for the purpose of ide... more We studied communication patterns in face-to-face dialogues between people for the purpose of identifying conversation features that can be exploited to improve human-computer interactions. We chose to study the psychological counseling setting as it provides good examples of task-oriented dialogues. The dialogues between two participants, therapist and patient, were video recorded. The participants' head movements were measured by using head-mounted accelerometers. The relationship between the dialogue process and head nodding frequency was analyzed on the basis of manual annotations. The segments where nods of the two participants correlated were identified on the basis of the accelerometer data. Our analysis suggests that there are characteristic nodding patterns in different dialogue stages.

Gestural cue analysis in automated semantic miscommunication annotation

Multimedia Tools and Applications, 2011

Automated annotation of conversational video using semantic labels is a challenging topic. We inv... more Automated annotation of conversational video using semantic labels is a challenging topic. We investigated miscommunication as an example of a higher-level semantic concept in conversations. Miscommunication is an important obstacle in solving psychological problems such as those in psychotherapeutic interviews. Although miscommunications are often obvious to the speakers as well as observers, it is difficult for machines to detect them from low-level features. This difficulty is due to the lack of understanding on which cues contribute to miscommunication. Miscommunications are mainly associated with spoken content, though non-verbal elements may play a role. Among various non-verbal features, we investigate gestural cues. Various features are taken from gesture data, and both simple and complex classifiers are constructed using machine learning. The experimental results suggest that there is no single gestural feature that can predict or explain the occurrence of semantic miscommunication.

Multimodal Corpus for Psychotherapeutic Situations

Multimodal Corpora: How Should Multimodal Corpora Deal with the Situation? Workshop Programme

This paper presents a design principle for construction of an in-house multimodal corpus for comp... more This paper presents a design principle for construction of an in-house multimodal corpus for computationally analysing and better understanding conversations during psychotherapy. We discuss some sharable information about research community data collection procedures such as recording devices, consent forms, and privacy considerations. We also explain multimodal coding schema and metadata that are needed in the domain. The corpus has three distinguishing properties: 1) it was constructed only for our research ...

Human Judgment on Humor Expressions in a Community-Based Question-Answering Service

For understanding humorous dialogue, a collection of humorous expressions is needed. In addition ... more For understanding humorous dialogue, a collection of humorous expressions is needed. In addition to humorous expressions, their annotations are important to be used as language resources. In this paper, we analyzed how human assessors annotate humorous expressions extracted from an online community-based questionanswering (CQA) corpus, which contains many interesting examples of humorous communication. We analyzed the annotation results of a collection of humorous expressions as done by 28 annotators in ...

Collecting Humorous Expressions from a Community-based Question-answering-service Corpus

We proposed a method of collecting humorous expressions from an online community-based question-a... more We proposed a method of collecting humorous expressions from an online community-based question-answering (CQA) corpus where some users post a variety of questions and other users post relevant answers. Although the service is created for the purpose of knowledge exchange, there are users who enjoy posting humorous responses. Therefore, the corpus contains many interesting humour communication examples that might be useful in understanding the nature of online communications and variations in ...

The remarkable search topic-finding task to share success stories of cross-language information retrieval

Proceedings of the Fifth Workshop on Important Unresolved Matters, Jun 29, 2005

The performance of cross-language information retrieval (CLIR) systems has been improved to the l... more The performance of cross-language information retrieval (CLIR) systems has been improved to the level of practical use. The next step is to inform potential users that CLIR technologies are ready to be used. A good way of doing this is to present attractive scenarios of using multilingual information sources. For this purpose, we need to obtain more knowledge on the occasions when CLIR is more beneficial as compared with monolingual information retrieval from the utility perspective. The difficulty lies in the ...

On the need for annotation-based image retrieval

Proceedings of the Workshop on Information Retrieval in Context (IRiX), Sheffield, UK, Jul 1, 2004

Compared with content-based image retrieval, annotationbased image retrieval is more practical in... more Compared with content-based image retrieval, annotationbased image retrieval is more practical in some application domains. Users' information needs and the semantic contents of images can be represented by textual information more easily. We describe two problems which are unique to annotation-based image retrieval and would be worthy of further research. Contextual information embedded in data may be used to address these problems.

Retrieving lightly annotated images using image similarities

Proceedings of the 2005 ACM symposium on Applied computing, 2005

Users' search needs are often represented by words and images are retrieved according to such tex... more Users' search needs are often represented by words and images are retrieved according to such textual queries. Annotation words assigned to the stored images are most useful to connect queries to the images. However, due to annotation cost, quite limited amount of annotation words are available in many cases. When annotations are not given at all, there needs to be some techniques that assign annotations automatically. When only a few annotation words are given to each image (lightly annotated), there need to be some enhancement techniques that best use the available annotations. We address the later problem by estimating word associations to fill in the lexical gap between queries and annotations. The model of word associations can be learned from the data. However, since images are only lightly annotated, their sparseness in computing word associations becomes crucial. To compensate the sparseness, we propose a novel data exploration technique in which image similarities contribute to the estimation of word associations on the assumption that similar images have similar semantic concepts. We experimentally show the potential benefit of our approach.

Human Interpretation of Inter-robot Communication

Lecture Notes in Computer Science, 2022

Statistical learning from multiple information sources

Nara Institute of Science and Technology, Mar 24, 2004

In intelligent information processing tasks such as pattern recognition and information retrieval... more In intelligent information processing tasks such as pattern recognition and information retrieval (IR), probabilistic models are now widely used because they can represent ambiguities of observed data and are robust against noises. Parameters of probabilistic models are statistically estimated (learned) from given training data. However, when the training data contain an insufficient amount of information, the learned model becomes unreliable and its performance severely deteriorates. This thesis proposes two novel learning algorithms that use multiple information sources to mitigate this information scarcity problem in the following two applications. The first application is solving classification problems in which optimal class labels are automatically assigned to observations whose class labels are unknown. Among various types of classification problems, this paper considers classification of sequences that consist of sequential observation points. As a classifier, we focus on the hidden Markov model (HMM), which has been widely used for the classification of sequences. Generally, an HMM is trained on labeled data that consist of observed feature values and class labels. However, due to the high labeling cost, the amount of labeled training data is often small. In this thesis, we propose a learning scheme called semi-supervised learning to improve the classification performance even if the amount of labeled training data is small. The proposed scheme uses both of a small amount of labeled data, and unlabeled data that are not usually used for learning. First, we design a suitable HMM structure for using the unlabeled sequences. Then, we formally derive a semisupervised learning algorithm in which the convergence property is theoretically guaranteed. Next, we apply the proposed method to two types of time series sequences: those acquired from sign language sign data and those acquired from speech phoneme data. Experimental results show that the proposed method outperforms conventional methods. The second application is solving IR problems, especially cross-media IR in which queries and corresponding target data belong to different media. More specifically, this thesis focuses on situations where queries are represented as text and target data are images. When textual annotations explaining the contents of images are provided, such cross-media image retrieval can be regarded as ordinary textual IR. In text retrieval, associations between words can be learned from data and used to relate queries and text in the documents. However, in image retrieval, the number of annotations is too small to learn the word relationships. Using the fact that annotations and images are related, we regard images as the paired data of annotations.

Exploitation of Unlabeled Sequences in Hidden Markov Models

This paper presents a method for effectively using unlabeled sequential data in the learning of h... more This paper presents a method for effectively using unlabeled sequential data in the learning of hidden Markov models (HMMs). With the conventional approach, class labels for unlabeled data are assigned deterministically by HMMs learned from labeled data. Such labeling often becomes unreliable when the number of labeled data is small. We propose an extended Baum-Welch (EBW) algorithm in which the labeling is undertaken probabilistically and iteratively so that the labeled and unlabeled data likelihoods are improved. Unlike the conventional approach, the EBW algorithm guarantees convergence to a local maximum of the likelihood. Experimental results on gesture data and speech data show that when labeled training data are scarce, by using unlabeled data, the EBW algorithm improves the classification performance of HMMs more robustly than the conventional naive labeling (NL) approach.

Observational and Accelerometer Analysis of Head Movement Patterns in Psychotherapeutic Dialogue †

Sensors (Basel, Switzerland), 2021

Psychotherapists, who use their communicative skills to assist people, review their dialogue prac... more Psychotherapists, who use their communicative skills to assist people, review their dialogue practices and improve their skills from their experiences. However, technology has not been fully exploited for this purpose. In this study, we analyze the use of head movements during actual psychotherapeutic dialogues between two participants—therapist and client—using video recordings and head-mounted accelerometers. Accelerometers have been utilized in the mental health domain but not for analyzing mental health related communications. We examined the relationship between the state of the interaction and temporally varying head nod and movement patterns in psychological counseling sessions. Head nods were manually annotated and the head movements were measured using accelerometers. Head nod counts were analyzed based on annotations taken from video data. We conducted cross-correlation analysis of the head movements of the two participants using the accelerometer data. The results of two ...

Web Resource Selection for Dialogue System Generating Natural Responses

Communications in Computer and Information Science, 2011

Using Web information in example-based dialogue systems is considered to be a good way to increas... more Using Web information in example-based dialogue systems is considered to be a good way to increase the topical relevance of system responses. However, Web content is mostly written documents, not colloquial utterances. To alleviate the discrepancy in style between written and spoken language, we suggest that the corpus should be selected subsets of the Web, viz., online bulletin boards. The naturalness provided by casual utterances is especially important when the goal of a dialogue is chatting rather than formal question answering. By appropriately selecting the information source, our textbased dialogue system can generate friendly responses to users. These characteristics were evaluated with a questionnaire.

Using Visual Linkages for Multilingual Image Retrieval

Lecture Notes in Computer Science, 2007

The use of visual features in text-based ad-hoc image retrieval is challenging. Visual and textua... more The use of visual features in text-based ad-hoc image retrieval is challenging. Visual and textual information have so far been treated equally in terms of their properties and combined with weighting mechanisms for balancing their contributions to the ranking. The use of visual and textual information in a single retrieval system sometimes limits its applicability due to the lack of modularity. In this paper, we propose an image retrieval method that separates the usage of visual information from that of textual information. Visual clustering establishes linkages between images and the relationships are later used for the reranking. By applying clustering on visual features prior to the ranking, the main retrieval process becomes purely text-based. Experimental results on the ImageCLEFphoto ad-hoc task show this scheme is suitable for querying multilingual collections on some search topics.

Mining Visual Knowledge for Multi-Lingual Image Retrieval

21st International Conference on Advanced Information Networking and Applications Workshops (AINAW'07), 2007

Users commonly rely just on scarce textual annotation when their searches for images are semantic... more Users commonly rely just on scarce textual annotation when their searches for images are semantic or conceptual based. Rich visual information is often thrown away in basic annotation-based image retrieval because its relationship to the semantic content is not always clear. To ensure that appropriate visual information is included, we propose using visual clustering within pre-processing and post-processing steps of text-based retrieval. A clustering algorithm finds pairs of images that are nearly identical and are, therefore, presumed semantically similar. The output from basic retrieval systems is a ranked list of images based only on lexical term matching. The obtained cluster knowledge is then used to modify the ranking result during the post-processing step. Low ranked images considered nearly identical to more highly ranked images are then pulled up. The modularity of this architecture allows us to integrate a data mining process without having to change core information retrieval systems. Evaluation on a cross-language image retrieval test collection showed that this method improved retrieval performance for certain queries in multilingual settings.

Image retrieval: Research and use in the information explosion

Progress in Informatics, 2009

We surveyed the current research on the retrieval and utilization of images. This paper provides ... more We surveyed the current research on the retrieval and utilization of images. This paper provides an overview of the difficulties and possibilities of technological or technology-related research topics resulting from the huge amount of images currently available and highlights some of the important research topics. We looked at the ongoing research activities and analyzed them from four aspects, information access and organization technology, the computing infrastructure that enables access to large-scale image resources, issues in human-system interaction and human factors related to using images, and the social aspect of image media. On the technical side, we noticed that as the number of digital images increases, so does the importance of the accuracy and scalability in relation to the image retrieval. The accuracy and scalability are in fact needed to cope with the current explosion in digital images. On the social side, not so long ago, image retrieval technologies were only experimental tools or used by experts within a limited domain. However, now the general public has access to a wide range of digital images, which means that image retrieval technologies are being used by various users in a large diversity of social contexts. Thus, as we show in this paper, the accuracy and scalability are not the only important factors in the era of information explosion, but that researchers must also be concerned with the social aspects of these technologies.

Utility of Gestural Cues in Indexing Semantic Miscommunication

2010 5th International Conference on Future Information Technology, 2010

In multimedia data analysis, automated indexing of conversational video is an emerging topic. One... more In multimedia data analysis, automated indexing of conversational video is an emerging topic. One challenging problem in this topic is the recognition of higher-level concepts, such as miscommunications in conversations. While detecting miscommunications is generally easy for speakers as well as observers, it is not currently understood which cues contribute to their detection and to what extent. To make use of the knowledge on gestural cues in multimedia systems, the applicability of machine learning is investigated as a means of detecting miscommunication from gestural patterns observed in psychotherapeutic face-to-face conversations. Various features are taken from gesture data, and both simple and complex classifiers are constructed using these features. Both short-term and long-term effects are tested using different time window sizes. Also, two types of gestures, communicative and non-communicative, are considered. The experimental results suggest that there is no single gestural feature that can explain the occurrence of semantic miscommunication. Another interesting finding is that gestural cues correlate more with long-term gestural patterns than with short-term ones.

Interaction and user experiences with multimedia technologies

Proceeding of the 3rd ACM international workshop on Human-centered computing - HCC '08, 2008

In this paper, we investigate future topics and challenges of interaction and user experience in ... more In this paper, we investigate future topics and challenges of interaction and user experience in multimedia. We bring together different perspectives from overlapping fields of research such as multimedia, human-computer interaction, information retrieval, networked multimedia, and creative arts. Based on potential intersections, we define three application domains to be investigated further, as they create high demand and good prospect for long-lasting developments in the future. These application domains are: media working environments, media enter-/edutainment, and social media engagement. Each application domain is analyzed along five dimensions, namely: information quality, presentation quality, ambience, interactivity, and user expectations. Based on this analysis, we identify the most pressing research questions and key challenges for each area. Finally, we advocate a user-centered approach to tackle these challenges and questions in order to develop relevant multimedia applications that best meet the users' expectations.

User-Model-Based Evaluation for Interactive Image Retrieval

2010 IEEE/ACIS 9th International Conference on Computer and Information Science, 2010

User-system interaction is sometimes a cumbersome element of non-textual information access. Imag... more User-system interaction is sometimes a cumbersome element of non-textual information access. Image retrieval systems now incorporate various interaction mechanisms. However, there is no standard or established way of interacting that has been proven to be most effective. Therefore, many system designers are introducing or eliminating various functionalities. The bottleneck in the development effort is finding an effective and fair way to evaluate these systems. For static information access, several test collections have commonly been used. For interactive information access, expensive laboratory testing is needed. This paper presents a user semi-automatic user-modelbased evaluation method that is designed to reduce cost while maintaining objectivity.

Query Types and Visual Concept-Based Post-retrieval Clustering

Lecture Notes in Computer Science, 2009

In the photo retrieval task of ImageCLEF 2008, we examined the influences of image representation... more In the photo retrieval task of ImageCLEF 2008, we examined the influences of image representations and clustering methods for enhancing instance recall when they are used in post-retrieval clustering. Two types of visual concepts and hierarchical and partitioning clustering methods were compared. We used the title fields in the search topics, and either only the title fields or both the title and description fields of the annotations were in English. The experimental results showed that one type of visual concept representation dominated the other except for in one condition. Also, it was found that hierarchical clustering can enhance instance recall while preserving the precision when the threshold parameters are appropriately set. In contrast, partitioning clustering degraded the results. We also categorized the queries based on the clustering criteria, geographical and non-geographical, and found that the geographical queries are relatively easy and post-retrieval clustering works better for them.

Manual and Accelerometer Analysis of Head Nodding Patterns in Goal-oriented Dialogues

Lecture Notes in Computer Science, 2011

We studied communication patterns in face-to-face dialogues between people for the purpose of ide... more We studied communication patterns in face-to-face dialogues between people for the purpose of identifying conversation features that can be exploited to improve human-computer interactions. We chose to study the psychological counseling setting as it provides good examples of task-oriented dialogues. The dialogues between two participants, therapist and patient, were video recorded. The participants' head movements were measured by using head-mounted accelerometers. The relationship between the dialogue process and head nodding frequency was analyzed on the basis of manual annotations. The segments where nods of the two participants correlated were identified on the basis of the accelerometer data. Our analysis suggests that there are characteristic nodding patterns in different dialogue stages.

Gestural cue analysis in automated semantic miscommunication annotation

Multimedia Tools and Applications, 2011

Automated annotation of conversational video using semantic labels is a challenging topic. We inv... more Automated annotation of conversational video using semantic labels is a challenging topic. We investigated miscommunication as an example of a higher-level semantic concept in conversations. Miscommunication is an important obstacle in solving psychological problems such as those in psychotherapeutic interviews. Although miscommunications are often obvious to the speakers as well as observers, it is difficult for machines to detect them from low-level features. This difficulty is due to the lack of understanding on which cues contribute to miscommunication. Miscommunications are mainly associated with spoken content, though non-verbal elements may play a role. Among various non-verbal features, we investigate gestural cues. Various features are taken from gesture data, and both simple and complex classifiers are constructed using machine learning. The experimental results suggest that there is no single gestural feature that can predict or explain the occurrence of semantic miscommunication.

Multimodal Corpus for Psychotherapeutic Situations

Multimodal Corpora: How Should Multimodal Corpora Deal with the Situation? Workshop Programme

This paper presents a design principle for construction of an in-house multimodal corpus for comp... more This paper presents a design principle for construction of an in-house multimodal corpus for computationally analysing and better understanding conversations during psychotherapy. We discuss some sharable information about research community data collection procedures such as recording devices, consent forms, and privacy considerations. We also explain multimodal coding schema and metadata that are needed in the domain. The corpus has three distinguishing properties: 1) it was constructed only for our research ...

Human Judgment on Humor Expressions in a Community-Based Question-Answering Service

For understanding humorous dialogue, a collection of humorous expressions is needed. In addition ... more For understanding humorous dialogue, a collection of humorous expressions is needed. In addition to humorous expressions, their annotations are important to be used as language resources. In this paper, we analyzed how human assessors annotate humorous expressions extracted from an online community-based questionanswering (CQA) corpus, which contains many interesting examples of humorous communication. We analyzed the annotation results of a collection of humorous expressions as done by 28 annotators in ...

Collecting Humorous Expressions from a Community-based Question-answering-service Corpus

We proposed a method of collecting humorous expressions from an online community-based question-a... more We proposed a method of collecting humorous expressions from an online community-based question-answering (CQA) corpus where some users post a variety of questions and other users post relevant answers. Although the service is created for the purpose of knowledge exchange, there are users who enjoy posting humorous responses. Therefore, the corpus contains many interesting humour communication examples that might be useful in understanding the nature of online communications and variations in ...

The remarkable search topic-finding task to share success stories of cross-language information retrieval

Proceedings of the Fifth Workshop on Important Unresolved Matters, Jun 29, 2005

The performance of cross-language information retrieval (CLIR) systems has been improved to the l... more The performance of cross-language information retrieval (CLIR) systems has been improved to the level of practical use. The next step is to inform potential users that CLIR technologies are ready to be used. A good way of doing this is to present attractive scenarios of using multilingual information sources. For this purpose, we need to obtain more knowledge on the occasions when CLIR is more beneficial as compared with monolingual information retrieval from the utility perspective. The difficulty lies in the ...

On the need for annotation-based image retrieval

Proceedings of the Workshop on Information Retrieval in Context (IRiX), Sheffield, UK, Jul 1, 2004

Compared with content-based image retrieval, annotationbased image retrieval is more practical in... more Compared with content-based image retrieval, annotationbased image retrieval is more practical in some application domains. Users' information needs and the semantic contents of images can be represented by textual information more easily. We describe two problems which are unique to annotation-based image retrieval and would be worthy of further research. Contextual information embedded in data may be used to address these problems.