In this article, we present our preliminary research in the field of the automatic sentence gener... more In this article, we present our preliminary research in the field of the automatic sentence generation. With a corpus used as a basis to generate new sentences, we have explored various techniques of word and n grams combinations in conjunction with original approaches of filtering generated sentences to reduce the impact of combinatory explosion. Finally, we have adapted the statistical
In this paper we study the Enertex model that has been applied to fundamental tasks in Natural La... more In this paper we study the Enertex model that has been applied to fundamental tasks in Natural Language Processing (NLP) including automatic document summarization and topic segmentation. The model is language independent. It is based on the intuitive concept of Textual Energy, inspired by Neural Networks and Statistical Physics of magnetic systems. It can be implemented using simple matrix operations and on the contrary of PageRank algorithms, it avoids any iterative process.
This paper deals with a new strategy to evaluate a Natural Language Processing (NLP) complex task... more This paper deals with a new strategy to evaluate a Natural Language Processing (NLP) complex task using the Turing test. Automatic summarization based on sentence compression requires to asses informativeness and modify inner sentence structures. This is much more intrinsically related with real rephrasing than plain sentence extraction and ranking paradigm so new evaluation methods are needed. We propose a novel imitation game to evaluate Automatic Summarization by Compression (ASC). Rationale of this Turing-like evaluation could be applied to many other NLP complex tasks like Machine translation or Text Generation. We show that a state of the art ASC system can pass such a test and simulate a human summary in 60% of the cases.
A Symbolic Approach for Automatic Detection of Nuclearity and Rhetorical Relations among Intra-se... more A Symbolic Approach for Automatic Detection of Nuclearity and Rhetorical Relations among Intra-sentence Discourse Segments in Spanish 2 3.2. RST Spanish Treebank 4. Corpus analysis 5. System implementation 6. Experiments and evaluation 7. Conclusions and future work A Symbolic Approach for Automatic Detection of Nuclearity and Rhetorical Relations among Intra-sentence Discourse Segments in Spanish 5
We have developed three Automatic Summarization systems (Cortex and Enertex based on the vectoria... more We have developed three Automatic Summarization systems (Cortex and Enertex based on the vectorial model, and another based on Latent Semantic Analysis LSA). These systems use methods that do not require linguistic resources. In this study, we confront them to the summaries made by students of different levels (middle school, high school and graduate school). Finally, the results of this study allow us to determine which summarization system best corresponds to a given school level. Résumé Nous avons développé trois systèmes de Résumé Automatique (Cortex et Enertex basés sur le modèle vectoriel, et un autre basé sur l'Analyse de la Sémantique Latente LSA). Ces systèmes reposent principalement sur des méthodes qui n'ont (presque) pas besoin de ressources linguistiques. Dans ce travail nous les confrontons avec les jugements d'élèves et d'étudiants de différents niveaux scolaires (collège, lycée et master). Cette expérience nous a permis de déterminer le niveau scolaire auquel correspondent le mieux les systèmes résumant automatiquement des textes.
Dans cet article, nous présentons des applications du système Enertex au Traitement Automatique d... more Dans cet article, nous présentons des applications du système Enertex au Traitement Automatique de la Langue Naturelle. Enertex est basé sur l'énergie textuelle, une approche par réseaux de neurones inspirée de la physique statistique des systèmes magnétiques. Nous avons appliqué cette approche aux problèmes du résumé automatique multi-documents et de la détection de frontières thématiques. Les résultats, en trois langues : anglais, espagnol et français, sont très encourageants.
Focused Retrieval and Evaluation (INEX 2009)-Lecture Notes in Computer Science LNCS
Abstract. QA@ INEX aims to evaluate a complex question-answering task. In such a task, the set of... more Abstract. QA@ INEX aims to evaluate a complex question-answering task. In such a task, the set of questions is composed of factoid, precise questions that expect short answers, as well as more complex questions that can be answered by several sentences or by an aggregation of texts from different documents. Question-answering, XML/passage retrieval and automatic summarization are combined in order to get closer to real information needs.
Twitter is increasingly used for on-line client and audience fishing, this motivated the tweet co... more Twitter is increasingly used for on-line client and audience fishing, this motivated the tweet contextualization task at INEX. The objective is to help a user to understand a tweet by providing him with a short summary (500 words). This summary should be built automatically using local resources like the Wikipedia and generated by extracting relevant passages and aggregating them into a coherent summary. The task is evaluated considering informativeness which is computed using a variant of Kullback-Leibler divergence and passage pooling. Meanwhile effective readability in context of summaries is checked using binary questionnaires on small samples of results. Running since 2010, results show that only systems that efficiently combine passage retrieval, sentence segmentation and scoring, named entity recognition, text POS analysis, anaphora detection, diversity content measure as well as sentence reordering are effective.
INEX investigates focused retrieval from structured documents by providing large test collections... more INEX investigates focused retrieval from structured documents by providing large test collections of structured documents, uniform evaluation measures, and a forum for organizations to compare their results. This paper reports on the INEX 2010 evaluation campaign, which consisted of a wide range of tracks:
In this paper, we investigate the issue of modeling corporate entities' online reputation. We int... more In this paper, we investigate the issue of modeling corporate entities' online reputation. We introduce a bayesian latent probabilistic model approach for e-Reputation analysis based on Dimensions (Reputational Concepts) Categorization and Opinion Mining from textual content. Dimensions to analyze e-Reputation are set up by analyst as latent variables. Machine Learning (ML) Natural Language Processing (NLP) approaches are used to label large sets of text passages. For each Dimension, several estimations of the relationship with each text passage are computed as well as Opinion and Priority. The proposed automatic path modeling algorithm explains Opinion or Priority scores based on selected Dimensions. Model Robustness' is evaluated over RepLab dataset.
We study a new content-based method for the evaluation of text summarization systems without huma... more We study a new content-based method for the evaluation of text summarization systems without human models which is used to produce system rankings. The research is carried out using a new content-based evaluation framework called FRESA to compute a variety of divergences among probability distributions. We apply our comparison framework to various well-established content-based evaluation measures in text summarization such as COVERAGE, RESPONSIVENESS, PYRAMIDS and ROUGE studying their associations in various text summarization tasks including generic multi-document summarization in English and French, focus-based multi-document summarization in English and generic single-document summarization in French and Spanish.
Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval - SIGIR '13, 2013
Improving query understanding is crucial for providing the user with information that suits her n... more Improving query understanding is crucial for providing the user with information that suits her needs. To this end, the retrieval system must be able to deal with several sources of knowledge from which it could infer a topical context. The use of external sources of information for improving document retrieval has been extensively studied. Improvements with either structured or large sets of data have been reported. However, in these studies resources are often used separately and rarely combined together. We experiment in this paper a method that discounts documents based on their weighted divergence from a set of external resources. We present an evaluation of the combination of four resources on two standard TREC test collections. Our proposed method significantly outperforms a state-of-the-art Mixture of Relevance Models on one test collection, while no significant differences are detected on the other one.
A keyword query is the representation of the information need of a user, and is the result of a c... more A keyword query is the representation of the information need of a user, and is the result of a complex cognitive process which often results in under-specification. We propose an unsupervised method namely Latent Concept Modeling (LCM) for mining and modeling latent search concepts in order to recreate the conceptual view of the original information need. We use Latent Dirichlet Allocation (LDA) to exhibit highly-specific query-related topics from pseudo-relevant feedback documents. We define these topics as the latent concepts of the user query. We perform a thorough evaluation of our approach over two large ad-hoc TREC collections. Our findings reveal that the proposed method accurately models latent concepts, while being very effective in a query expansion retrieval setting. RÉSUMÉ. Une requête est la représentation du besoin d'information d'un utilisateur, et est le résultat d'un processus cognitif complexe qui mène souvent à un mauvais choix de mots-clés. Nous proposons une méthode non supervisée pour la modélisation de concepts implicites d'une requête, dans le but de recréer la représentation conceptuelle du besoin d'information initial. Nous utilisons l'allocation de Dirichlet latente (LDA) pour détecter les concepts implicites de la requête en utilisant des documents pseudo-pertinents. Nous évaluons cette méthode en profondeur en utilisant deux collections de test de TREC. Nous trouvons notamment que notre approche permet de modéliser précisément les concepts implicites de la requête, tout en obtenant de bonnes performances dans le cadre d'une recherche de documents.
INEX investigates focused retrieval from structured documents by providing large test collections... more INEX investigates focused retrieval from structured documents by providing large test collections of structured documents, uniform evaluation measures, and a forum for organizations to compare their results. This paper reports on the INEX'12 evaluation campaign, which consisted of a five tracks: Linked Data, Relevance Feedback, Snippet Retrieval, Social Book Search, and Tweet Contextualization. INEX'12 was an exciting year for INEX in which we joined forces with CLEF and for the first time ran our workshop as part of the CLEF labs in order to facilitate knowledge transfer between the evaluation forums.
INEX investigates focused retrieval from structured documents by providing large test collections... more INEX investigates focused retrieval from structured documents by providing large test collections of structured documents, uniform evaluation measures, and a forum for organizations to compare their results. This paper reports on the INEX 2013 evaluation campaign, which consisted of four activities addressing three themes: searching professional and user generated data (Social Book Search track); searching structured or semantic data (Linked Data track); and focused retrieval (Snippet Retrieval and Tweet Contextualization tracks). INEX 2013 was an exciting year for INEX in which we consolidated the collaboration with (other activities in) CLEF and for the second time ran our workshop as part of the CLEF labs in order to facilitate knowledge transfer between the evaluation forums. This paper gives an overview of all the INEX 2013 tracks, their aims and task, the built test-collections, and gives an initial analysis of the results.
In this article, we present our preliminary research in the field of the automatic sentence gener... more In this article, we present our preliminary research in the field of the automatic sentence generation. With a corpus used as a basis to generate new sentences, we have explored various techniques of word and n grams combinations in conjunction with original approaches of filtering generated sentences to reduce the impact of combinatory explosion. Finally, we have adapted the statistical
In this paper we study the Enertex model that has been applied to fundamental tasks in Natural La... more In this paper we study the Enertex model that has been applied to fundamental tasks in Natural Language Processing (NLP) including automatic document summarization and topic segmentation. The model is language independent. It is based on the intuitive concept of Textual Energy, inspired by Neural Networks and Statistical Physics of magnetic systems. It can be implemented using simple matrix operations and on the contrary of PageRank algorithms, it avoids any iterative process.
This paper deals with a new strategy to evaluate a Natural Language Processing (NLP) complex task... more This paper deals with a new strategy to evaluate a Natural Language Processing (NLP) complex task using the Turing test. Automatic summarization based on sentence compression requires to asses informativeness and modify inner sentence structures. This is much more intrinsically related with real rephrasing than plain sentence extraction and ranking paradigm so new evaluation methods are needed. We propose a novel imitation game to evaluate Automatic Summarization by Compression (ASC). Rationale of this Turing-like evaluation could be applied to many other NLP complex tasks like Machine translation or Text Generation. We show that a state of the art ASC system can pass such a test and simulate a human summary in 60% of the cases.
A Symbolic Approach for Automatic Detection of Nuclearity and Rhetorical Relations among Intra-se... more A Symbolic Approach for Automatic Detection of Nuclearity and Rhetorical Relations among Intra-sentence Discourse Segments in Spanish 2 3.2. RST Spanish Treebank 4. Corpus analysis 5. System implementation 6. Experiments and evaluation 7. Conclusions and future work A Symbolic Approach for Automatic Detection of Nuclearity and Rhetorical Relations among Intra-sentence Discourse Segments in Spanish 5
We have developed three Automatic Summarization systems (Cortex and Enertex based on the vectoria... more We have developed three Automatic Summarization systems (Cortex and Enertex based on the vectorial model, and another based on Latent Semantic Analysis LSA). These systems use methods that do not require linguistic resources. In this study, we confront them to the summaries made by students of different levels (middle school, high school and graduate school). Finally, the results of this study allow us to determine which summarization system best corresponds to a given school level. Résumé Nous avons développé trois systèmes de Résumé Automatique (Cortex et Enertex basés sur le modèle vectoriel, et un autre basé sur l'Analyse de la Sémantique Latente LSA). Ces systèmes reposent principalement sur des méthodes qui n'ont (presque) pas besoin de ressources linguistiques. Dans ce travail nous les confrontons avec les jugements d'élèves et d'étudiants de différents niveaux scolaires (collège, lycée et master). Cette expérience nous a permis de déterminer le niveau scolaire auquel correspondent le mieux les systèmes résumant automatiquement des textes.
Dans cet article, nous présentons des applications du système Enertex au Traitement Automatique d... more Dans cet article, nous présentons des applications du système Enertex au Traitement Automatique de la Langue Naturelle. Enertex est basé sur l'énergie textuelle, une approche par réseaux de neurones inspirée de la physique statistique des systèmes magnétiques. Nous avons appliqué cette approche aux problèmes du résumé automatique multi-documents et de la détection de frontières thématiques. Les résultats, en trois langues : anglais, espagnol et français, sont très encourageants.
Focused Retrieval and Evaluation (INEX 2009)-Lecture Notes in Computer Science LNCS
Abstract. QA@ INEX aims to evaluate a complex question-answering task. In such a task, the set of... more Abstract. QA@ INEX aims to evaluate a complex question-answering task. In such a task, the set of questions is composed of factoid, precise questions that expect short answers, as well as more complex questions that can be answered by several sentences or by an aggregation of texts from different documents. Question-answering, XML/passage retrieval and automatic summarization are combined in order to get closer to real information needs.
Twitter is increasingly used for on-line client and audience fishing, this motivated the tweet co... more Twitter is increasingly used for on-line client and audience fishing, this motivated the tweet contextualization task at INEX. The objective is to help a user to understand a tweet by providing him with a short summary (500 words). This summary should be built automatically using local resources like the Wikipedia and generated by extracting relevant passages and aggregating them into a coherent summary. The task is evaluated considering informativeness which is computed using a variant of Kullback-Leibler divergence and passage pooling. Meanwhile effective readability in context of summaries is checked using binary questionnaires on small samples of results. Running since 2010, results show that only systems that efficiently combine passage retrieval, sentence segmentation and scoring, named entity recognition, text POS analysis, anaphora detection, diversity content measure as well as sentence reordering are effective.
INEX investigates focused retrieval from structured documents by providing large test collections... more INEX investigates focused retrieval from structured documents by providing large test collections of structured documents, uniform evaluation measures, and a forum for organizations to compare their results. This paper reports on the INEX 2010 evaluation campaign, which consisted of a wide range of tracks:
In this paper, we investigate the issue of modeling corporate entities' online reputation. We int... more In this paper, we investigate the issue of modeling corporate entities' online reputation. We introduce a bayesian latent probabilistic model approach for e-Reputation analysis based on Dimensions (Reputational Concepts) Categorization and Opinion Mining from textual content. Dimensions to analyze e-Reputation are set up by analyst as latent variables. Machine Learning (ML) Natural Language Processing (NLP) approaches are used to label large sets of text passages. For each Dimension, several estimations of the relationship with each text passage are computed as well as Opinion and Priority. The proposed automatic path modeling algorithm explains Opinion or Priority scores based on selected Dimensions. Model Robustness' is evaluated over RepLab dataset.
We study a new content-based method for the evaluation of text summarization systems without huma... more We study a new content-based method for the evaluation of text summarization systems without human models which is used to produce system rankings. The research is carried out using a new content-based evaluation framework called FRESA to compute a variety of divergences among probability distributions. We apply our comparison framework to various well-established content-based evaluation measures in text summarization such as COVERAGE, RESPONSIVENESS, PYRAMIDS and ROUGE studying their associations in various text summarization tasks including generic multi-document summarization in English and French, focus-based multi-document summarization in English and generic single-document summarization in French and Spanish.
Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval - SIGIR '13, 2013
Improving query understanding is crucial for providing the user with information that suits her n... more Improving query understanding is crucial for providing the user with information that suits her needs. To this end, the retrieval system must be able to deal with several sources of knowledge from which it could infer a topical context. The use of external sources of information for improving document retrieval has been extensively studied. Improvements with either structured or large sets of data have been reported. However, in these studies resources are often used separately and rarely combined together. We experiment in this paper a method that discounts documents based on their weighted divergence from a set of external resources. We present an evaluation of the combination of four resources on two standard TREC test collections. Our proposed method significantly outperforms a state-of-the-art Mixture of Relevance Models on one test collection, while no significant differences are detected on the other one.
A keyword query is the representation of the information need of a user, and is the result of a c... more A keyword query is the representation of the information need of a user, and is the result of a complex cognitive process which often results in under-specification. We propose an unsupervised method namely Latent Concept Modeling (LCM) for mining and modeling latent search concepts in order to recreate the conceptual view of the original information need. We use Latent Dirichlet Allocation (LDA) to exhibit highly-specific query-related topics from pseudo-relevant feedback documents. We define these topics as the latent concepts of the user query. We perform a thorough evaluation of our approach over two large ad-hoc TREC collections. Our findings reveal that the proposed method accurately models latent concepts, while being very effective in a query expansion retrieval setting. RÉSUMÉ. Une requête est la représentation du besoin d'information d'un utilisateur, et est le résultat d'un processus cognitif complexe qui mène souvent à un mauvais choix de mots-clés. Nous proposons une méthode non supervisée pour la modélisation de concepts implicites d'une requête, dans le but de recréer la représentation conceptuelle du besoin d'information initial. Nous utilisons l'allocation de Dirichlet latente (LDA) pour détecter les concepts implicites de la requête en utilisant des documents pseudo-pertinents. Nous évaluons cette méthode en profondeur en utilisant deux collections de test de TREC. Nous trouvons notamment que notre approche permet de modéliser précisément les concepts implicites de la requête, tout en obtenant de bonnes performances dans le cadre d'une recherche de documents.
INEX investigates focused retrieval from structured documents by providing large test collections... more INEX investigates focused retrieval from structured documents by providing large test collections of structured documents, uniform evaluation measures, and a forum for organizations to compare their results. This paper reports on the INEX'12 evaluation campaign, which consisted of a five tracks: Linked Data, Relevance Feedback, Snippet Retrieval, Social Book Search, and Tweet Contextualization. INEX'12 was an exciting year for INEX in which we joined forces with CLEF and for the first time ran our workshop as part of the CLEF labs in order to facilitate knowledge transfer between the evaluation forums.
INEX investigates focused retrieval from structured documents by providing large test collections... more INEX investigates focused retrieval from structured documents by providing large test collections of structured documents, uniform evaluation measures, and a forum for organizations to compare their results. This paper reports on the INEX 2013 evaluation campaign, which consisted of four activities addressing three themes: searching professional and user generated data (Social Book Search track); searching structured or semantic data (Linked Data track); and focused retrieval (Snippet Retrieval and Tweet Contextualization tracks). INEX 2013 was an exciting year for INEX in which we consolidated the collaboration with (other activities in) CLEF and for the second time ran our workshop as part of the CLEF labs in order to facilitate knowledge transfer between the evaluation forums. This paper gives an overview of all the INEX 2013 tracks, their aims and task, the built test-collections, and gives an initial analysis of the results.
Uploads
Papers by E. Sanjuan