The identification of the key concepts in a set of documents is a useful source of information fo... more The identification of the key concepts in a set of documents is a useful source of information for several information access applications. We are interested in its application to multi-document summarization, both for the automatic generation of summaries and for interactive summarization systems.
In this paper, we describe UNED’s participation in the iCLEF 2005 track. We have compared two str... more In this paper, we describe UNED’s participation in the iCLEF 2005 track. We have compared two strategies for finding an answer using an interactive question answering system: i) a search system over full documents and ii) a search system over passages (document’s paragraphs). We have added an interesting feature to both system in order to facilitate reading: the possibility to enable/disable the highlighting of named entities such as proper names, temporal references and numbers likely to contain the right answer. Our Document Searcher obtained better overall accuracy (.53 vs..45) but our subjects found browsing passages simpler and faster. However, most of them presented a similar search behavior (regarding time consumption, confidence in their answers and query refinements) using both systems. All our users considered helpful the highlighting of named entities and they all made extensive use of this possibility as a quick way of discriminating between relevant and non relevant doc...
We have participated in the SENSEVAL-2 English tasks (all words and lexical sample) with an unsup... more We have participated in the SENSEVAL-2 English tasks (all words and lexical sample) with an unsupervised system based on mutual information measured over a large corpus (277 million words) and some additional heuristics. A supervised extension of the system was also presented to the lexical sample task. Our system scored first among unsupervised systems in both tasks: 56.9 % recall in all words, 40.2 % in lexical sample. This is slightly worse than the first sense heuristic for all. words and 3.6 % better for the lexical sample, a strong indication that unsupervised Word Sense Disambiguation remains being a strong challenge.
This paper presents a probabilistic framework, QARLA, for the evaluation of text summarisation sy... more This paper presents a probabilistic framework, QARLA, for the evaluation of text summarisation systems. The input of the framework is a set of manual (reference) summaries, a set of baseline (automatic) summaries and a set of similarity metrics between summaries. It provides i) a measure to evaluate the quality of any set of similarity metrics, ii) a measure to evaluate the quality of a summary using an optimal set of similarity metrics, and iii) a measure to evaluate whether the set of baseline summaries is reliable or may produce biased results. Compared to previous approaches, our framework is able to combine different metrics and evaluate the quality of a set of metrics without any a-priori weighting of their relative importance. We provide quantitative evidence about the effectiveness of the approach to improve the automatic evaluation of text summarisation systems by combining several similarity metrics. 1
This paper presents the ITEM multilingual search engine. This search engine performs full lexical... more This paper presents the ITEM multilingual search engine. This search engine performs full lexical processing (morphological analysis, tagging and Word Sense Disambiguation) on documents and queries in order to provide language-neutral indexes for querying and retrieval. The indexing terms are the EuroWordNet/ITEM InterLingual Index records that link wordnets in 10 languages of the European Community (the search engine currently supports Spanish, English and Catalan). The goal of this application is to provide a way of comparing in context the behavior of different Natural Language Processing strategies for Cross-Language Information Retrieval (CLIR) and, in particular, different Word Sense Disambiguation strategies for query translation and conceptual indexing. 1.
This paper describes the exemplar based approach presented by UNED at Senseval-3. Instead of repr... more This paper describes the exemplar based approach presented by UNED at Senseval-3. Instead of representing contexts as bags of terms and defining a similarity measure between contexts, we propose to represent terms as bags of contexts and define a similarity measure between terms. Thus, words, lemmas and senses are represented in the same space (the context space), and similarity measures can be defined between them. New contexts are transformed into this representation in order to calculate their similarity to the candidate senses. We show how standard similarity measures obtain better results in this framework. A new similarity measure in the context space is proposed for selecting the senses and performing disambiguation. Results of this approach at Senseval-3 are here reported.
Although computing similarity is one of the fundamental challenges of Information Access tasks, t... more Although computing similarity is one of the fundamental challenges of Information Access tasks, the notion of similarity is not yet completely understood from a formal, axiomatic perspective. In this paper we show how axiomatic explanations of similarity from other fields (i.e. Tversky’s axioms from the point of view of cognitive sciences, and metric spaces from the point of view of algebra) do not completely fit the problem of similarity in Information Access, and we propose a new set of axioms which can be synthesized into a single Similarity Information Monotonicity axiom (SIM). Directly grounded on SIM, we then introduce a new similarity model, the Information Contrast Model, which generalizes both Tversky’s linear contrast model and Pointwise Mutual Information, and, unlike previous similarity models, satisfies the SIM axiom for a certain range of values of its parameters.
This paper reports the description of the developed system and the results obtained in the partic... more This paper reports the description of the developed system and the results obtained in the participation of the UNED in the Second Recognizing Textual Entailment (RTE) Challenge. New techniques and tools have been added: enriched queries to WordNet, detection of numeric expressions and their entailment, and Support Vector Machine classi cation (SVM) are the more relevant. The accuracy performed is slightly higher than the one from the previous edition system.
Resumen La distribuci??n de contenidos y/o metadatos en Repositorios de Objetos de Aprendizaje es... more Resumen La distribuci??n de contenidos y/o metadatos en Repositorios de Objetos de Aprendizaje es un aspecto clave a la hora de favorecer el intercambio de conocimiento dentro de comunidades de aprendizaje. En este art??culo se presenta una soluci??n de ...
This paper reports the participation of the Spanish Distance Learning University (UNED) in the Fi... more This paper reports the participation of the Spanish Distance Learning University (UNED) in the First Answer Validation Exercise (AVE) celebrated within the Cross Language Evaluation Forum (CLEF) 2006 edition. The system works for the Spanish language. It is based on a Support Vector Machine (SVM) classification of the pairs <text, hypothesis> given by the organization. This classification is accomplished by means of a set of features obtained from lexical analysis. Yet Another Learning Environment (Yale 3.0) was used for the SVM classification. Freeling was the toolkit elected for lemmatization and named entities recognition. Two runs were submitted and the results obtained, as defined by the organizers, were the following: precision run1 = 0.467, recall run1 = 0.7168, F run1 = 0.5655, precision run2 = 0.4652, recall run2 = 0.7079, F run2 = 0.5615.
In this article the incorporation of interoperability into a distributed eLearning system, namely... more In this article the incorporation of interoperability into a distributed eLearning system, namely the ActiveDocument platform, is considered. Three degrees of interoperability are defined for such a platform, namely syntactic, semantic, and functional. Subsequently, for the first two degrees of interoperability, a proposal is made for the way in which it can be obtained, backed up with details about the design and implementation of the ActiveDocument platform and its tools, which show how it has been (or is being) ...
We describe an algorithm that combines lexical information (from WordNet 1.7) with Web directorie... more We describe an algorithm that combines lexical information (from WordNet 1.7) with Web directories (from the Open Directory Project) to associate word senses with such directories. Such associations can be used as rich characterizations to acquire sense-tagged corpora automatically, cluster topically-related senses and detect sense specializations. The algorithm is evaluated for the 29 nouns (147 senses) used in the Senseval 2 competition, obtaining 148 (word sense,Web directory) associations covering 88 % of the domain-specific word senses in the test data with 86 % accuracy. The richness of Web directories as sense characterizations is evaluated in a supervised Word Sense Disambiguation task using the Senseval 2 test suite. The results indicate that, when the directory/word sense association is correct, the samples automatically acquired from the Web directories are nearly as valid for training as the original Senseval 2 training instances. The results support our hypothesis that ...
Journal of Artificial Intelligence Research, Sep 1, 2011
ABSTRACT Many Artificial Intelligence tasks cannot be evaluated with a single quality criterion a... more ABSTRACT Many Artificial Intelligence tasks cannot be evaluated with a single quality criterion and some sort of weighted combination is needed to provide system rankings. A problem of weighted combination measures is that slight changes in the relative weights may produce substantial changes in the system rankings. This paper introduces the Unanimous Improvement Ratio (UIR), a measure that complements standard metric combination criteria (such as van Rijsbergen&#39;s F-measure) and indicates how robust the measured differences are to changes in the relative weights of the individual metrics. UIR is meant to elucidate whether a perceived difference between two systems is an artifact of how individual metrics are weighted. Besides discussing the theoretical foundations of UIR, this paper presents empirical results that confirm the validity and usefulness of the metric for the Text Clustering problem, where there is a tradeoff between precision and recall based metrics and results are particularly sensitive to the weighting scheme used to combine them. Remarkably, our experiments show that UIR can be used as a predictor of how well differences between systems measured on a given test bed will also hold in a different test bed.
Este artículo muestra la aplicación del marco de evaluación QARLA sobre los resúmenes evaluados e... more Este artículo muestra la aplicación del marco de evaluación QARLA sobre los resúmenes evaluados en el foro DUC-2004, para las tareas 2 y 5. El marco QARLA permite evaluar de forma automática los sistemas según diferentes aspectos (métricas de similitud) en relación a un conjunto de resúmenes modelo, identificando así los aspectos más deficitarios de las estrategias de resumen existentes. Por otro lado, el marco QARLA permite combinar y meta-evaluar diferentes métricas de similitud, otorgando más peso a los aspectos que caracterizan a los modelos en relación a los resúmenes automáticos.
This paper summarizes the participation of the UNED group in the CLEF 2002 Interactive Track. We ... more This paper summarizes the participation of the UNED group in the CLEF 2002 Interactive Track. We focused on interactive query formulation and refinement, comparing two approaches: a) a reference system that assists the user to provide adequate translations for terms in the query; and b) a proposed system that assists the user to formulate the query as a set of relevant phrases, and to select promising phrases in the documents to enhance the query. All collected evidence indicates that the phrasebased approach is preferable: the official F α=0.8 measure is 65% better for the proposed system, and all users in our experiment preferred the phrase-based system as a simpler and faster way of searching.
The identification of the key concepts in a set of documents is a useful source of information fo... more The identification of the key concepts in a set of documents is a useful source of information for several information access applications. We are interested in its application to multi-document summarization, both for the automatic generation of summaries and for interactive summarization systems.
In this paper, we describe UNED’s participation in the iCLEF 2005 track. We have compared two str... more In this paper, we describe UNED’s participation in the iCLEF 2005 track. We have compared two strategies for finding an answer using an interactive question answering system: i) a search system over full documents and ii) a search system over passages (document’s paragraphs). We have added an interesting feature to both system in order to facilitate reading: the possibility to enable/disable the highlighting of named entities such as proper names, temporal references and numbers likely to contain the right answer. Our Document Searcher obtained better overall accuracy (.53 vs..45) but our subjects found browsing passages simpler and faster. However, most of them presented a similar search behavior (regarding time consumption, confidence in their answers and query refinements) using both systems. All our users considered helpful the highlighting of named entities and they all made extensive use of this possibility as a quick way of discriminating between relevant and non relevant doc...
We have participated in the SENSEVAL-2 English tasks (all words and lexical sample) with an unsup... more We have participated in the SENSEVAL-2 English tasks (all words and lexical sample) with an unsupervised system based on mutual information measured over a large corpus (277 million words) and some additional heuristics. A supervised extension of the system was also presented to the lexical sample task. Our system scored first among unsupervised systems in both tasks: 56.9 % recall in all words, 40.2 % in lexical sample. This is slightly worse than the first sense heuristic for all. words and 3.6 % better for the lexical sample, a strong indication that unsupervised Word Sense Disambiguation remains being a strong challenge.
This paper presents a probabilistic framework, QARLA, for the evaluation of text summarisation sy... more This paper presents a probabilistic framework, QARLA, for the evaluation of text summarisation systems. The input of the framework is a set of manual (reference) summaries, a set of baseline (automatic) summaries and a set of similarity metrics between summaries. It provides i) a measure to evaluate the quality of any set of similarity metrics, ii) a measure to evaluate the quality of a summary using an optimal set of similarity metrics, and iii) a measure to evaluate whether the set of baseline summaries is reliable or may produce biased results. Compared to previous approaches, our framework is able to combine different metrics and evaluate the quality of a set of metrics without any a-priori weighting of their relative importance. We provide quantitative evidence about the effectiveness of the approach to improve the automatic evaluation of text summarisation systems by combining several similarity metrics. 1
This paper presents the ITEM multilingual search engine. This search engine performs full lexical... more This paper presents the ITEM multilingual search engine. This search engine performs full lexical processing (morphological analysis, tagging and Word Sense Disambiguation) on documents and queries in order to provide language-neutral indexes for querying and retrieval. The indexing terms are the EuroWordNet/ITEM InterLingual Index records that link wordnets in 10 languages of the European Community (the search engine currently supports Spanish, English and Catalan). The goal of this application is to provide a way of comparing in context the behavior of different Natural Language Processing strategies for Cross-Language Information Retrieval (CLIR) and, in particular, different Word Sense Disambiguation strategies for query translation and conceptual indexing. 1.
This paper describes the exemplar based approach presented by UNED at Senseval-3. Instead of repr... more This paper describes the exemplar based approach presented by UNED at Senseval-3. Instead of representing contexts as bags of terms and defining a similarity measure between contexts, we propose to represent terms as bags of contexts and define a similarity measure between terms. Thus, words, lemmas and senses are represented in the same space (the context space), and similarity measures can be defined between them. New contexts are transformed into this representation in order to calculate their similarity to the candidate senses. We show how standard similarity measures obtain better results in this framework. A new similarity measure in the context space is proposed for selecting the senses and performing disambiguation. Results of this approach at Senseval-3 are here reported.
Although computing similarity is one of the fundamental challenges of Information Access tasks, t... more Although computing similarity is one of the fundamental challenges of Information Access tasks, the notion of similarity is not yet completely understood from a formal, axiomatic perspective. In this paper we show how axiomatic explanations of similarity from other fields (i.e. Tversky’s axioms from the point of view of cognitive sciences, and metric spaces from the point of view of algebra) do not completely fit the problem of similarity in Information Access, and we propose a new set of axioms which can be synthesized into a single Similarity Information Monotonicity axiom (SIM). Directly grounded on SIM, we then introduce a new similarity model, the Information Contrast Model, which generalizes both Tversky’s linear contrast model and Pointwise Mutual Information, and, unlike previous similarity models, satisfies the SIM axiom for a certain range of values of its parameters.
This paper reports the description of the developed system and the results obtained in the partic... more This paper reports the description of the developed system and the results obtained in the participation of the UNED in the Second Recognizing Textual Entailment (RTE) Challenge. New techniques and tools have been added: enriched queries to WordNet, detection of numeric expressions and their entailment, and Support Vector Machine classi cation (SVM) are the more relevant. The accuracy performed is slightly higher than the one from the previous edition system.
Resumen La distribuci??n de contenidos y/o metadatos en Repositorios de Objetos de Aprendizaje es... more Resumen La distribuci??n de contenidos y/o metadatos en Repositorios de Objetos de Aprendizaje es un aspecto clave a la hora de favorecer el intercambio de conocimiento dentro de comunidades de aprendizaje. En este art??culo se presenta una soluci??n de ...
This paper reports the participation of the Spanish Distance Learning University (UNED) in the Fi... more This paper reports the participation of the Spanish Distance Learning University (UNED) in the First Answer Validation Exercise (AVE) celebrated within the Cross Language Evaluation Forum (CLEF) 2006 edition. The system works for the Spanish language. It is based on a Support Vector Machine (SVM) classification of the pairs <text, hypothesis> given by the organization. This classification is accomplished by means of a set of features obtained from lexical analysis. Yet Another Learning Environment (Yale 3.0) was used for the SVM classification. Freeling was the toolkit elected for lemmatization and named entities recognition. Two runs were submitted and the results obtained, as defined by the organizers, were the following: precision run1 = 0.467, recall run1 = 0.7168, F run1 = 0.5655, precision run2 = 0.4652, recall run2 = 0.7079, F run2 = 0.5615.
In this article the incorporation of interoperability into a distributed eLearning system, namely... more In this article the incorporation of interoperability into a distributed eLearning system, namely the ActiveDocument platform, is considered. Three degrees of interoperability are defined for such a platform, namely syntactic, semantic, and functional. Subsequently, for the first two degrees of interoperability, a proposal is made for the way in which it can be obtained, backed up with details about the design and implementation of the ActiveDocument platform and its tools, which show how it has been (or is being) ...
We describe an algorithm that combines lexical information (from WordNet 1.7) with Web directorie... more We describe an algorithm that combines lexical information (from WordNet 1.7) with Web directories (from the Open Directory Project) to associate word senses with such directories. Such associations can be used as rich characterizations to acquire sense-tagged corpora automatically, cluster topically-related senses and detect sense specializations. The algorithm is evaluated for the 29 nouns (147 senses) used in the Senseval 2 competition, obtaining 148 (word sense,Web directory) associations covering 88 % of the domain-specific word senses in the test data with 86 % accuracy. The richness of Web directories as sense characterizations is evaluated in a supervised Word Sense Disambiguation task using the Senseval 2 test suite. The results indicate that, when the directory/word sense association is correct, the samples automatically acquired from the Web directories are nearly as valid for training as the original Senseval 2 training instances. The results support our hypothesis that ...
Journal of Artificial Intelligence Research, Sep 1, 2011
ABSTRACT Many Artificial Intelligence tasks cannot be evaluated with a single quality criterion a... more ABSTRACT Many Artificial Intelligence tasks cannot be evaluated with a single quality criterion and some sort of weighted combination is needed to provide system rankings. A problem of weighted combination measures is that slight changes in the relative weights may produce substantial changes in the system rankings. This paper introduces the Unanimous Improvement Ratio (UIR), a measure that complements standard metric combination criteria (such as van Rijsbergen&#39;s F-measure) and indicates how robust the measured differences are to changes in the relative weights of the individual metrics. UIR is meant to elucidate whether a perceived difference between two systems is an artifact of how individual metrics are weighted. Besides discussing the theoretical foundations of UIR, this paper presents empirical results that confirm the validity and usefulness of the metric for the Text Clustering problem, where there is a tradeoff between precision and recall based metrics and results are particularly sensitive to the weighting scheme used to combine them. Remarkably, our experiments show that UIR can be used as a predictor of how well differences between systems measured on a given test bed will also hold in a different test bed.
Este artículo muestra la aplicación del marco de evaluación QARLA sobre los resúmenes evaluados e... more Este artículo muestra la aplicación del marco de evaluación QARLA sobre los resúmenes evaluados en el foro DUC-2004, para las tareas 2 y 5. El marco QARLA permite evaluar de forma automática los sistemas según diferentes aspectos (métricas de similitud) en relación a un conjunto de resúmenes modelo, identificando así los aspectos más deficitarios de las estrategias de resumen existentes. Por otro lado, el marco QARLA permite combinar y meta-evaluar diferentes métricas de similitud, otorgando más peso a los aspectos que caracterizan a los modelos en relación a los resúmenes automáticos.
This paper summarizes the participation of the UNED group in the CLEF 2002 Interactive Track. We ... more This paper summarizes the participation of the UNED group in the CLEF 2002 Interactive Track. We focused on interactive query formulation and refinement, comparing two approaches: a) a reference system that assists the user to provide adequate translations for terms in the query; and b) a proposed system that assists the user to formulate the query as a set of relevant phrases, and to select promising phrases in the documents to enhance the query. All collected evidence indicates that the phrasebased approach is preferable: the official F α=0.8 measure is 65% better for the proposed system, and all users in our experiment preferred the phrase-based system as a simpler and faster way of searching.
Uploads
Papers by Felisa Verdejo