Proceedings of the 20th international conference on Computational Linguistics - COLING '04, 2004
This paper shows how to construct semantic representations from the derivations produced by a wid... more This paper shows how to construct semantic representations from the derivations produced by a wide-coverage CCG parser. Unlike the dependency structures returned by the parser itself, these can be used directly for semantic interpretation. We demonstrate that well-formed semantic representations can be produced for over 97% of the sentences in unseen WSJ text. We believe this is a major step towards widecoverage semantic interpretation, one of the key objectives of the field of NLP.
Verb Phrase Ellipsis (VPE) has been studied in great depth in theoretical linguistics, but empiri... more Verb Phrase Ellipsis (VPE) has been studied in great depth in theoretical linguistics, but empirical studies of VPE are rare. We extend the few previous corpus studies with an annotated corpus of VPE in all 25 sections of the Wall Street Journal corpus (WSJ) distributed with the Penn Treebank. We annotated the raw files using a stand-off annotation scheme that codes the auxiliary verb triggering the elided verb phrase, the start and end of the antecedent, the syntactic type of antecedent (VP, TV, NP, PP or AP), and the type of syntactic pattern between the source and target clauses of the VPE and its antecedent. We found 487 instances of VPE (including predicative ellipsis, antecedent-contained deletion, comparative constructions, and pseudo-gapping) plus 67 cases of related phenomena such as do so anaphora. Inter-annotator agreement was high, with a 0.97 average F-score for three annotators for one section of the WSJ. Our annotation is theory neutral, and has better coverage than earlier efforts that relied on automatic methods, e.g. simply searching the parsed version of the Penn Treebank for empty VP's achieves a high precision (0.95) but low recall (0.58) when compared with our manual annotation. The distribution of VPE source-target patterns deviates highly from the standard examples found in the theoretical linguistics literature on VPE, once more underlining the value of corpus studies. The resulting corpus will be useful for studying VPE phenomena as well as for evaluating natural language processing systems equipped with ellipsis resolution algorithms, and we propose evaluation measures for VPE detection and VPE antecedent selection. The stand-off annotation is freely available for research purposes.
In this paper we present a pragmatic account of scope alternation involving universal quantifiers... more In this paper we present a pragmatic account of scope alternation involving universal quantifiers in a lexicalist framework based on CCG and DRT. This account can derive the desired reading for 96% of all cases of scope interaction involving universal quantification mediated by prepositions in a real corpus. We show how this account allows for recasting scope resolution as a simple token classification task, providing a simpler handle for statistical approaches to scope resolution than previous accounts.
What would be a good method to provide a large collection of semantically annotated texts with fo... more What would be a good method to provide a large collection of semantically annotated texts with formal, deep semantics rather than shallow? We argue that a bootstrapping approach comprising state-of-the-art NLP tools for parsing and semantic interpretation, in combination with a wiki-like interface for collaborative annotation of experts, and a game with a purpose for crowdsourcing, are the starting ingredients for fulfilling this enterprise. The result is a semantic resource that anyone can edit and that integrates various phenomena, including predicate-argument structure, scope, tense, thematic roles, rhetorical relations and presuppositions, into a single semantic formalism: Discourse Representation Theory. Taking texts rather than sentences as the units of annotation results in deep semantic representations that incorporate discourse structure and dependencies. To manage the various (possibly conflicting) annotations provided by experts and non-experts, we introduce a method that stores "Bits of Wisdom" in a database as stand-off annotations.
In this paper the use of model building for natural language understanding in practical systems i... more In this paper the use of model building for natural language understanding in practical systems is investigated. After outlining several interpretation tasks where model construction would be useful, we perform experiments using two state-of-the-art model builders on the interpretation of imperatives in discourse. The results are acceptable for small discourses and small domains, but don't scale up for larger domains or longer discouses: the complexity of model generation rises sharply in relation to the difficulty of a task and the size of the dialogue. Several suggestions are provided for future research to improve the performance of model building for larger domains and richer texts.
This paper describes a technique to include acoustic confidence scores as returned by automated s... more This paper describes a technique to include acoustic confidence scores as returned by automated speech recognisers in generic semantic representations. The method we propose requires only minimal changes to an existing grammar used for speech applications. Special attention is paid to the treatment of multi-word lexemes and combining several (N-best) speech recognition results into one semantic representation. The approach has been implemented and tested using the Nuance speech recognition software and a chart parser, in the formalism of underspecified discourse representations. The potential relevance of confidence scores in rich semantic representations is illustrated by generating more flexible clarification questions in dialogue systems.
We approach cross-lingual question answering by using a mono-lingual QA system for the source lan... more We approach cross-lingual question answering by using a mono-lingual QA system for the source language and by translating resulting answers into the target language. As far as we are aware, this is the first cross-lingual QA system in the history of CLEF that uses this method-all other cross-lingual QA systems known to us use translation of the question or query instead. We demonstrate the feasibility of this approach by using a mono-lingual QA system for English, and translating answers and finding appropriate documents in Italian and Dutch. For factoid and definition questions, we achieve overall accuracy scores ranging from 13% (EN→NL) to 17% (EN→IT) and lenient accuracy figures from 19% (EN→NL) to 25% (EN→IT). The advantage of this strategy to cross-lingual QA is that translation of answers is easier than translating questions-the disadvantage is that answers might be missing from the source corpus and additional effort is required for finding supporting documents of the target language.
Reading comprehension tests are receiving increased attention within the NLP community as a contr... more Reading comprehension tests are receiving increased attention within the NLP community as a controlled test-bed for developing, evaluating and comparing robust question answering (NLQA) methods. To support this, we have enriched the MITRE CBC4Kids corpus with multiple XML annotation layers recording the output of various tokenizers, lemmatizers, a stemmer, a semantic tagger, POS taggers and syntactic parsers. To demonstrate its use, we have built a baseline NLQA system for word-overlap based answer retrieval, NLQA evaluation and corpus browsing.
This report describes a new open-domain answer retrieval system developed at the University of Ed... more This report describes a new open-domain answer retrieval system developed at the University of Edinburgh and gives results for the TREC-12 question answering track. Phrasal answers are identified by increasingly narrowing down the search space from a large text collection to a single phrase. The system uses document retrieval, query-based passage segmentation and ranking, semantic analysis from a wide-coverage parser, and a unification-like matching procedure to extract potential answers. A simple Web-based answer validation stage is also applied. The system is based on the Open Agent Architecture and has a parallel design so that multiple questions can be answered simultaneously on a Beowulf cluster.
This report describes the experiments of the University of Edinburgh and the University of Sydney... more This report describes the experiments of the University of Edinburgh and the University of Sydney at the TREC-2004 question answering evaluation exercise. Our system combines two approaches: one with deep linguistic analysis using IR on the AQUAINT corpus applied to answer extraction from text passages, and one with a shallow linguistic analysis and shallow inference applied to a large set of snippets retrieved from the web. The results of our experiments support the following claims: (1) Webbased IR is a good alternative to "traditional" IR; and (2) deep linguistic analysis improves quality of exact answers.
One can only exploit inference in Question-Answering (QA) and assess its contribution systematica... more One can only exploit inference in Question-Answering (QA) and assess its contribution systematically, if one knows what inference is contributing to. Thus we identify a set of tasks specific to QA and discuss what inference could contribute to their achievement. We conclude with a proposal for graduated test suites as a tool for assessing the performance and impact of inference.
1. To give some simple examples of Prolog programs. This will introduce us to the three basic con... more 1. To give some simple examples of Prolog programs. This will introduce us to the three basic constructs in Prolog: facts, rules, and queries. It will also introduce us to a number of other themes, like the role of logic in Prolog, and the idea of performing matching with the aid of variables.
Multilingual Information Access for Text, Speech and Images, 2005
We show how to adapt an existing monolingual open-domain QA system to perform in a cross-lingual ... more We show how to adapt an existing monolingual open-domain QA system to perform in a cross-lingual environment, using off-the-shelf machine translation software. In our experiments we use French and German as source language, and English as target language. For answering factoid questions, our system performs with an accuracy of 16% (German to English) and 20% (French to English), respectively. The loss of correctly answered questions caused by the MT component is estimated at 10% for French, and 15% for German. The accuracy of our system on correctly translated questions is 28% for German and 29% for French.
Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing - EMNLP '06, 2006
In this paper we introduce an empirical approach to the semantic interpretation of superlative ad... more In this paper we introduce an empirical approach to the semantic interpretation of superlative adjectives. We present a corpus annotated for superlatives and propose an interpretation algorithm that uses a wide-coverage parser and produces semantic representations. We achieve Fscores between 0.84 and 0.91 for detecting attributive superlatives and an accuracy in the range of 0.69-0.84 for determining the correct comparison set. As far as we are aware, this is the first automated approach to superlatives for open-domain texts and questions.
From a purely theoretical point of view, it makes sense to approach recognizing textual entailmen... more From a purely theoretical point of view, it makes sense to approach recognizing textual entailment (RTE) with the help of logic. After all, entailment matters are all about logic. In practice, only few RTE systems follow the bumpy road from words to logic. This is probably because it requires a combination of robust, deep semantic analysis and logical inference-and why develop something with this complexity if you perhaps can get away with something simpler? In this article, with the help of an RTE system based on Combinatory Categorial Grammar, Discourse Representation Theory, and first-order theorem proving, we make an empirical assessment of the logic-based approach. High precision paired with low recall is a key characteristic of this system. The bottleneck in achieving high recall is the lack of a systematic way to produce relevant background knowledge. There is a place for logic in RTE, but it is (still) overshadowed by the knowledge acquisition problem.
A large spoken dialogue translation system imposes both engineering and linguistic constraints on... more A large spoken dialogue translation system imposes both engineering and linguistic constraints on the way in which linguistic information is communicated between modules. We describe the design and use of interface terms, whose formal, functional and communicative role has been tested in a sequence of integrated systems and which have proven adequate to these constraints.
Proceedings of the 20th international conference on Computational Linguistics - COLING '04, 2004
This paper shows how to construct semantic representations from the derivations produced by a wid... more This paper shows how to construct semantic representations from the derivations produced by a wide-coverage CCG parser. Unlike the dependency structures returned by the parser itself, these can be used directly for semantic interpretation. We demonstrate that well-formed semantic representations can be produced for over 97% of the sentences in unseen WSJ text. We believe this is a major step towards widecoverage semantic interpretation, one of the key objectives of the field of NLP.
Verb Phrase Ellipsis (VPE) has been studied in great depth in theoretical linguistics, but empiri... more Verb Phrase Ellipsis (VPE) has been studied in great depth in theoretical linguistics, but empirical studies of VPE are rare. We extend the few previous corpus studies with an annotated corpus of VPE in all 25 sections of the Wall Street Journal corpus (WSJ) distributed with the Penn Treebank. We annotated the raw files using a stand-off annotation scheme that codes the auxiliary verb triggering the elided verb phrase, the start and end of the antecedent, the syntactic type of antecedent (VP, TV, NP, PP or AP), and the type of syntactic pattern between the source and target clauses of the VPE and its antecedent. We found 487 instances of VPE (including predicative ellipsis, antecedent-contained deletion, comparative constructions, and pseudo-gapping) plus 67 cases of related phenomena such as do so anaphora. Inter-annotator agreement was high, with a 0.97 average F-score for three annotators for one section of the WSJ. Our annotation is theory neutral, and has better coverage than earlier efforts that relied on automatic methods, e.g. simply searching the parsed version of the Penn Treebank for empty VP's achieves a high precision (0.95) but low recall (0.58) when compared with our manual annotation. The distribution of VPE source-target patterns deviates highly from the standard examples found in the theoretical linguistics literature on VPE, once more underlining the value of corpus studies. The resulting corpus will be useful for studying VPE phenomena as well as for evaluating natural language processing systems equipped with ellipsis resolution algorithms, and we propose evaluation measures for VPE detection and VPE antecedent selection. The stand-off annotation is freely available for research purposes.
In this paper we present a pragmatic account of scope alternation involving universal quantifiers... more In this paper we present a pragmatic account of scope alternation involving universal quantifiers in a lexicalist framework based on CCG and DRT. This account can derive the desired reading for 96% of all cases of scope interaction involving universal quantification mediated by prepositions in a real corpus. We show how this account allows for recasting scope resolution as a simple token classification task, providing a simpler handle for statistical approaches to scope resolution than previous accounts.
What would be a good method to provide a large collection of semantically annotated texts with fo... more What would be a good method to provide a large collection of semantically annotated texts with formal, deep semantics rather than shallow? We argue that a bootstrapping approach comprising state-of-the-art NLP tools for parsing and semantic interpretation, in combination with a wiki-like interface for collaborative annotation of experts, and a game with a purpose for crowdsourcing, are the starting ingredients for fulfilling this enterprise. The result is a semantic resource that anyone can edit and that integrates various phenomena, including predicate-argument structure, scope, tense, thematic roles, rhetorical relations and presuppositions, into a single semantic formalism: Discourse Representation Theory. Taking texts rather than sentences as the units of annotation results in deep semantic representations that incorporate discourse structure and dependencies. To manage the various (possibly conflicting) annotations provided by experts and non-experts, we introduce a method that stores "Bits of Wisdom" in a database as stand-off annotations.
In this paper the use of model building for natural language understanding in practical systems i... more In this paper the use of model building for natural language understanding in practical systems is investigated. After outlining several interpretation tasks where model construction would be useful, we perform experiments using two state-of-the-art model builders on the interpretation of imperatives in discourse. The results are acceptable for small discourses and small domains, but don't scale up for larger domains or longer discouses: the complexity of model generation rises sharply in relation to the difficulty of a task and the size of the dialogue. Several suggestions are provided for future research to improve the performance of model building for larger domains and richer texts.
This paper describes a technique to include acoustic confidence scores as returned by automated s... more This paper describes a technique to include acoustic confidence scores as returned by automated speech recognisers in generic semantic representations. The method we propose requires only minimal changes to an existing grammar used for speech applications. Special attention is paid to the treatment of multi-word lexemes and combining several (N-best) speech recognition results into one semantic representation. The approach has been implemented and tested using the Nuance speech recognition software and a chart parser, in the formalism of underspecified discourse representations. The potential relevance of confidence scores in rich semantic representations is illustrated by generating more flexible clarification questions in dialogue systems.
We approach cross-lingual question answering by using a mono-lingual QA system for the source lan... more We approach cross-lingual question answering by using a mono-lingual QA system for the source language and by translating resulting answers into the target language. As far as we are aware, this is the first cross-lingual QA system in the history of CLEF that uses this method-all other cross-lingual QA systems known to us use translation of the question or query instead. We demonstrate the feasibility of this approach by using a mono-lingual QA system for English, and translating answers and finding appropriate documents in Italian and Dutch. For factoid and definition questions, we achieve overall accuracy scores ranging from 13% (EN→NL) to 17% (EN→IT) and lenient accuracy figures from 19% (EN→NL) to 25% (EN→IT). The advantage of this strategy to cross-lingual QA is that translation of answers is easier than translating questions-the disadvantage is that answers might be missing from the source corpus and additional effort is required for finding supporting documents of the target language.
Reading comprehension tests are receiving increased attention within the NLP community as a contr... more Reading comprehension tests are receiving increased attention within the NLP community as a controlled test-bed for developing, evaluating and comparing robust question answering (NLQA) methods. To support this, we have enriched the MITRE CBC4Kids corpus with multiple XML annotation layers recording the output of various tokenizers, lemmatizers, a stemmer, a semantic tagger, POS taggers and syntactic parsers. To demonstrate its use, we have built a baseline NLQA system for word-overlap based answer retrieval, NLQA evaluation and corpus browsing.
This report describes a new open-domain answer retrieval system developed at the University of Ed... more This report describes a new open-domain answer retrieval system developed at the University of Edinburgh and gives results for the TREC-12 question answering track. Phrasal answers are identified by increasingly narrowing down the search space from a large text collection to a single phrase. The system uses document retrieval, query-based passage segmentation and ranking, semantic analysis from a wide-coverage parser, and a unification-like matching procedure to extract potential answers. A simple Web-based answer validation stage is also applied. The system is based on the Open Agent Architecture and has a parallel design so that multiple questions can be answered simultaneously on a Beowulf cluster.
This report describes the experiments of the University of Edinburgh and the University of Sydney... more This report describes the experiments of the University of Edinburgh and the University of Sydney at the TREC-2004 question answering evaluation exercise. Our system combines two approaches: one with deep linguistic analysis using IR on the AQUAINT corpus applied to answer extraction from text passages, and one with a shallow linguistic analysis and shallow inference applied to a large set of snippets retrieved from the web. The results of our experiments support the following claims: (1) Webbased IR is a good alternative to "traditional" IR; and (2) deep linguistic analysis improves quality of exact answers.
One can only exploit inference in Question-Answering (QA) and assess its contribution systematica... more One can only exploit inference in Question-Answering (QA) and assess its contribution systematically, if one knows what inference is contributing to. Thus we identify a set of tasks specific to QA and discuss what inference could contribute to their achievement. We conclude with a proposal for graduated test suites as a tool for assessing the performance and impact of inference.
1. To give some simple examples of Prolog programs. This will introduce us to the three basic con... more 1. To give some simple examples of Prolog programs. This will introduce us to the three basic constructs in Prolog: facts, rules, and queries. It will also introduce us to a number of other themes, like the role of logic in Prolog, and the idea of performing matching with the aid of variables.
Multilingual Information Access for Text, Speech and Images, 2005
We show how to adapt an existing monolingual open-domain QA system to perform in a cross-lingual ... more We show how to adapt an existing monolingual open-domain QA system to perform in a cross-lingual environment, using off-the-shelf machine translation software. In our experiments we use French and German as source language, and English as target language. For answering factoid questions, our system performs with an accuracy of 16% (German to English) and 20% (French to English), respectively. The loss of correctly answered questions caused by the MT component is estimated at 10% for French, and 15% for German. The accuracy of our system on correctly translated questions is 28% for German and 29% for French.
Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing - EMNLP '06, 2006
In this paper we introduce an empirical approach to the semantic interpretation of superlative ad... more In this paper we introduce an empirical approach to the semantic interpretation of superlative adjectives. We present a corpus annotated for superlatives and propose an interpretation algorithm that uses a wide-coverage parser and produces semantic representations. We achieve Fscores between 0.84 and 0.91 for detecting attributive superlatives and an accuracy in the range of 0.69-0.84 for determining the correct comparison set. As far as we are aware, this is the first automated approach to superlatives for open-domain texts and questions.
From a purely theoretical point of view, it makes sense to approach recognizing textual entailmen... more From a purely theoretical point of view, it makes sense to approach recognizing textual entailment (RTE) with the help of logic. After all, entailment matters are all about logic. In practice, only few RTE systems follow the bumpy road from words to logic. This is probably because it requires a combination of robust, deep semantic analysis and logical inference-and why develop something with this complexity if you perhaps can get away with something simpler? In this article, with the help of an RTE system based on Combinatory Categorial Grammar, Discourse Representation Theory, and first-order theorem proving, we make an empirical assessment of the logic-based approach. High precision paired with low recall is a key characteristic of this system. The bottleneck in achieving high recall is the lack of a systematic way to produce relevant background knowledge. There is a place for logic in RTE, but it is (still) overshadowed by the knowledge acquisition problem.
A large spoken dialogue translation system imposes both engineering and linguistic constraints on... more A large spoken dialogue translation system imposes both engineering and linguistic constraints on the way in which linguistic information is communicated between modules. We describe the design and use of interface terms, whose formal, functional and communicative role has been tested in a sequence of integrated systems and which have proven adequate to these constraints.
Uploads
Papers by J. Bos