Abstract: We now have a rich and growing set of modeling tools and algorithms for inducing lingui... more Abstract: We now have a rich and growing set of modeling tools and algorithms for inducing linguistic structure from text that is less than fully annotated. In this paper, we discuss some of the weaknesses of our current methodology. We present a new abstract framework for evaluating natural language processing (NLP) models in general and unsupervised NLP models in particular.
Abstract: We propose AD3, a new algorithm for approximate maximum a posteriori (MAP) inference on... more Abstract: We propose AD3, a new algorithm for approximate maximum a posteriori (MAP) inference on factor graphs based on the alternating directions method of multipliers. Like dual decomposition algorithms, AD3 uses worker nodes to iteratively solve local subproblems and a controller node to combine these local solutions into a global update.
Abstract Linear models have enjoyed great success in structured prediction in NLP. While a lot of... more Abstract Linear models have enjoyed great success in structured prediction in NLP. While a lot of progress has been made on efficient training with several loss functions, the problem of endowing learners with a mechanism for feature selection is still unsolved. Common approaches employ ad hoc filtering or L 1-regularization; both ignore the structure of the feature space, preventing practicioners from encoding structural prior knowledge.
Abstract Unsupervised multilingual learning has been shown to be effective for NLP (natural langu... more Abstract Unsupervised multilingual learning has been shown to be effective for NLP (natural language processing) tasks such as POS (part of speech) tag induction [Snyder and Barzilay (2010)] and grammar induction [Cohen and Smith (2009)][Berg-Kirkpatrick and Klein (2010)]. This thesis follows the work of Ben Snyder that improves unsupervised part of speech tags with the help of word alignments. Word alignments are useful for part of speech tag induction because there are very regular patterns for the tags of aligned words.
Executive Summary Forecasting the future hinges on understanding the present. The web—particularl... more Executive Summary Forecasting the future hinges on understanding the present. The web—particularly the social web—now gives us an up-to-the-minute snapshot of the world as it is and as it is perceived by many people, right now, but that snapshot is distributed in a way that is incomprehensible to a human.
Abstract With Twitter and Facebook blocked in China, the stream of information from Chinese domes... more Abstract With Twitter and Facebook blocked in China, the stream of information from Chinese domestic social media provides a case study of social media behavior under the influence of active censorship. While much work has looked at efforts to prevent access to information in China (including IP blocking of foreign Web sites or search engine filtering), we present here the first large–scale analysis of political content censorship in social media, ie, the active deletion of messages published by individuals.
Abstract We describe a competitive question generation and answering project used in our undergra... more Abstract We describe a competitive question generation and answering project used in our undergraduate natural language processing courses. This semester-long project challenges teams of three or four students to use available NLP components (or develop their own) to construct systems that ask and answer questions about an arbitrary Wikipedia article. We describe how the project and competition were structured, the outcomes, and lessons learned.
Abstract We present a statistical model for canonicalizing named entity mentions into a table who... more Abstract We present a statistical model for canonicalizing named entity mentions into a table whose rows represent entities and whose columns are attributes (or parts of attributes). The model is novel in that it incorporates entity context, surface features, first-order dependencies among attribute-parts, and a notion of noise. Transductive learning from a few seeds and a collection of mention tokens combines Bayesian inference and conditional estimation.
Abstract We address the problem of selecting a point from a unit simplex, uniformly at random. Th... more Abstract We address the problem of selecting a point from a unit simplex, uniformly at random. This problem is important, for instance, when random multinomial probability distributions are required. We show that a previously proposed algorithm is incorrect, and demonstrate a corrected algorithm.
Abstract We consider the problem of NER in Arabic Wikipedia, a semi-supervised domain adaptation ... more Abstract We consider the problem of NER in Arabic Wikipedia, a semi-supervised domain adaptation setting for which we have no labeled training data in the target domain. To facilitate evaluation, we obtain annotations for articles in four topical groups, allowing annotators to identify domain-specific entity types in addition to standard categories. Standard supervised learning on newswire text leads to poor target-domain recall.
Abstract A US Congressional bill is a textual artifact that must pass through a series of hurdles... more Abstract A US Congressional bill is a textual artifact that must pass through a series of hurdles to become a law. In this paper, we focus on one of the most precarious and least understood stages in a bill's life: its consideration, behind closed doors, by a Congressional committee. We construct predictive models of whether a bill will survive committee, starting with a strong, novel baseline that uses features of the bill's sponsor and the committee it is referred to.
Abstract We propose a Bayesian generative model of how demographic social factors influence lexic... more Abstract We propose a Bayesian generative model of how demographic social factors influence lexical choice. We apply the method to a corpus of geo-tagged Twitter messages originating from mobile phones, cross-referenced against US Census demographic data. Our method discovers communities jointly defined by linguistic and demographic properties.
Abstract We use Amazon Mechanical Turk to rate computer-generated reading comprehension questions... more Abstract We use Amazon Mechanical Turk to rate computer-generated reading comprehension questions about Wikipedia articles. Such application-specific ratings can be used to train statistical rankers to improve systems' final output, or to evaluate technologies that generate natural language. We discuss the question rating scheme we developed, assess the quality of the ratings that we gathered through Amazon Mechanical Turk, and show evidence that these ratings can be used to improve question generation.
Abstract Probabilistic grammars are generative statistical models that are useful for composition... more Abstract Probabilistic grammars are generative statistical models that are useful for compositional and sequential structures. We present a framework, reminiscent of structural risk minimization, for empirical risk minimization of the parameters of a fixed probabilistic grammar using the log-loss. We derive sample complexity bounds in this framework that apply both to the supervised setting and the unsupervised setting.
Abstract: We describe an extensible approach to generating questions for the purpose of reading c... more Abstract: We describe an extensible approach to generating questions for the purpose of reading comprehension assessment and practice. Our framework for question generation composes general-purpose rules to transform declarative sentences into questions, is modular in that existing NLP tools can be leveraged, and includes a statistical component for scoring questions based on features of the input, output, and transformations performed.
Abstract We address a text regression problem: given a piece of text, predict a real-world contin... more Abstract We address a text regression problem: given a piece of text, predict a real-world continuous quantity associated with the text's meaning. In this work, the text is an SEC-mandated financial report published annually by a publicly-traded company, and the quantity to be predicted is volatility of stock returns, an empirical measure of financial risk. We apply well-known regression techniques to a large corpus of freely available financial reports, constructing regression models of volatility for the period following a report.
Abstract We present the first version of a new declarative programming language. Dyna has many us... more Abstract We present the first version of a new declarative programming language. Dyna has many uses but was designed especially for rapid development of new statistical NLP systems. A Dyna program is a small set of equations, resembling Prolog inference rules, that specify the abstract structure of a dynamic programming algorithm. It compiles into efficient, portable, C++ classes that can be easily invoked from a larger application.
Abstract We describe a novel training criterion for probabilistic grammar induction models, contr... more Abstract We describe a novel training criterion for probabilistic grammar induction models, contrastive estimation [Smith and Eisner, 2005], which can be interpreted as exploiting implicit negative evidence and includes a wide class of likelihood-based objective functions. This criterion is a generalization of the function maximized by the Expectation-Maximization algorithm [Dempster et al., 1977]. CE is a natural fit for log-linear models, which can include arbitrary features but for which EM is computationally difficult.
Abstract We describe our entry in the CoNLL-X shared task. The system consists of three phases: a... more Abstract We describe our entry in the CoNLL-X shared task. The system consists of three phases: a probabilistic vine parser (Eisner and N. Smith, 2005) that produces unlabeled dependency trees, a probabilistic relation-labeling model, and a discriminative minimum risk reranker (D. Smith and Eisner, 2006). The system is designed for fast training and decoding and for high precision. We describe sources of cross-lingual error and ways to ameliorate them.
Abstract: We present a unified framework for online learning of structured classifiers that handl... more Abstract: We present a unified framework for online learning of structured classifiers that handles a wide family of convex loss functions, properly including CRFs, structured SVMs, and the structured perceptron. We introduce a new aggressive online algorithm that optimizes any loss in this family. For the structured hinge loss, this algorithm reduces to 1-best MIRA; in general, it can be regarded as a dual coordinate ascent algorithm. The approximate inference scenario is also addressed.
Abstract: We now have a rich and growing set of modeling tools and algorithms for inducing lingui... more Abstract: We now have a rich and growing set of modeling tools and algorithms for inducing linguistic structure from text that is less than fully annotated. In this paper, we discuss some of the weaknesses of our current methodology. We present a new abstract framework for evaluating natural language processing (NLP) models in general and unsupervised NLP models in particular.
Abstract: We propose AD3, a new algorithm for approximate maximum a posteriori (MAP) inference on... more Abstract: We propose AD3, a new algorithm for approximate maximum a posteriori (MAP) inference on factor graphs based on the alternating directions method of multipliers. Like dual decomposition algorithms, AD3 uses worker nodes to iteratively solve local subproblems and a controller node to combine these local solutions into a global update.
Abstract Linear models have enjoyed great success in structured prediction in NLP. While a lot of... more Abstract Linear models have enjoyed great success in structured prediction in NLP. While a lot of progress has been made on efficient training with several loss functions, the problem of endowing learners with a mechanism for feature selection is still unsolved. Common approaches employ ad hoc filtering or L 1-regularization; both ignore the structure of the feature space, preventing practicioners from encoding structural prior knowledge.
Abstract Unsupervised multilingual learning has been shown to be effective for NLP (natural langu... more Abstract Unsupervised multilingual learning has been shown to be effective for NLP (natural language processing) tasks such as POS (part of speech) tag induction [Snyder and Barzilay (2010)] and grammar induction [Cohen and Smith (2009)][Berg-Kirkpatrick and Klein (2010)]. This thesis follows the work of Ben Snyder that improves unsupervised part of speech tags with the help of word alignments. Word alignments are useful for part of speech tag induction because there are very regular patterns for the tags of aligned words.
Executive Summary Forecasting the future hinges on understanding the present. The web—particularl... more Executive Summary Forecasting the future hinges on understanding the present. The web—particularly the social web—now gives us an up-to-the-minute snapshot of the world as it is and as it is perceived by many people, right now, but that snapshot is distributed in a way that is incomprehensible to a human.
Abstract With Twitter and Facebook blocked in China, the stream of information from Chinese domes... more Abstract With Twitter and Facebook blocked in China, the stream of information from Chinese domestic social media provides a case study of social media behavior under the influence of active censorship. While much work has looked at efforts to prevent access to information in China (including IP blocking of foreign Web sites or search engine filtering), we present here the first large–scale analysis of political content censorship in social media, ie, the active deletion of messages published by individuals.
Abstract We describe a competitive question generation and answering project used in our undergra... more Abstract We describe a competitive question generation and answering project used in our undergraduate natural language processing courses. This semester-long project challenges teams of three or four students to use available NLP components (or develop their own) to construct systems that ask and answer questions about an arbitrary Wikipedia article. We describe how the project and competition were structured, the outcomes, and lessons learned.
Abstract We present a statistical model for canonicalizing named entity mentions into a table who... more Abstract We present a statistical model for canonicalizing named entity mentions into a table whose rows represent entities and whose columns are attributes (or parts of attributes). The model is novel in that it incorporates entity context, surface features, first-order dependencies among attribute-parts, and a notion of noise. Transductive learning from a few seeds and a collection of mention tokens combines Bayesian inference and conditional estimation.
Abstract We address the problem of selecting a point from a unit simplex, uniformly at random. Th... more Abstract We address the problem of selecting a point from a unit simplex, uniformly at random. This problem is important, for instance, when random multinomial probability distributions are required. We show that a previously proposed algorithm is incorrect, and demonstrate a corrected algorithm.
Abstract We consider the problem of NER in Arabic Wikipedia, a semi-supervised domain adaptation ... more Abstract We consider the problem of NER in Arabic Wikipedia, a semi-supervised domain adaptation setting for which we have no labeled training data in the target domain. To facilitate evaluation, we obtain annotations for articles in four topical groups, allowing annotators to identify domain-specific entity types in addition to standard categories. Standard supervised learning on newswire text leads to poor target-domain recall.
Abstract A US Congressional bill is a textual artifact that must pass through a series of hurdles... more Abstract A US Congressional bill is a textual artifact that must pass through a series of hurdles to become a law. In this paper, we focus on one of the most precarious and least understood stages in a bill's life: its consideration, behind closed doors, by a Congressional committee. We construct predictive models of whether a bill will survive committee, starting with a strong, novel baseline that uses features of the bill's sponsor and the committee it is referred to.
Abstract We propose a Bayesian generative model of how demographic social factors influence lexic... more Abstract We propose a Bayesian generative model of how demographic social factors influence lexical choice. We apply the method to a corpus of geo-tagged Twitter messages originating from mobile phones, cross-referenced against US Census demographic data. Our method discovers communities jointly defined by linguistic and demographic properties.
Abstract We use Amazon Mechanical Turk to rate computer-generated reading comprehension questions... more Abstract We use Amazon Mechanical Turk to rate computer-generated reading comprehension questions about Wikipedia articles. Such application-specific ratings can be used to train statistical rankers to improve systems' final output, or to evaluate technologies that generate natural language. We discuss the question rating scheme we developed, assess the quality of the ratings that we gathered through Amazon Mechanical Turk, and show evidence that these ratings can be used to improve question generation.
Abstract Probabilistic grammars are generative statistical models that are useful for composition... more Abstract Probabilistic grammars are generative statistical models that are useful for compositional and sequential structures. We present a framework, reminiscent of structural risk minimization, for empirical risk minimization of the parameters of a fixed probabilistic grammar using the log-loss. We derive sample complexity bounds in this framework that apply both to the supervised setting and the unsupervised setting.
Abstract: We describe an extensible approach to generating questions for the purpose of reading c... more Abstract: We describe an extensible approach to generating questions for the purpose of reading comprehension assessment and practice. Our framework for question generation composes general-purpose rules to transform declarative sentences into questions, is modular in that existing NLP tools can be leveraged, and includes a statistical component for scoring questions based on features of the input, output, and transformations performed.
Abstract We address a text regression problem: given a piece of text, predict a real-world contin... more Abstract We address a text regression problem: given a piece of text, predict a real-world continuous quantity associated with the text's meaning. In this work, the text is an SEC-mandated financial report published annually by a publicly-traded company, and the quantity to be predicted is volatility of stock returns, an empirical measure of financial risk. We apply well-known regression techniques to a large corpus of freely available financial reports, constructing regression models of volatility for the period following a report.
Abstract We present the first version of a new declarative programming language. Dyna has many us... more Abstract We present the first version of a new declarative programming language. Dyna has many uses but was designed especially for rapid development of new statistical NLP systems. A Dyna program is a small set of equations, resembling Prolog inference rules, that specify the abstract structure of a dynamic programming algorithm. It compiles into efficient, portable, C++ classes that can be easily invoked from a larger application.
Abstract We describe a novel training criterion for probabilistic grammar induction models, contr... more Abstract We describe a novel training criterion for probabilistic grammar induction models, contrastive estimation [Smith and Eisner, 2005], which can be interpreted as exploiting implicit negative evidence and includes a wide class of likelihood-based objective functions. This criterion is a generalization of the function maximized by the Expectation-Maximization algorithm [Dempster et al., 1977]. CE is a natural fit for log-linear models, which can include arbitrary features but for which EM is computationally difficult.
Abstract We describe our entry in the CoNLL-X shared task. The system consists of three phases: a... more Abstract We describe our entry in the CoNLL-X shared task. The system consists of three phases: a probabilistic vine parser (Eisner and N. Smith, 2005) that produces unlabeled dependency trees, a probabilistic relation-labeling model, and a discriminative minimum risk reranker (D. Smith and Eisner, 2006). The system is designed for fast training and decoding and for high precision. We describe sources of cross-lingual error and ways to ameliorate them.
Abstract: We present a unified framework for online learning of structured classifiers that handl... more Abstract: We present a unified framework for online learning of structured classifiers that handles a wide family of convex loss functions, properly including CRFs, structured SVMs, and the structured perceptron. We introduce a new aggressive online algorithm that optimizes any loss in this family. For the structured hinge loss, this algorithm reduces to 1-best MIRA; in general, it can be regarded as a dual coordinate ascent algorithm. The approximate inference scenario is also addressed.
Uploads
Papers by Noah A. Smith