Effective representation of Web search results remains an open problem in the Information Retriev... more Effective representation of Web search results remains an open problem in the Information Retrieval community. For ambiguous queries, a traditional approach is to organize search results into groups (clusters), one for each meaning of the query. These groups are usually constructed according to the topical similarity of the retrieved documents, but it is possible for documents to be totally dissimilar and still correspond to the same meaning of the query. To overcome this problem, we exploit the thematic locality of the ...
ABSTRACT Topic Detection and Tracking (TDT) is a DARPA-sponsored initiative to investigate the st... more ABSTRACT Topic Detection and Tracking (TDT) is a DARPA-sponsored initiative to investigate the state of the art in finding and following new events in a stream of broadcast news stories. The TDT problem consists of three major tasks:(1) segmenting a stream of data, especially recognized speech, into distinct stories;(2) identifying those news stories that are the first to discuss a new event occurring in the news; and (3) given a small number of sample news stories about an event, finding all following stories in the stream.
Automatic routing and ad-hoc retrieval using SMART: TREC 2
Automatic Routing and Ad-hoc Retrieval Using SMART : TREC 2 Chris BuCkley*, James Allan, and Gera... more Automatic Routing and Ad-hoc Retrieval Using SMART : TREC 2 Chris BuCkley*, James Allan, and Gerard Salton Abstract The Smart information retrieval project emphasizes completely automatic approaches to the understanding and retrieval of large quantities of text. We continue our work in the TREC 2 environment7 performing both routing and ad-hoc experiments.
Abstract With the overwhelming volume of online news available today, there is an increasing need... more Abstract With the overwhelming volume of online news available today, there is an increasing need for automatic techniques to analyze and present news to the user in a meaningful and efficient manner. Previous research focused only on organizing news stories by their topics into a flat hierarchy. We believe viewing a news topic as a flat collection of stories is too restrictive and inefficient for a user to understand the topic quickly.
ABSTRACT We propose entity models, a representation of the language used to describe a named enti... more ABSTRACT We propose entity models, a representation of the language used to describe a named entity (person, organization, or location). The model is purely statistical and constructed from snippets of text surrounding mentions of an entity. We evaluate the effectiveness of entity models for fact-based question answering. The results obtained on question answering are promising indicating that entity models contain useful information which would aid textual data mining and other related tasks.
Abstract In traditional relevance feedback, researchers have explored relevant document feedback,... more Abstract In traditional relevance feedback, researchers have explored relevant document feedback, wherein, the query representation is updated based on a set of relevant documents returned by the user. In this work, we investigate relevant query feedback, in which we update a document's representation based on a set of relevant queries. We propose four statistical models to incorporate relevant query feedback.
Abstract The goal of the Rutgers TREC-6 Interactive Track study was to compare the performance an... more Abstract The goal of the Rutgers TREC-6 Interactive Track study was to compare the performance and usability of a system offering positive relevance feedback with one offering positive and negative relevance feedback. Our hypothesis was that the latter system would better support the aspect identification task than the former. Although aspectual recall was higher for the system supporting both kinds of relevance feedback (0.53 vs.
Abstract We describe a user study that examined the relationship between the quality of an Inform... more Abstract We describe a user study that examined the relationship between the quality of an Information Retrieval system and the effectiveness of its users in performing a task. The task involves finding answer facets of questions pertaining to a collection of newswire documents over a six month period. We artificially created sets of ranked lists at increasing levels of quality by blending the output of a state-of-the-art retrieval system with truth data created by annotators.
We are interested in how ideas from document clustering can be used to improve the retrieval accu... more We are interested in how ideas from document clustering can be used to improve the retrieval accuracy of ranked lists in interactive systems. In particular, we are interested in ways to evaluate the effectiveness of such systems to decide how they might best be constructed. In this study, we construct and evaluate systems that present the user with ranked lists and a visualization of inter-document similarities.
ABSTRACT We present a unique approach to identifying news stories that influence the behavior of ... more ABSTRACT We present a unique approach to identifying news stories that influence the behavior of financial markets. Specifically, we describe the design and implementation of rnalyst, a system that can recommend interesting news stories-stories that are likely to affect market behavior. rnalyst operates by correlating the content of news stories with trends in financial time series. We identify trends in time series using piecewise linear fitting and then assign labels to the trends according to an automated binning procedure.
A new, improved, and authentic life of James Allan. With notes by E. Mackenzie
The amusing Memoirs of James Allan,-the unrivalled piper, have been published by various persons,... more The amusing Memoirs of James Allan,-the unrivalled piper, have been published by various persons, and have commanded different degrees of attention and approbation. But the present writer has succeeded in procuring many interesting and well-authenticated anecdotes in the eventful life of this singular character, and which, he trusts, will render this work peculiarly valuable and essentially original.
Abstract We describe a demonstration system built upon Topic Detection and Tracking (TDT) technol... more Abstract We describe a demonstration system built upon Topic Detection and Tracking (TDT) technology. The demonstration system monitors a stream of news stories, organizes them into clusters that represent topics, presents the clusters to a user, and visually describes the changes that occur in those clusters over time. A user may also mark certain clusters as interesting, so that they can be" tracked" more easily.
Abstract: Information retrieval researchers have studied passage retrieval extensively, yet there... more Abstract: Information retrieval researchers have studied passage retrieval extensively, yet there is no consensus within the community about how to evaluate the results of passage retrieval experiments. This paper describes five character-level passage evaluation measures and tasks for which they may be appropriate. In the second half of the paper we compare several passage retrieval models, including a new generative mixture model that outperforms strong baselines on many of the evaluation measures discussed in part one.
ABSTRACT This report presents the system used by the University of Massachusetts for its particip... more ABSTRACT This report presents the system used by the University of Massachusetts for its participation in three of the five TDT tasks this year: detection, first story detection, and story link detection. For each task, we discuss the parameter setting approach that we used and the results of our system on the test data. In addition, we use TDT evaluation approaches to show that the tracking performance that sites are achieving is what is expected from Information Retrieval technology.
Abstract: For the TREC 2004 Novelty track, UMass participated in all four tasks. Although finding... more Abstract: For the TREC 2004 Novelty track, UMass participated in all four tasks. Although finding relevant sentences was harder this year than last, we continue to show marked improvements over the baseline of calling all sentences relevant, with a variant of tfidf being the most successful approach. We achieve 5-9% improvements over the baseline in locating novel sentences, primarily by looking at the similarity of a sentence to earlier sentences and focusing on named entities.
Abstract: In this work, we consider the task of extracting key-words such as key-players, key-loc... more Abstract: In this work, we consider the task of extracting key-words such as key-players, key-locations, key-nouns and key-verbs from news stories. We cast this problem as a classification problem wherein we assign appropriate labels to each word in a news story. We considered statistical models such as naive Bayes model, hidden Markov model and maximum entropy model in our work. We have also experimented with various features.
Abstract We describe a new probabilistic Sentence Tree Language Modeling approach that captures t... more Abstract We describe a new probabilistic Sentence Tree Language Modeling approach that captures term dependency patterns in Topic Detection and Tracking's (TDT) Story Link Detection task.
Abstract. In this paper we investigate a general purpose interactive information organization sys... more Abstract. In this paper we investigate a general purpose interactive information organization system. The system organizes documents by placing them into 1-, 2-, or 3dimensional space based on their similarity and a springembedding algorithm. We begin by developing a method for estimating the quality of the organization when it is applied to a set of documents returned in response to a query. We show how the relevant documents tend to clump together in space.
Abstract In this paper, we will compare and evaluate the effectiveness of different statistical m... more Abstract In this paper, we will compare and evaluate the effectiveness of different statistical methods in the task of cross-document coreference resolution. We created entity models for different test sets and compare the following disambiguation and clustering techniques to cluster the entity models in order to create coreference chains:
Query expansion [12] refers to the process of including related terms in the original query to pr... more Query expansion [12] refers to the process of including related terms in the original query to produce expanded queries, while query relaxation [8] refers to the dropping or down-weighting of terms from the original query to produce sub-queries. The automatic versions of both query expansion (AQE) and query relaxation (AQR) are known to fail in a large fraction of queries, and overall (average) improvements in performance can be attributed to high gains on a smaller fraction [7].
Effective representation of Web search results remains an open problem in the Information Retriev... more Effective representation of Web search results remains an open problem in the Information Retrieval community. For ambiguous queries, a traditional approach is to organize search results into groups (clusters), one for each meaning of the query. These groups are usually constructed according to the topical similarity of the retrieved documents, but it is possible for documents to be totally dissimilar and still correspond to the same meaning of the query. To overcome this problem, we exploit the thematic locality of the ...
ABSTRACT Topic Detection and Tracking (TDT) is a DARPA-sponsored initiative to investigate the st... more ABSTRACT Topic Detection and Tracking (TDT) is a DARPA-sponsored initiative to investigate the state of the art in finding and following new events in a stream of broadcast news stories. The TDT problem consists of three major tasks:(1) segmenting a stream of data, especially recognized speech, into distinct stories;(2) identifying those news stories that are the first to discuss a new event occurring in the news; and (3) given a small number of sample news stories about an event, finding all following stories in the stream.
Automatic routing and ad-hoc retrieval using SMART: TREC 2
Automatic Routing and Ad-hoc Retrieval Using SMART : TREC 2 Chris BuCkley*, James Allan, and Gera... more Automatic Routing and Ad-hoc Retrieval Using SMART : TREC 2 Chris BuCkley*, James Allan, and Gerard Salton Abstract The Smart information retrieval project emphasizes completely automatic approaches to the understanding and retrieval of large quantities of text. We continue our work in the TREC 2 environment7 performing both routing and ad-hoc experiments.
Abstract With the overwhelming volume of online news available today, there is an increasing need... more Abstract With the overwhelming volume of online news available today, there is an increasing need for automatic techniques to analyze and present news to the user in a meaningful and efficient manner. Previous research focused only on organizing news stories by their topics into a flat hierarchy. We believe viewing a news topic as a flat collection of stories is too restrictive and inefficient for a user to understand the topic quickly.
ABSTRACT We propose entity models, a representation of the language used to describe a named enti... more ABSTRACT We propose entity models, a representation of the language used to describe a named entity (person, organization, or location). The model is purely statistical and constructed from snippets of text surrounding mentions of an entity. We evaluate the effectiveness of entity models for fact-based question answering. The results obtained on question answering are promising indicating that entity models contain useful information which would aid textual data mining and other related tasks.
Abstract In traditional relevance feedback, researchers have explored relevant document feedback,... more Abstract In traditional relevance feedback, researchers have explored relevant document feedback, wherein, the query representation is updated based on a set of relevant documents returned by the user. In this work, we investigate relevant query feedback, in which we update a document's representation based on a set of relevant queries. We propose four statistical models to incorporate relevant query feedback.
Abstract The goal of the Rutgers TREC-6 Interactive Track study was to compare the performance an... more Abstract The goal of the Rutgers TREC-6 Interactive Track study was to compare the performance and usability of a system offering positive relevance feedback with one offering positive and negative relevance feedback. Our hypothesis was that the latter system would better support the aspect identification task than the former. Although aspectual recall was higher for the system supporting both kinds of relevance feedback (0.53 vs.
Abstract We describe a user study that examined the relationship between the quality of an Inform... more Abstract We describe a user study that examined the relationship between the quality of an Information Retrieval system and the effectiveness of its users in performing a task. The task involves finding answer facets of questions pertaining to a collection of newswire documents over a six month period. We artificially created sets of ranked lists at increasing levels of quality by blending the output of a state-of-the-art retrieval system with truth data created by annotators.
We are interested in how ideas from document clustering can be used to improve the retrieval accu... more We are interested in how ideas from document clustering can be used to improve the retrieval accuracy of ranked lists in interactive systems. In particular, we are interested in ways to evaluate the effectiveness of such systems to decide how they might best be constructed. In this study, we construct and evaluate systems that present the user with ranked lists and a visualization of inter-document similarities.
ABSTRACT We present a unique approach to identifying news stories that influence the behavior of ... more ABSTRACT We present a unique approach to identifying news stories that influence the behavior of financial markets. Specifically, we describe the design and implementation of rnalyst, a system that can recommend interesting news stories-stories that are likely to affect market behavior. rnalyst operates by correlating the content of news stories with trends in financial time series. We identify trends in time series using piecewise linear fitting and then assign labels to the trends according to an automated binning procedure.
A new, improved, and authentic life of James Allan. With notes by E. Mackenzie
The amusing Memoirs of James Allan,-the unrivalled piper, have been published by various persons,... more The amusing Memoirs of James Allan,-the unrivalled piper, have been published by various persons, and have commanded different degrees of attention and approbation. But the present writer has succeeded in procuring many interesting and well-authenticated anecdotes in the eventful life of this singular character, and which, he trusts, will render this work peculiarly valuable and essentially original.
Abstract We describe a demonstration system built upon Topic Detection and Tracking (TDT) technol... more Abstract We describe a demonstration system built upon Topic Detection and Tracking (TDT) technology. The demonstration system monitors a stream of news stories, organizes them into clusters that represent topics, presents the clusters to a user, and visually describes the changes that occur in those clusters over time. A user may also mark certain clusters as interesting, so that they can be" tracked" more easily.
Abstract: Information retrieval researchers have studied passage retrieval extensively, yet there... more Abstract: Information retrieval researchers have studied passage retrieval extensively, yet there is no consensus within the community about how to evaluate the results of passage retrieval experiments. This paper describes five character-level passage evaluation measures and tasks for which they may be appropriate. In the second half of the paper we compare several passage retrieval models, including a new generative mixture model that outperforms strong baselines on many of the evaluation measures discussed in part one.
ABSTRACT This report presents the system used by the University of Massachusetts for its particip... more ABSTRACT This report presents the system used by the University of Massachusetts for its participation in three of the five TDT tasks this year: detection, first story detection, and story link detection. For each task, we discuss the parameter setting approach that we used and the results of our system on the test data. In addition, we use TDT evaluation approaches to show that the tracking performance that sites are achieving is what is expected from Information Retrieval technology.
Abstract: For the TREC 2004 Novelty track, UMass participated in all four tasks. Although finding... more Abstract: For the TREC 2004 Novelty track, UMass participated in all four tasks. Although finding relevant sentences was harder this year than last, we continue to show marked improvements over the baseline of calling all sentences relevant, with a variant of tfidf being the most successful approach. We achieve 5-9% improvements over the baseline in locating novel sentences, primarily by looking at the similarity of a sentence to earlier sentences and focusing on named entities.
Abstract: In this work, we consider the task of extracting key-words such as key-players, key-loc... more Abstract: In this work, we consider the task of extracting key-words such as key-players, key-locations, key-nouns and key-verbs from news stories. We cast this problem as a classification problem wherein we assign appropriate labels to each word in a news story. We considered statistical models such as naive Bayes model, hidden Markov model and maximum entropy model in our work. We have also experimented with various features.
Abstract We describe a new probabilistic Sentence Tree Language Modeling approach that captures t... more Abstract We describe a new probabilistic Sentence Tree Language Modeling approach that captures term dependency patterns in Topic Detection and Tracking's (TDT) Story Link Detection task.
Abstract. In this paper we investigate a general purpose interactive information organization sys... more Abstract. In this paper we investigate a general purpose interactive information organization system. The system organizes documents by placing them into 1-, 2-, or 3dimensional space based on their similarity and a springembedding algorithm. We begin by developing a method for estimating the quality of the organization when it is applied to a set of documents returned in response to a query. We show how the relevant documents tend to clump together in space.
Abstract In this paper, we will compare and evaluate the effectiveness of different statistical m... more Abstract In this paper, we will compare and evaluate the effectiveness of different statistical methods in the task of cross-document coreference resolution. We created entity models for different test sets and compare the following disambiguation and clustering techniques to cluster the entity models in order to create coreference chains:
Query expansion [12] refers to the process of including related terms in the original query to pr... more Query expansion [12] refers to the process of including related terms in the original query to produce expanded queries, while query relaxation [8] refers to the dropping or down-weighting of terms from the original query to produce sub-queries. The automatic versions of both query expansion (AQE) and query relaxation (AQR) are known to fail in a large fraction of queries, and overall (average) improvements in performance can be attributed to high gains on a smaller fraction [7].
Uploads
Papers by James Allan