Adaptive web information extraction

2004

Abstract The Web wrapping proble, ie, the problem of extracting structured information from HTML documents, is one of great practical importance. The often observed information overload that users of the Web experience witnesses the lack of intelligent and encompassing Web services that provide high-quality collected and value-added inforamtion. The Web wrapping problem has been addressed by a significant amount of research work.

1993

Information Systems, 2001

The amount of useful semi-structured data on the web continues to grow at a stunning pace. Often interesting web data are not in database systems but in HTML pages, XML pages, or text files. Data in these formats are not directly usable by standard SQL-like query processing engines that support sophisticated querying and reporting beyond keyword-based retrieval. Hence, the web users or applications need a smart way of extracting data from these web sources. One of the popular approaches is to write wrappers around the sources, either manually or with software assistance, to bring the web data within the reach of more sophisticated query tools and general mediator-based information integration systems. In this paper, we describe the methodology and the software development of an XMLenabled wrapper construction systemFXWRAP for semi-automatic generation of wrapper programs. By XMLenabled we mean that the metadata about information content that are implicit in the original web pages will be extracted and encoded explicitly as XML tags in the wrapped documents. In addition, the query-based content filtering process is performed against the XML documents. The XWRAP wrapper generation framework has three distinct features. First, it explicitly separates tasks of building wrappers that are specific to a web source from the tasks that are repetitive for any source, and uses a component library to provide basic building blocks for wrapper programs. Second, it provides inductive learning algorithms that derive or discover wrapper patterns by reasoning about sample pages or sample specifications. Third and most importantly, we introduce and develop a two-phase code generation framework. The first phase utilizes an interactive interface facility to encode the source-specific metadata knowledge identified by individual wrapper developers as declarative information extraction rules. The second phase combines the information extraction rules generated at the first phase with the XWRAP component library to construct an executable wrapper program for the given web source. r

2004

Most recent research in the field of information extraction from the Web has concentrated on the task of extracting the underlying content of a set of similarly structured web pages. However in order to build real-world web information extraction applications this is not sufficient. Indeed, building such applications requires fully automating the access to web sources. This does not just involve the extraction of the data from web pages. There is a need to set up the necessary infrastructure allowing to query a source, retrieve the result pages, extract the results from these pages and filter out the unwanted results. In this paper we show how such an infrastructure can be set up. We propose to build a web information extraction application by decomposing it into sub-tasks and describing it in an XML based language named WetDL. Each of the sub-tasks consists in applying a web information extraction specific operation onto its input, one of these operators being the application of an extractor. By connecting such operations together it is possible to simply define complex applications. This is shown in the paper by applying this approach to real-world information extraction tasks such as extracting DVD listings from Amazon.com, extracting addresses from online telephone directories superpages.com, etc.

PACIS 2006 Proceedings, 2006

A new wrapper induction algorithm WTM for generating rules that describe the general web page layout template is presented. WTM is mainly designed for use in weblog crawling and indexing system. Most weblogs are maintained by content management systems and have similar layout structures in all pages. In addition, they provide RSS feeds to describe the latest entries. These entries appear in the weblog homepage in HTML format as well. WTM is built upon these two observations. It uses RSS feed data to automatically label the corresponding HTML file (weblog homepage) and induces general template rules from the labeled page. The rules can then be used to extract data from other pages of similar layout template. WTM is tested on some selected weblogs and the results are satisfactory.

The Amorphic system is an adaptive web information extraction scheme for building intelligent systems for mining information from web pages. It can locate data of interest based on domain-knowledge or page structure, can automatically generate a wrapper for an information source, and can detect when the structure of a web-based resource has changed and act on this knowledge to search the updated resource to locate the desired information. This allows Amorphic to adapt to changing structures of websites allowing users to manage their information extraction more effectively. Five different example implementations are described to illustrate the need for information extraction systems capable of extracting information from semi-structured web documents. They demonstrate the versatility of the system, showing how a system, like Amorphic, can be used in systematic data extraction applications that require data collection to be conducted over an extended period of time. The current Amorph...

2007

Web Information Extraction (WIE) is a very popular topic, however we have yet to find a fully operational implementation of WIE, especially in the training courses domain. This paper explores the variety of technologies that can be used for this kind of project and introduces some of the issues that we have experienced. Our aim is to show a different view of WIE, as a reference model for future projects.

Decision Support Systems, 2003

The World Wide Web is now undeniably the richest and most dense source of information; yet, its structure makes it difficult to make use of that information in a systematic way. This paper proposes a pattern discovery approach to the rapid generation of information extractors that can extract structured data from semi-structured Web documents. Previous work in wrapper induction aims at learning extraction rules from user-labeled training examples, which, however, can be expensive in some practical applications. In this paper, we introduce IEPAD (an acronym for Information Extraction based on PAttern Discovery), a system that discovers extraction patterns from Web pages without user-labeled examples. IEPAD applies several pattern discovery techniques, including PAT-trees, multiple string alignments and pattern matching algorithms. Extractors generated by IEPAD can be generalized over unseen pages from the same Web data source. We empirically evaluate the performance of IEPAD on an information extraction task from 14 real Web data sources. Experimental results show that with the extraction rules discovered from a single page, IEPAD achieves 96% average retrieval rate, and with less than five example pages, IEPAD achieves 100% retrieval rate for 10 of the sample Web data sources. D

2002

In this paper, we examine an important recent rule-based information extraction (IE) technique named Boosted Wrapper Induction (BWI), by conducting experiments on a wider variety of tasks than previously studied, inc luding tasks using several collections of natural text documents. We provide a systematic analysis of how each algorithmic component of BWI, in particular boostin g, contributes to its success. We show that the benefit of boosting arises from the a bility to reweight examples to learn specific rules (resulting in high precision) combined with the ability to continue learning rules after all positive examples have bee n covered (resulting in high recall). As a quantitative indicator of the regularity of an extraction task, we propose a new measure that we call SWI ratio. We show that this measure is a good predictor of IE success. Based on these results, we analyze the st rengths and limitations of current rule-based IE methods in general. Specifically, we explain limitations in the information made available to these methods, and in the representations they use. We also discuss how confidence values returned duri ng extraction are not true probabilities. In this analysis, we investigate th e benefits of including grammatical and semantic information for natural text documents , as well as parse tree and attribute-value information for XML and HTML docume nts. We show experimentally that incorporating even limited gram matical information can improve the regularity of and hence performance on natural text extraction tasks. We conclude with proposals for enriching the represent ational power of rule-based IE methods to exploit these and other types of regular ities. generate their rules automatically given labeled or partially labe led data. Second, generating a single, general rule for extracting all instances of a given field is often impossible (Muslea et al ., 1999). Therefore, most systems attempt to learn a number of rules that together cover the training examples for a field, and t hen combine these rules in some way. Some recent techniques for generating rules in the realm of text ext raction are called "wrapper induction" methods. These techniques have proved to be fai rly successful for IE tasks in their intended domains, which are colle ctions of highly structured documents such as web pages generated from a template sc ript (Muslea et al ., 1999; Kushmerick, 2000). However, wrapper induction methods do not extend wel l to natural language documents because of the specificity of the induce d rules.

2004

Many online information sources are available on the Web. Giving machine access to such sources leads to many interesting applications, such as using web data in mediators or software agents. Up to now most work in the field of information extraction from the web has concentrated on building wrappers, i.e. programs allowing to reformat presentational data in HTML into a more machine comprehensible format. While being an important part of a web information extraction application such wrappers are not sufficient to fully access a source. Indeed, it is necessary to setup an infrastructure allowing to build queries, fetch pages, extract specific links, etc. In this paper we propose a language called WetDL allowing to describe an information extraction task as a network of operators whose execution performs the desired extraction task.