Continuous join queries (CJQ) are needed for correlating data from multiple streams. One fundamen... more Continuous join queries (CJQ) are needed for correlating data from multiple streams. One fundamental problem for processing such queries is that since the data streams are infinite, this would require the join operator to store infinite states and eventually run out of space. Punctuation semantics has been proposed to specifically address this problem. In particular, punctuations explicitly mark the end of a subset of data and, hence, enable purging of the stored data which will not contribute to any new query results. Given a set of available punctuation schemes, if one can identify that a CJQ still requires unbounded storage, then this query can be flagged as unsafe and can be prevented from running. Unfortunately, while punctuation semantics is clearly useful, the mechanisms to identify if and how a particular CJQ could benefit from a given set of punctuation schemes are not yet known. In this paper, we provide sufficient and necessary conditions for checking whether a CJQ can be safely executed under a given set of punctuation schemes or not. In particular, we introduce a novel punctuation graph to aid the analysis of the safety for a given query. We show that the safety checking problem can be done in polynomial time based on this punctuation graph construct. In addition, various issues and challenges related to the safety checking of CJQs are highlighted.
XML message filtering problem involves searching for instances of a given, potentially large, set... more XML message filtering problem involves searching for instances of a given, potentially large, set of patterns in a continuous stream of XML messages. Since the messages arrive continuously, it is essential that the filtering rate matches the data arrival rate. Therefore, the given set of filter patterns needs to be indexed appropriately to enable real-time processing of the streaming XML data. In this paper, we propose AFilter, an adaptable, and thus scalable, path expression filtering approach. AFilter has a base memory requirement linear in filter expression and data size. Furthermore, when additional memory is available, AFilter can exploit prefix commonalities in the set of filter expressions using a loosely-coupled prefix caching mechanism as opposed to tightly-coupled active state representation of alternative approaches. Unlike existing systems, AFilter can also exploit suffix-commonalities across filter expressions, while simultaneously leveraging the prefix-commonalities through the cache. Finally, AFilter uses a triggering mechanism to prevent excessive consumption of resources by delaying processing until a trigger condition is observed. Experiment results show that AFilter provides significantly better scalability and runtime performance when compared to state of the art filtering systems.
Tree pattern matching is one of the most fundamental tasks for XML query processing. Holistic twi... more Tree pattern matching is one of the most fundamental tasks for XML query processing. Holistic twig query processing techniques [4, 16] have been developed to minimize the intermediate results, namely, those root-to-leaf path matches that are not in the final twig results. However, useless path matches cannot be completely avoided, especially when there is a parent-child relationship in the twig query. Furthermore, existing approaches do not consider the fact that in practice, in order to process XPath or XQuery statements, a more powerful form of twig queries, namely, Generalized-Tree-Pattern (GTP) [8] queries, is required. Most existing works on processing GTP queries generally calls for costly post-processing for eliminating redundant data and/or grouping of the matching results. In this paper, we first propose a novel hierarchical stack encoding scheme to compactly represent the twig results. We introduce Twig 2 Stack, a bottom-up algorithm for processing twig queries based on this encoding scheme. Then we show how to efficiently enumerate the query results from the encodings for a given GTP query. To our knowledge, this is the first GTP matching solution that avoids any post path-join, sort, duplicate elimination and grouping operations. Extensive performance studies on various data sets and queries show that the proposed Twig 2 Stack algorithm not only has better twig query processing performance than state-of-the-art algorithms, but is also capable of efficiently processing the more complex GTP queries. * The work has been done during the author's internship at NEC Laboratories America.
21st International Conference on Data Engineering (ICDE'05)
Data warehousing and on-line analytical processing (OLAP) are essential for decision support appl... more Data warehousing and on-line analytical processing (OLAP) are essential for decision support applications. Common OLAP operations include for example drill down, roll up, pivot and unpivot. Typically, such queries are fairly complex and are often executed over huge volumes of data. The solution in practice is to use materialized views to reduce the query cost. Utilizing materialized views that incorporate not just traditional simple SELECT-PROJECT-JOIN operators but also complex OLAP operators such as pivot and unpivot is crucial to improve the OLAP query performance but as of now unexplored topic. In this work, we demonstrate that the efficient maintenance of views with pivot and unpivot operators requires the definition of more generalized operators, which we call GPIVOT and GUNPIVOT. We propose rewriting rules, combination rules and propagation rules for such operators. We also design a novel view maintenance framework for applying these rules to obtain an efficient maintenance plan. Extensive experimental evaluation reveals the efficiency of our proposed maintenance techniques. Our query transformation rules are thus dual purpose serving both view maintenance and query optimization. This paves the way for the inclusion of the GPIVOT and GUNPIVOT into any DBMS engine. Extensively performance study reveals the effectiveness of our proposed maintenance strategies.
Data integration over multiple heterogeneous data sources has become increasingly important for m... more Data integration over multiple heterogeneous data sources has become increasingly important for modern applications. The integrated data is usually stored in materialized views for better access, performance and high availability. Such views must be maintained after the data sources change. In a loosely-coupled environment, such as the Data Grid, the source updates are autonomous and may cause erroneous maintenance results. State-of-the-art maintenance strategies apply compensating queries to correct such errors. However, they assume that the source schema remain static. This is an unrealistic assumption for such dynamic environments, where the data sources may change not only their data but also their schema, query capabilities or semantics. Consequently, either the maintenance or compensating queries may fail. In this paper, first, we analyze the maintenance errors and classify them into different classes of dependencies. We then propose Dyno, a two-pronged strategy composed of dependency detection and correction algorithms to handle these new classes of concurrency. Our techniques are not tied to specific maintenance algorithms nor to a particular data model. To our knowledge, this is the first complete solution to the view maintenance concurrency problems for both data and schema changes. Our experimental results illustrate that Dyno imposes an almost negligible overhead on existing maintenance algorithms for data updates while now allowing for this extended functionality.
Data integration over multiple heterogeneous data sources has become increasingly important for m... more Data integration over multiple heterogeneous data sources has become increasingly important for modern applications. The integrated data is usually stored in materialized views (MV) to allow better access, performance and high availability. MV must be maintained after the data sources change. In a loosely-coupled environment, such as the Data Grid, the data sources are autonomous. Hence the source updates can be concurrent and cause erroneous maintenance results. State-of-the-art maintenance strategies apply compensating queries to correct such errors, making the restricting assumption that all source schemata remain static over time. However, in such dynamic environments, the data sources may change not only their data but also their schema, query capabilities or semantics. Consequently, either the maintenance queries or compensating queries would fail. In this paper, we propose a framework called DyDa that overcomes these limitations and handles both source data updates and schema changes. First, we identify three types of maintenance anomalies, caused by either data updates and/or rename and/or drop schema changes. We propose a compensation algorithm to solve the first two types of anomalies. We identify that the third type of anomaly is caused by the violation of dependencies between the maintenance processes. We propose a detection and correction algorithm to remove such anomalies based on the formalisms of dependencies. A new view adaptation algorithm is designed to incrementally adapt some complex updates introduced by the correction algorithm. Put together, these algorithms are the first complete solution to the concurrency problems for MV maintenance in loosely-coupled environments. We have implemented the DyDa system. The experimental results show that our new concurrency handling strategy imposes a minimal overhead on normal data update processing while allowing for the extended functionality to maintain the materialized views even under concurrent schema changes.
Proceedings of the 2008 ACM SIGMOD international conference on Management of data, 2008
OLAP is an important tool in decision support. With the help of domain knowledge, such as hierarc... more OLAP is an important tool in decision support. With the help of domain knowledge, such as hierarchies of attribute values, OLAP helps the user observe the effects of various decisions. One assumption of most OLAP operations is that the available domain knowledge is precise. In particular, they assume that the hierarchy of values over which the user can navigate forms a taxonomy. In this paper, we first note that when multiple heterogeneous data sources are involved in the gathering of the data and the associated domain knowledge, the integrated knowledge-base, constructed by combining locally available taxonomies based on the concept matchings, may not be a taxonomy itself. Specifically, existence of intersections among concepts from different sources compromises the tree-structure of the integrated taxonomy and prevents effective use of hierarchical navigation techniques, such as drill-down and roll-up. To cope with this, we introduce concept un-classification, where a select few of the concepts are eliminated to ensure that the remaining structure is a navigable taxonomy, without concept intersections. Since un-classifying an originally classified data is not desirable, we consider ways to minimize un-classification in the process. We introduce a cost model which captures the imprecision caused by the un-classification process and we formulate the problem of finding an un-classification strategy which eliminates intersections and which adds minimal imprecision to the resulting structure. We show that, when performed naively, this task can be very costly and thus we propose a bottom-up preprocessing strategy which supports basic navigational analytics operations, such as drill-down and roll-up. Experiments over synthetic and real-life data verified the effectiveness and efficiency of our approach.
A Data Warehouse Management System (DWMS) incrementally maintains materialized views by issuing m... more A Data Warehouse Management System (DWMS) incrementally maintains materialized views by issuing maintenance queries to the data sources. To address erroneous query results caused by concurrent source updates, state-of-the-art maintenance strategies typically apply compensations to resolve the conflicts. For this, they assume however that the source schema are not updated and remain stable over time. However, if schema changes occur in any of the sources, then an anomaly may arise, namely, the maintenance or the compensation queries may be broken. We now tackle this open problem by modeling the complete maintenance process as a special transaction, called a DWMS Transaction. The anomaly problem can now be rephrased as the serializability of DWMS Transactions. This allows us to apply wellestablished transaction theory to address this new anomaly problem. To achieve such serializability, we propose a multiversion concurrency control technique appropriate for loosely-coupled environments with autonomous sources. TxnWrap is complementary to maintenance algorithms from the literature by removing concurrency issues from their consideration. The experimental results confirm that TxnWrap achieves predictable steady performance even under a varying rate of concurrency.
2008 IEEE 24th International Conference on Data Engineering, 2008
Detecting complex patterns in event streams, i.e., complex event processing (CEP), has become inc... more Detecting complex patterns in event streams, i.e., complex event processing (CEP), has become increasingly important for modern enterprises to react quickly to critical situations. In many practical cases business events are generated based on pre-defined business logics. Hence constraints, such as occurrence and order constraints, often hold among events. Reasoning using these known constraints enables us to predict the non-occurrences of certain future events, thereby helping us to identify and then terminate the long running query processes that are guaranteed to not lead to successful matches. In this work, we focus on exploiting event constraints to optimize CEP over large volumes of business transaction streams. Since the optimization opportunities arise at runtime, we develop a runtime query unsatisfiability (RunSAT) checking technique that detects optimal points for terminating query evaluation. To assure efficiency of RunSAT checking, we propose mechanisms to precompute the query failure conditions to be checked at runtime. This guarantees a constant-time RunSAT reasoning cost, making our technique highly scalable. We realize our optimal query termination strategies by augmenting the query with Event-Condition-Action rules encoding the pre-computed failure conditions. This results in an event processing solution compatible with state-of-the-art CEP architectures. Extensive experimental results demonstrate that significant performance gains are achieved, while the optimization overhead is small.
FACT (Fast and ACcuraTe) is a query processing system aimed at providing users with facilities so... more FACT (Fast and ACcuraTe) is a query processing system aimed at providing users with facilities so that they can get the query results from the Web in a database-like fashion. The system takes user queries in the form of keywords (free text) and returns segments of Web pages that contain the required information. It works as follows. The input from a user is passed to a general- purpose search engine to obtain a set of URLs of Web pages that may contain the required information. The system later locates the query results from the Web pages reachable from these URLs. Since queries expressed in keywords may be not able to express query requirements precisely or may not guarantee the discovery of required information inherently, the user is asked to first browse a few pages, during which the system learns from her/him about the exact query requirements and heuristics of finding the required information through a series of hyperlinks. The system will process the rest URLs and present the...
Large-scale data analysis has become increasingly important for many enterprises. Recently, a new... more Large-scale data analysis has become increasingly important for many enterprises. Recently, a new distributed computing paradigm, called MapReduce, and its open source implementation Hadoop, has been widely adopted due to its impressive scalability and flexibility to handle structured as well as unstructured data. In this paper, we describe our data warehouse system, called Cheetah, built on top of MapReduce. Cheetah is designed specifically for our online advertising application to allow various simplifications and custom optimizations. First, we take a fresh look at the data warehouse schema design. In particular, we define a virtual view on top of the common star or snowflake data warehouse schema. This virtual view abstraction not only allows us to design a SQL-like but much more succinct query language, but also makes it easier to support many advanced query processing features. Next, we describe a stack of optimization techniques ranging from data compression and access method...
Proceedings of the eleventh international conference on Information and knowledge management, 2002
Data warehouse view maintenance is an important issue due to the growing use of warehouse technol... more Data warehouse view maintenance is an important issue due to the growing use of warehouse technology for information integration and data analysis. Given the dynamic nature of modern distributed environments, both data updates and schema changes are likely to occur in different data sources. In applications for which the real-time refresh of the data warehouse extent under source change is not critical, the source updates are usually maintained in a batch fashion to reduce the maintenance overhead. However, most prior work can only deal with the batching source data updates. In this paper, we now provide a solution strategy that is capable of batching both source data updates and schema changes. We propose techniques to first preprocess the initial source updates to construct summarized delta changes for each source. We then design a view adaptation algorithm to adapt the warehouse view under these delta changes. We have implemented our proposed batching solution and incorporated it into an existing data warehouse prototype system. The experimental studies demonstrate the excellent performance achievable by our batch technique.
Proceedings of the 2008 ACM SIGMOD international conference on Management of data, 2008
The UQBE is a mashup tool for non-programmers that supports query-by-example (QBE) over a schema ... more The UQBE is a mashup tool for non-programmers that supports query-by-example (QBE) over a schema made up by the user without knowing the schema of the original sources. Based on automated schema matching with uncertainty, the UQBE system returns the best confident results. The system lets the user refine them interactively. A tuple in the query result is associated with lineage that is a boolean formula over schema matching decisions representing underlying conditions on which the corresponding tuple is included in the result. Given binary feedbacks on tuples by the user, which are possibly imprecise, the system solves it as an optimization problem to refine confidence values of matching decisions. The demo features graphical user interaction on the UQBE system, including querying and refinement.
The Ninth International Conference on Mobile Data Management (mdm 2008), 2008
Monitoring moving objects is one of the key application domains for sensor networks. In the absen... more Monitoring moving objects is one of the key application domains for sensor networks. In the absence of cooperative objects and devices attached to these objects, target tracking algorithms have to be used for monitoring. In this paper, we present that many of the applications of moving object monitoring systems could be addressed with low-frequency snapshot-based queries. With the realization of this query type, we show that existing target tracking algorithms may not be the least expensive solutions. We introduce an approach that uses two alternating strategies. We maintain a cheap low-quality knowledge of moving objects' location between snapshots and trigger expensive sensor readings only when a snapshot period has elapsed. With extensive experiments we show that our approach is significantly more energy efficient than established methods. It is also more effective than existing data-and-query centric in-network query processing schemes as it can maintain object identities between snapshots.
In this paper, we describe a novel Web query processing approach with learning capabilities. Unde... more In this paper, we describe a novel Web query processing approach with learning capabilities. Under this approach, user queries are in the form of keywords and search engines are employed to find URLs of Web sites that might contain the required information. The first few URLs are presented to the user for browsing. Meanwhile, the query processor learns both the information required by the user and the way that the user navigates through hyperlinks to locate such information. With the learned knowledge, it processes the rest URLs and produces precise query results in the form of segments of Web pages without user involvement. The preliminary experimental results indicate that the approach can process a range of Web queries with satisfactory performance. The architecture of such a query processor, techniques of modeling HTML pages, and knowledge for query processing are discussed. Experiments on the effectiveness of the approach, the required knowledge, and the training strategies are presented.
IEEE Transactions on Knowledge and Data Engineering, 2008
An XML publish/subscribe system needs to filter a large number of queries over XML streams. Most ... more An XML publish/subscribe system needs to filter a large number of queries over XML streams. Most existing systems only consider filtering the simple XPath statements. In this paper, we focus on filtering of the more complex Generalized-Tree-Pattern (GTP) queries. Our filtering mechanism is based on a novel Tree-of-Path (TOP) encoding scheme, which compactly represents the path matches for the entire document. First, we show that the TOP encodings can be efficiently produced via shared bottom-up path matching. Second, with the aid of this TOP encoding, we can 1) achieve polynomial time and space complexity for postprocessing, 2) avoid redundant predicate evaluations, 3) allow an efficient duplicate-free and merge join-based algorithm for merging multiple encoded path matches, and 4) simplify the processing of GTP queries. Overall, our approach maximizes the sharing opportunity across queries by exploiting the suffix as well as prefix sharing. At the same time, our TOP encodings allow efficient postprocessing for GTP queries. Extensive performance studies show that GFilter not only achieves significantly better filtering performance than state-ofthe-art algorithms but also is capable of efficiently filtering the more complex GTP queries.
IEEE Transactions on Knowledge and Data Engineering, 2006
Data integration over multiple heterogeneous data sources has become increasingly important for m... more Data integration over multiple heterogeneous data sources has become increasingly important for modern applications. The integrated data is usually stored as materialized views to allow better access, performance, and high availability. In loosely coupled environments, such as the Data Grid, the data sources are autonomous. Hence, the source updates can be concurrent and cause erroneous results during view maintenance. State-of-the-art maintenance strategies apply compensating queries to correct such errors, making the restricting assumption that all source schemata remain static over time. However, in such dynamic environments, the data sources may change not only their data but also their schema. Consequently, either the maintenance queries or the compensating queries may fail. In this paper, we propose a novel framework called DyDa that overcomes these limitations and handles both source data updates and schema changes. We identify three types of maintenance anomalies, caused by either source data updates, data-preserving schema changes, or non-data-preserving schema changes. We propose a compensation algorithm to solve the first two types of anomalies. We show that the third type of anomaly is caused by the violation of dependencies between maintenance processes. Then, we propose dependency detection and correction algorithms to identify and resolve the violations. Put together, DyDa extends prior maintenance solutions to solve all types of view maintenance anomalies. The experimental results show that DyDa imposes a minimal overhead on data update processing while allowing for the extended functionality to handle concurrent schema changes.
Continuous join queries (CJQ) are needed for correlating data from multiple streams. One fundamen... more Continuous join queries (CJQ) are needed for correlating data from multiple streams. One fundamental problem for processing such queries is that since the data streams are infinite, this would require the join operator to store infinite states and eventually run out of space. Punctuation semantics has been proposed to specifically address this problem. In particular, punctuations explicitly mark the end of a subset of data and, hence, enable purging of the stored data which will not contribute to any new query results. Given a set of available punctuation schemes, if one can identify that a CJQ still requires unbounded storage, then this query can be flagged as unsafe and can be prevented from running. Unfortunately, while punctuation semantics is clearly useful, the mechanisms to identify if and how a particular CJQ could benefit from a given set of punctuation schemes are not yet known. In this paper, we provide sufficient and necessary conditions for checking whether a CJQ can be safely executed under a given set of punctuation schemes or not. In particular, we introduce a novel punctuation graph to aid the analysis of the safety for a given query. We show that the safety checking problem can be done in polynomial time based on this punctuation graph construct. In addition, various issues and challenges related to the safety checking of CJQs are highlighted.
XML message filtering problem involves searching for instances of a given, potentially large, set... more XML message filtering problem involves searching for instances of a given, potentially large, set of patterns in a continuous stream of XML messages. Since the messages arrive continuously, it is essential that the filtering rate matches the data arrival rate. Therefore, the given set of filter patterns needs to be indexed appropriately to enable real-time processing of the streaming XML data. In this paper, we propose AFilter, an adaptable, and thus scalable, path expression filtering approach. AFilter has a base memory requirement linear in filter expression and data size. Furthermore, when additional memory is available, AFilter can exploit prefix commonalities in the set of filter expressions using a loosely-coupled prefix caching mechanism as opposed to tightly-coupled active state representation of alternative approaches. Unlike existing systems, AFilter can also exploit suffix-commonalities across filter expressions, while simultaneously leveraging the prefix-commonalities through the cache. Finally, AFilter uses a triggering mechanism to prevent excessive consumption of resources by delaying processing until a trigger condition is observed. Experiment results show that AFilter provides significantly better scalability and runtime performance when compared to state of the art filtering systems.
Tree pattern matching is one of the most fundamental tasks for XML query processing. Holistic twi... more Tree pattern matching is one of the most fundamental tasks for XML query processing. Holistic twig query processing techniques [4, 16] have been developed to minimize the intermediate results, namely, those root-to-leaf path matches that are not in the final twig results. However, useless path matches cannot be completely avoided, especially when there is a parent-child relationship in the twig query. Furthermore, existing approaches do not consider the fact that in practice, in order to process XPath or XQuery statements, a more powerful form of twig queries, namely, Generalized-Tree-Pattern (GTP) [8] queries, is required. Most existing works on processing GTP queries generally calls for costly post-processing for eliminating redundant data and/or grouping of the matching results. In this paper, we first propose a novel hierarchical stack encoding scheme to compactly represent the twig results. We introduce Twig 2 Stack, a bottom-up algorithm for processing twig queries based on this encoding scheme. Then we show how to efficiently enumerate the query results from the encodings for a given GTP query. To our knowledge, this is the first GTP matching solution that avoids any post path-join, sort, duplicate elimination and grouping operations. Extensive performance studies on various data sets and queries show that the proposed Twig 2 Stack algorithm not only has better twig query processing performance than state-of-the-art algorithms, but is also capable of efficiently processing the more complex GTP queries. * The work has been done during the author's internship at NEC Laboratories America.
21st International Conference on Data Engineering (ICDE'05)
Data warehousing and on-line analytical processing (OLAP) are essential for decision support appl... more Data warehousing and on-line analytical processing (OLAP) are essential for decision support applications. Common OLAP operations include for example drill down, roll up, pivot and unpivot. Typically, such queries are fairly complex and are often executed over huge volumes of data. The solution in practice is to use materialized views to reduce the query cost. Utilizing materialized views that incorporate not just traditional simple SELECT-PROJECT-JOIN operators but also complex OLAP operators such as pivot and unpivot is crucial to improve the OLAP query performance but as of now unexplored topic. In this work, we demonstrate that the efficient maintenance of views with pivot and unpivot operators requires the definition of more generalized operators, which we call GPIVOT and GUNPIVOT. We propose rewriting rules, combination rules and propagation rules for such operators. We also design a novel view maintenance framework for applying these rules to obtain an efficient maintenance plan. Extensive experimental evaluation reveals the efficiency of our proposed maintenance techniques. Our query transformation rules are thus dual purpose serving both view maintenance and query optimization. This paves the way for the inclusion of the GPIVOT and GUNPIVOT into any DBMS engine. Extensively performance study reveals the effectiveness of our proposed maintenance strategies.
Data integration over multiple heterogeneous data sources has become increasingly important for m... more Data integration over multiple heterogeneous data sources has become increasingly important for modern applications. The integrated data is usually stored in materialized views for better access, performance and high availability. Such views must be maintained after the data sources change. In a loosely-coupled environment, such as the Data Grid, the source updates are autonomous and may cause erroneous maintenance results. State-of-the-art maintenance strategies apply compensating queries to correct such errors. However, they assume that the source schema remain static. This is an unrealistic assumption for such dynamic environments, where the data sources may change not only their data but also their schema, query capabilities or semantics. Consequently, either the maintenance or compensating queries may fail. In this paper, first, we analyze the maintenance errors and classify them into different classes of dependencies. We then propose Dyno, a two-pronged strategy composed of dependency detection and correction algorithms to handle these new classes of concurrency. Our techniques are not tied to specific maintenance algorithms nor to a particular data model. To our knowledge, this is the first complete solution to the view maintenance concurrency problems for both data and schema changes. Our experimental results illustrate that Dyno imposes an almost negligible overhead on existing maintenance algorithms for data updates while now allowing for this extended functionality.
Data integration over multiple heterogeneous data sources has become increasingly important for m... more Data integration over multiple heterogeneous data sources has become increasingly important for modern applications. The integrated data is usually stored in materialized views (MV) to allow better access, performance and high availability. MV must be maintained after the data sources change. In a loosely-coupled environment, such as the Data Grid, the data sources are autonomous. Hence the source updates can be concurrent and cause erroneous maintenance results. State-of-the-art maintenance strategies apply compensating queries to correct such errors, making the restricting assumption that all source schemata remain static over time. However, in such dynamic environments, the data sources may change not only their data but also their schema, query capabilities or semantics. Consequently, either the maintenance queries or compensating queries would fail. In this paper, we propose a framework called DyDa that overcomes these limitations and handles both source data updates and schema changes. First, we identify three types of maintenance anomalies, caused by either data updates and/or rename and/or drop schema changes. We propose a compensation algorithm to solve the first two types of anomalies. We identify that the third type of anomaly is caused by the violation of dependencies between the maintenance processes. We propose a detection and correction algorithm to remove such anomalies based on the formalisms of dependencies. A new view adaptation algorithm is designed to incrementally adapt some complex updates introduced by the correction algorithm. Put together, these algorithms are the first complete solution to the concurrency problems for MV maintenance in loosely-coupled environments. We have implemented the DyDa system. The experimental results show that our new concurrency handling strategy imposes a minimal overhead on normal data update processing while allowing for the extended functionality to maintain the materialized views even under concurrent schema changes.
Proceedings of the 2008 ACM SIGMOD international conference on Management of data, 2008
OLAP is an important tool in decision support. With the help of domain knowledge, such as hierarc... more OLAP is an important tool in decision support. With the help of domain knowledge, such as hierarchies of attribute values, OLAP helps the user observe the effects of various decisions. One assumption of most OLAP operations is that the available domain knowledge is precise. In particular, they assume that the hierarchy of values over which the user can navigate forms a taxonomy. In this paper, we first note that when multiple heterogeneous data sources are involved in the gathering of the data and the associated domain knowledge, the integrated knowledge-base, constructed by combining locally available taxonomies based on the concept matchings, may not be a taxonomy itself. Specifically, existence of intersections among concepts from different sources compromises the tree-structure of the integrated taxonomy and prevents effective use of hierarchical navigation techniques, such as drill-down and roll-up. To cope with this, we introduce concept un-classification, where a select few of the concepts are eliminated to ensure that the remaining structure is a navigable taxonomy, without concept intersections. Since un-classifying an originally classified data is not desirable, we consider ways to minimize un-classification in the process. We introduce a cost model which captures the imprecision caused by the un-classification process and we formulate the problem of finding an un-classification strategy which eliminates intersections and which adds minimal imprecision to the resulting structure. We show that, when performed naively, this task can be very costly and thus we propose a bottom-up preprocessing strategy which supports basic navigational analytics operations, such as drill-down and roll-up. Experiments over synthetic and real-life data verified the effectiveness and efficiency of our approach.
A Data Warehouse Management System (DWMS) incrementally maintains materialized views by issuing m... more A Data Warehouse Management System (DWMS) incrementally maintains materialized views by issuing maintenance queries to the data sources. To address erroneous query results caused by concurrent source updates, state-of-the-art maintenance strategies typically apply compensations to resolve the conflicts. For this, they assume however that the source schema are not updated and remain stable over time. However, if schema changes occur in any of the sources, then an anomaly may arise, namely, the maintenance or the compensation queries may be broken. We now tackle this open problem by modeling the complete maintenance process as a special transaction, called a DWMS Transaction. The anomaly problem can now be rephrased as the serializability of DWMS Transactions. This allows us to apply wellestablished transaction theory to address this new anomaly problem. To achieve such serializability, we propose a multiversion concurrency control technique appropriate for loosely-coupled environments with autonomous sources. TxnWrap is complementary to maintenance algorithms from the literature by removing concurrency issues from their consideration. The experimental results confirm that TxnWrap achieves predictable steady performance even under a varying rate of concurrency.
2008 IEEE 24th International Conference on Data Engineering, 2008
Detecting complex patterns in event streams, i.e., complex event processing (CEP), has become inc... more Detecting complex patterns in event streams, i.e., complex event processing (CEP), has become increasingly important for modern enterprises to react quickly to critical situations. In many practical cases business events are generated based on pre-defined business logics. Hence constraints, such as occurrence and order constraints, often hold among events. Reasoning using these known constraints enables us to predict the non-occurrences of certain future events, thereby helping us to identify and then terminate the long running query processes that are guaranteed to not lead to successful matches. In this work, we focus on exploiting event constraints to optimize CEP over large volumes of business transaction streams. Since the optimization opportunities arise at runtime, we develop a runtime query unsatisfiability (RunSAT) checking technique that detects optimal points for terminating query evaluation. To assure efficiency of RunSAT checking, we propose mechanisms to precompute the query failure conditions to be checked at runtime. This guarantees a constant-time RunSAT reasoning cost, making our technique highly scalable. We realize our optimal query termination strategies by augmenting the query with Event-Condition-Action rules encoding the pre-computed failure conditions. This results in an event processing solution compatible with state-of-the-art CEP architectures. Extensive experimental results demonstrate that significant performance gains are achieved, while the optimization overhead is small.
FACT (Fast and ACcuraTe) is a query processing system aimed at providing users with facilities so... more FACT (Fast and ACcuraTe) is a query processing system aimed at providing users with facilities so that they can get the query results from the Web in a database-like fashion. The system takes user queries in the form of keywords (free text) and returns segments of Web pages that contain the required information. It works as follows. The input from a user is passed to a general- purpose search engine to obtain a set of URLs of Web pages that may contain the required information. The system later locates the query results from the Web pages reachable from these URLs. Since queries expressed in keywords may be not able to express query requirements precisely or may not guarantee the discovery of required information inherently, the user is asked to first browse a few pages, during which the system learns from her/him about the exact query requirements and heuristics of finding the required information through a series of hyperlinks. The system will process the rest URLs and present the...
Large-scale data analysis has become increasingly important for many enterprises. Recently, a new... more Large-scale data analysis has become increasingly important for many enterprises. Recently, a new distributed computing paradigm, called MapReduce, and its open source implementation Hadoop, has been widely adopted due to its impressive scalability and flexibility to handle structured as well as unstructured data. In this paper, we describe our data warehouse system, called Cheetah, built on top of MapReduce. Cheetah is designed specifically for our online advertising application to allow various simplifications and custom optimizations. First, we take a fresh look at the data warehouse schema design. In particular, we define a virtual view on top of the common star or snowflake data warehouse schema. This virtual view abstraction not only allows us to design a SQL-like but much more succinct query language, but also makes it easier to support many advanced query processing features. Next, we describe a stack of optimization techniques ranging from data compression and access method...
Proceedings of the eleventh international conference on Information and knowledge management, 2002
Data warehouse view maintenance is an important issue due to the growing use of warehouse technol... more Data warehouse view maintenance is an important issue due to the growing use of warehouse technology for information integration and data analysis. Given the dynamic nature of modern distributed environments, both data updates and schema changes are likely to occur in different data sources. In applications for which the real-time refresh of the data warehouse extent under source change is not critical, the source updates are usually maintained in a batch fashion to reduce the maintenance overhead. However, most prior work can only deal with the batching source data updates. In this paper, we now provide a solution strategy that is capable of batching both source data updates and schema changes. We propose techniques to first preprocess the initial source updates to construct summarized delta changes for each source. We then design a view adaptation algorithm to adapt the warehouse view under these delta changes. We have implemented our proposed batching solution and incorporated it into an existing data warehouse prototype system. The experimental studies demonstrate the excellent performance achievable by our batch technique.
Proceedings of the 2008 ACM SIGMOD international conference on Management of data, 2008
The UQBE is a mashup tool for non-programmers that supports query-by-example (QBE) over a schema ... more The UQBE is a mashup tool for non-programmers that supports query-by-example (QBE) over a schema made up by the user without knowing the schema of the original sources. Based on automated schema matching with uncertainty, the UQBE system returns the best confident results. The system lets the user refine them interactively. A tuple in the query result is associated with lineage that is a boolean formula over schema matching decisions representing underlying conditions on which the corresponding tuple is included in the result. Given binary feedbacks on tuples by the user, which are possibly imprecise, the system solves it as an optimization problem to refine confidence values of matching decisions. The demo features graphical user interaction on the UQBE system, including querying and refinement.
The Ninth International Conference on Mobile Data Management (mdm 2008), 2008
Monitoring moving objects is one of the key application domains for sensor networks. In the absen... more Monitoring moving objects is one of the key application domains for sensor networks. In the absence of cooperative objects and devices attached to these objects, target tracking algorithms have to be used for monitoring. In this paper, we present that many of the applications of moving object monitoring systems could be addressed with low-frequency snapshot-based queries. With the realization of this query type, we show that existing target tracking algorithms may not be the least expensive solutions. We introduce an approach that uses two alternating strategies. We maintain a cheap low-quality knowledge of moving objects' location between snapshots and trigger expensive sensor readings only when a snapshot period has elapsed. With extensive experiments we show that our approach is significantly more energy efficient than established methods. It is also more effective than existing data-and-query centric in-network query processing schemes as it can maintain object identities between snapshots.
In this paper, we describe a novel Web query processing approach with learning capabilities. Unde... more In this paper, we describe a novel Web query processing approach with learning capabilities. Under this approach, user queries are in the form of keywords and search engines are employed to find URLs of Web sites that might contain the required information. The first few URLs are presented to the user for browsing. Meanwhile, the query processor learns both the information required by the user and the way that the user navigates through hyperlinks to locate such information. With the learned knowledge, it processes the rest URLs and produces precise query results in the form of segments of Web pages without user involvement. The preliminary experimental results indicate that the approach can process a range of Web queries with satisfactory performance. The architecture of such a query processor, techniques of modeling HTML pages, and knowledge for query processing are discussed. Experiments on the effectiveness of the approach, the required knowledge, and the training strategies are presented.
IEEE Transactions on Knowledge and Data Engineering, 2008
An XML publish/subscribe system needs to filter a large number of queries over XML streams. Most ... more An XML publish/subscribe system needs to filter a large number of queries over XML streams. Most existing systems only consider filtering the simple XPath statements. In this paper, we focus on filtering of the more complex Generalized-Tree-Pattern (GTP) queries. Our filtering mechanism is based on a novel Tree-of-Path (TOP) encoding scheme, which compactly represents the path matches for the entire document. First, we show that the TOP encodings can be efficiently produced via shared bottom-up path matching. Second, with the aid of this TOP encoding, we can 1) achieve polynomial time and space complexity for postprocessing, 2) avoid redundant predicate evaluations, 3) allow an efficient duplicate-free and merge join-based algorithm for merging multiple encoded path matches, and 4) simplify the processing of GTP queries. Overall, our approach maximizes the sharing opportunity across queries by exploiting the suffix as well as prefix sharing. At the same time, our TOP encodings allow efficient postprocessing for GTP queries. Extensive performance studies show that GFilter not only achieves significantly better filtering performance than state-ofthe-art algorithms but also is capable of efficiently filtering the more complex GTP queries.
IEEE Transactions on Knowledge and Data Engineering, 2006
Data integration over multiple heterogeneous data sources has become increasingly important for m... more Data integration over multiple heterogeneous data sources has become increasingly important for modern applications. The integrated data is usually stored as materialized views to allow better access, performance, and high availability. In loosely coupled environments, such as the Data Grid, the data sources are autonomous. Hence, the source updates can be concurrent and cause erroneous results during view maintenance. State-of-the-art maintenance strategies apply compensating queries to correct such errors, making the restricting assumption that all source schemata remain static over time. However, in such dynamic environments, the data sources may change not only their data but also their schema. Consequently, either the maintenance queries or the compensating queries may fail. In this paper, we propose a novel framework called DyDa that overcomes these limitations and handles both source data updates and schema changes. We identify three types of maintenance anomalies, caused by either source data updates, data-preserving schema changes, or non-data-preserving schema changes. We propose a compensation algorithm to solve the first two types of anomalies. We show that the third type of anomaly is caused by the violation of dependencies between maintenance processes. Then, we propose dependency detection and correction algorithms to identify and resolve the violations. Put together, DyDa extends prior maintenance solutions to solve all types of view maintenance anomalies. The experimental results show that DyDa imposes a minimal overhead on data update processing while allowing for the extended functionality to handle concurrent schema changes.
Uploads
Papers by Songting Chen