ETALIS: A Rule-based System for Complex Event Processing
Paul Fodor
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
Complex Event Processing (CEP) is concerned with timely detection of complex events within a stream of atomic occurrences, and has useful applications in areas including financial services, mobile and sensor devices, click stream analysis etc. In this paper, we present an expressive formalism for specifying and combining complex events. For this language we provide both a clear declarative formal semantics as well as an effective event-driven execution model via a compilation strategy into Prolog. We also show how different consumption policies in CEP can be implemented in our formalism. We provide an implementation of the language, and present performance results of our running prototype which show that our approach is highly competitive to state-of-the-art systems.
Related papers
A Formal Framework for Complex Event Processing
A Formal Framework for Complex Event Processing, 2019
Complex Event Processing (CEP) has emerged as the unifying field for technologies that require processing and correlating distributed data sources in real-time. CEP finds applications in diverse domains, which has resulted in a large number of proposals for expressing and processing complex events. However, existing CEP languages lack from a clear semantics, making them hard to understand and generalize. Moreover, there are no general techniques for evaluating CEP query languages with clear performance guarantees. In this paper we embark on the task of giving a rigorous and efficient framework to CEP. We propose a formal language for specifying complex events, called CEL, that contains the main features used in the literature and has a denotational and compositional semantics. We also formalize the so-called selection strategies, which had only been presented as by-design extensions to existing frameworks. With a well-defined semantics at hand, we discuss how to efficiently process complex events by evaluating CEL formulas with unary filters. We start by studying the syntactical properties of CEL and propose rewriting optimization techniques for simplifying the evaluation of formulas. Then, we introduce a formal computational model for CEP, called complex event automata (CEA), and study how to compile CEL formulas with unary filters into CEA. Furthermore, we provide efficient algorithms for evaluating CEA over event streams using constant time per event followed by constant-delay enumeration of the results. Finally, we gather the main results of this work to present an efficient and declarative framework for CEP.
Foundations of Complex Event Processing
Cornell University, 2017
Complex Event Processing (CEP) has emerged as the unifying field for technologies that require processing and correlating distributed data sources in real-time. CEP finds applications in diverse domains, which has resulted in a large number of proposals for expressing and processing complex events. However, existing CEP languages lack from a clear semantics, making them hard to understand and generalize. Moreover, there are no general techniques for evaluating CEP query languages with clear performance guarantees. In this paper we embark on the task of giving a rigorous and efficient framework to CEP. We propose a formal language for specifying complex events, called CEL, that contains the main features used in the literature and has a de-notational and compositional semantics. We also formalize the so-called selection strategies, which had only been presented as by-design extensions to existing frameworks. With a well-defined semantics at hand, we study how to efficiently evaluate CEL for processing complex events in the case of unary filters. We start by studying the syntactical properties of CEL and propose rewriting optimization techniques for simplifying the evaluation of formulas. Then, we introduce a formal computational model for CEP, called complex event automata (CEA), and study how to compile CEL formulas into CEA. Furthermore, we provide efficient algorithms for evaluating CEA over event streams using constant time per event followed by constant-delay enumeration of the results. By gathering these results together, we propose a framework for efficiently evaluating CEL with unary filters. Finally, we show experimentally that this framework consistently out-performs the competition, and even over trivial queries can be orders of magnitude more efficient.
Blending Complex Event Processing with the RETE Algorithm
2008
Modern event-driven applications implement sophisticated and highly specialized algorithms for detecting and correlating events from event streams or clouds. The correlation logic and resulting actions are declaratively defined as EA (event action) rules. An EA language allows the definition of complex correlation rules with the help of logical, temporal, content-based and other operators. On the other hand, production rule systems provide a declarative means to express and compute CA (condition action) rules. CA rules define constraints. The most prominent scientific and commercial production rule systems rely on the RETE algorithm in order to efficiently compute CA rules. As the industry is on the way towards a loosely coupled service-oriented and eventdriven architecture, there is a high demand for an integrated solution of computing ECA (event condition action) rules. In this paper we give a survey of already finished as well as ongoing research in the field of combining event p...
A homogeneous reaction rule language for complex event processing
2007
Event-driven automation of reactive functionalities for complex event processing is an urgent need in today's distributed service-oriented architectures and Web-based event-driven environments. An important problem to be addressed is how to correctly and efficiently capture and process the event-based behavioral, reactive logic embodied in reaction rules, and combining this with other conditional decision logic embodied, e.g., in derivation rules. This paper elaborates a homogeneous integration approach that combines derivation rules, reaction rules and other rule types such as integrity constraints into the general framework of logic programming, the industrial-strength version of declarative programming. We describe syntax and semantics of the language, implement a distributed web-based middleware using enterprise service technologies and illustrate its adequacy in terms of expressiveness, efficiency and scalability through examples extracted from industrial use cases. The developed reaction rule language provides expressive features such as modular ID-based updates with support for external imports and self-updates of the intensional and extensional knowledge bases, transactions including integrity testing and roll-backs of update transition paths. It also supports distributed complex event processing, event messaging and event querying via efficient and scalable enterprise middleware technologies and event/action reasoning based on an event/action algebra implemented by an interval-based event calculus variant as a logic inference formalism.
Retractable Complex Event Processing and Stream Reasoning
Lecture Notes in Computer Science, 2011
Complex Event Processing (CEP) deals with processing of continuously arriving events with the goal of identifying meaningful patterns (complex events). In existing stream database approaches, CEP is manly concerned by temporal relations between events. This paper advocates for a knowledge-rich CEP with Stream Reasoning capabilities. Secondly, we address the problem of revision in event processing. Events are often assumed to be immutable and therefore always correct. Revision in event processing deals with the circumstance that certain events may be revoked. This necessitates to reconsider complex events which might have been computed based on the original, flawy history as soon as part of that history is corrected.
Standards for Complex Event Processing and Reaction Rules
Lecture Notes in Computer Science, 2011
1 In Rule-based Event Processing and Complex Event Processing (CEP), many areas of software development re-use existing technologies and methodologies, allowing their related standards to be reused. Other standards may be required to be developed to replace or augment existing standards. This paper introduces a general reference model for CEP standards with which existing and required standards will be discussed.
EVA: An event algebra supporting complex event specification
Information Systems, 2015
In applications such as digital libraries, stock tickers, traffic control, or supply chain management, composite events have been introduced to enable the capturing of rich situations. Composite events seem to follow common semantics. However, on closer inspection we observed that the evaluation semantics of events differs substantially from system to system. In this paper, we propose a parameterized event algebra that defines the detailed semantics of composite event operators. We introduce composite event operators that support explicit parameters for event selection and event consumption. These parameters define how to handle duplicates in both primitive and composite events. The event algebra EVA forms the foundation for a unifying reference language that allows for translation between arbitrary event composition languages using transformation rules. This translation, in turn, enables a mediator service that can federate heterogeneous eventbased systems. Our approach supports the seamless integration of event-based applications and changing event sources without the need to redefine user profiles. The event algebra is exemplified in the domain of transportation logistics.
Grand Challenge: Stateful Complex Event Detection on Event Streams using Parallelization of Event Stream Aggregations and Detection Tasks
Detection of stateful complex event patterns using parallel programming features is a challenging task because of statefulness of event detection operators. Parallelization of event detection tasks needs to be implemented in a way that keeps track of state changes by new arriving events. In this paper, we describe our implementation for a customized complex event detection engine by using Open Multi-Processing (OpenMP), a shared memory programming model. In our system event detection is implemented using Deterministic Finite Au-tomata (DFAs). We implemented a data stream aggregator that merges 4 given event streams into a sequence of C++ objects in a buffer used as source event stream for event detection in a next processing step. We describe implementation details and 3 architectural variations for stream aggregation and parallelized of event processing. We conducted performance experiments with each of the variations and report some of our experimental results. A comparison of our performance results shows that for event processing on single machine with multi cores and limited memory, using mutli-threads with shared buffer has better stream processing performance than an implementation with multi-processes and shared memory.
Ep-sparql: a unified language for event processing and stream reasoning
Proceedings of the 20th …, 2011
Streams of events appear increasingly today in various Web applications such as blogs, feeds, sensor data streams, geospatial information, on-line financial data, etc. Event Processing (EP) is concerned with timely detection of compound events within streams of simple events. State-of-the-art EP provides on-the-fly analysis of event streams, but cannot combine streams with background knowledge and cannot perform reasoning tasks. On the other hand, semantic tools can effectively handle background knowledge and perform reasoning thereon, but cannot deal with rapidly changing data provided by event streams.
Towards formal foundations of event queries and rules
Proc. Int. Workshop on Event-Driven Architecture, Processing and Systems, 2007
The field of complex event processing still lacks formal foundations. In particular, event queries require both declarative and operational semantics. We put forward for discussion a proposal towards formal foundations of event queries that aims at making well-known results from database queries applicable to event queries. Declarative semantics of event queries and rules are given as a model theory with accompanying fixpoint theory. Operational semantics are then obtained by translating the considered ...
Related papers
Siddhi: A Second Look at Complex Event Processing Architectures
people.apache.org
Today there are so much data being available from sources like sensors (RFIDs, Near Field Communication), web activities, transactions, social networks, etc. Making sense of this avalanche of data requires efficient and fast processing. Processing of high volume of events to derive higher-level information is a vital part of taking critical decisions, and Complex Event Processing (CEP) has become one of the most rapidly emerging fields in data processing. e-Science use-cases, business applications, financial trading applications, operational analytics applications and business activity monitoring applications are some use-cases that directly use CEP. This paper discusses different design decisions associated with CEP Engines, and proposes to improve CEP performance by using more stream processing style pipelines. Furthermore, the paper will discuss Siddhi, a CEP Engine that implements those suggestions. We present a performance study that exhibits that the resulting CEP Engine-Siddhi-has significantly improved performance. Primary contributions of this paper are performing a critical analysis of the CEP Engine design and identifying suggestions for improvements, implementing those improvements through Siddhi, and demonstrating the soundness of those suggestions through empirical evidence.
A Declarative Framework for Matching Iterative and Aggregative Patterns against Event Streams
Lecture Notes in Computer Science, 2011
Complex Event Processing as well as pattern matching against streams have become important in many areas including financial services, mobile devices, sensor-based applications, click stream analysis, real-time processing in Web 2.0 and 3.0 applications and so forth. However, there is a number of issues to be considered in order to enable effective pattern matching in modern applications. A language for describing patterns needs to feature a well-defined semantics, it needs be rich enough to express important classes of complex patterns such as iterative and aggregative patterns, and the language execution model needs to be efficient since event processing is a real-time processing. In this paper, we present an event processing framework which includes an expressive language featuring a precise semantics and a corresponding execution model, expressive enough to represent iterative and aggregative patterns. Our approach is based on a logic, hence we analyse deductive capabilities of such an event processing framework. Finally, we provide an open source implementation and present experimental results of our running system.
A CEP Babelfish: Languages for Complex Event Processing and Querying Surveyed
Studies in Computational Intelligence, 2011
Complex Event Processing (CEP) denotes algorithmic methods for making sense of events by deriving higher-level knowledge, or complex events, from lower-level events in a timely fashion and permanently. At the core of CEP are queries continuously monitoring the incoming stream of "simple" events and recognizing "complex" events from these simple events. Event queries monitoring incoming streams of simple events serve as specification of situations that manifest themselves as certain combinations of simple events occurring, or not occurring, over time and that cannot be detected solely from one or parts of the single events involved. Special purpose Event Query Languages (EQLs) have been developed for the expression of the complex events in a convenient, concise, effective and maintainable manner. This chapter identifies five language styles for CEP, namely composition operators, data stream query languages, production rules, timed state machines, and logic languages, describes their main traits, illustrates them on a sensor network use case and discusses suitable application areas of each language style.
NEEL: The Nested Complex Event Language for Real-Time Event Analytics
Complex event processing (CEP) over event streams has become increasingly important for real-time applications ranging from health care, supply chain management to business intelligence. These monitoring applications submit complex event queries to track sequences of events that match a given pattern. As these systems mature the need for increasingly complex nested sequence query support arises, while the state-of-art CEP systems mostly support the execution of only flat sequence queries. In this paper, we introduce our nested CEP query language NEEL for expressing nested queries composed of sequence, negation, AND and OR operators. Thereafter, we also define its formal semantics. Subtle issues with negation and predicates within the nested sequence context are discussed. An E-Analytics system for processing nested CEP queries expressed in the NEEL language has been developed. Lastly, we demonstrate the utility of this technology by describing a case study of applying this technology to a real-world application in health care.
Stream reasoning and complex event processing in ETALIS
Addressing dynamics and notifications in the Semantic Web realm has recently become an important area of research. Run time data is continuously generated by multiple social networks, sensor networks, various on-line services and so forth. How to get advantage of this continuously arriving data (events) remains a challenge–that is, how to integrate heterogeneous event streams, combine them with background knowledge (eg, an ontology), and perform event processing and stream reasoning.
A resource-efficient event detection algebra
2003
Event detection is an important aspect of many application t ypes, ranging from active databases over digital libraries and stock market ag nts, to reactive embedded systems. To allow systems to react to complex events pattern s rather than to simple primitive events, an event algebra can be used. We have developed an event algebra with formal semantics and a number of useful algebraic properties. An equivalent operational semantic s has been developed, and we have identified an important subset of event expressions f r which the detection mechanism can be implemented with constant memory. 1 Background and motivation A wide range of applications, including active databases, t raffic monitoring systems and rule based embedded systems, are based on the detection of ev ents that trigger an appropriate response from the system. Events can be simple, e.g., sampled directly from the environment or occurring within the system, but it is often n ecessary to react to more sophisticated situa...
Towards a Logic for Inferring Properties of Event Streams
Lecture Notes in Computer Science, 2016
We outline the background, motivation, and requirements of an approach to create abstractions of event streams, which are timetagged sequences of events generated by an executing software system. Our work is motivated by the need to process event streams with millions of events that are generated by a spacecraft, that must be processed quickly after they are received on the ground. Our approach involves building a tool that adds hierarchical labels to a received event stream. The labels add contextual information to the event stream, and thus make it easier to build tools for visualizing and analyzing telemetry. We describe a notation for writing hierarchical labeling rules; the notation is based on a modification of Allen Logic, augmented with rule-definitions and features for referring to data in data parameterized events. We illustrate our notation and its use with an example.
Formalizing Complex Event Processing Systems in Maude
IEEE Access
Complex event processing (CEP) is a cutting-edge technology for analyzing and correlating streams of information about events that happen in a system, and deriving conclusions from them. CEP permits defining complex events based on the events produced by the incoming sources, to identify complex meaningful circumstances and to respond to them as quickly as possible. Such event types and patterns are defined using event processing languages. However, as the complexity of CEP programs grows, they become difficult to understand and to prove correct. This paper proposes a formal framework for the specification of CEP applications, using rewriting logic and Maude, to allow developers to formally analyze and prove properties of their CEP programs. Several case studies are presented to illustrate the approach, as well as a discussion on the benefits of using Maude and its toolkit for modeling and analyzing CEP systems. INDEX TERMS Formal modeling, complex event processing, event processing language, rewriting logic, Maude. LOLI BURGUEÑO received the B.Sc. degree in computer science and engineering from the Universidad de Málaga, Spain, in 2011, and received the master's degree in software engineering and artificial intelligence in 2012 and the Ph.D. degree (Hons.) in 2016. She is currently a Post-Doctoral Researcher and a Lecturer with the Universidad de Málaga. Her research interests include modeldriven engineering; in concrete, she is involved in testing model transformations, the parallelization of the execution of model transformations, and the modeling of uncertainty in software models for its use in the Industry 4.0. Further information can be found at http://www.lcc.uma.es/~loli.
Active complex event processing over event streams
Proceedings of the VLDB Endowment, 2011
State-of-the-art Complex Event Processing technology (CEP), while effective for pattern query execution, is limited in its capability of reacting to opportunities and risks detected by pattern queries. Especially reactions that affect the query results in turn have not been addressed in the literature. We propose to tackle these unsolved problems by embedding active rule support within the CEP engine, henceforth called Active CEP (ACEP). Active rules in ACEP allow us to specify a pattern query's dynamic condition and real-time actions. The technical challenge is to handle interactions between queries and reactions to queries in the high-volume stream execution. We hence introduce a novel stream-oriented transactional model along with a family of stream transaction scheduling algorithms that ensure the correctness of concurrent stream execution. We demonstrate the power of ACEP technology by applying it to the development of a healthcare system being deployed in UMass Medical School hospital. Through extensive performance experiments using real data streams, we show that our unique Low-Water-Mark stream transaction scheduler, customized for streaming environments, successfully achieves near-realtime system responsiveness and gives orders-of-magnitude better throughput than our alternative schedulers.
Inferring event stream abstractions
Formal Methods in System Design, 2018
We propose a formalism for specifying event stream abstractions for use in spacecraft telemetry processing. Our work is motivated by the need to quickly process streams with millions of events generated e.g. by the Curiosity rover on Mars. The approach builds a hierarchy of event abstractions for telemetry visualization and querying to aid human comprehension. Such abstractions can also be used as input to other runtime verification tools. Our notation is inspired by Allen's Temporal Logic, and provides a rule-based declarative way to express event abstractions. We present an algorithm for applying specifications to an event stream and explore modifications to improve the algorithm's asymptotic complexity. The system is implemented in both Scala and C, with the specification language implemented as internal as well as external DSLs. We illustrate the solution with several examples, a performance evaluation, and a real telemetry analysis scenario.
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.