An Esper-based filtering system for real-time data streams

초록

In this paper, we deal with the filtering problem of data streams. The data stream is continuously generated, and its size is huge. In order to process and analyze the data stream in real time, we need to sufficiently remove unnecessary data through filtering. However, existing filtering algorithms can be applied to a single data format only, and it is very difficult to apply them to a variety and complex stream environments. To solve this problem, we propose a filtering system that can choose various filtering algorithms according to the stream format. The proposed system is based on Esper, which is a representative open source data stream management system (DSMS) for real-time filtering support. Using Esper we can filter data streams in real time anywhere, anytime, based on a Web-based client-server model. Our system supports real-time stream and/or bulk stream as the input data. In addition, we implement typical filtering algorithms including query filtering, Bloom filtering, and Bayesian filtering to operate in real time. Through the real implementation of the proposed filtering system, we show that the user can extract only meaningful data more accurately and efficiently by exploiting various filtering algorithms. © 2017 ICIC International.

키워드