A storm-based distributed framework for binary bernoulli sampling

초록

In recent years, a large volume of data streams have been rapidly produced in many applications such as SNS (social network service), smart devices, and IoT (in-ternet of things), and accordingly, there have been a lot of needs for sampling techniques on those rapid data streams. In this paper, we deal with the performance improvement of binary Bernoulli sampling which performs sampling in the multiple-input environment. Previous binary Bernoulli sampling, however, has a significant performance degradation problem if the number of input sites is large or the data stream explodes. To solve this problem, we first propose a distributed processing model for improving the performance and scalability of binary Bernoulli sampling. We then implement this model on Apache Storm, a distributed real-time computation system. We also compare its performance with the existing binary Bernoulli sampling on the stream environment. Experimental results show that the proposed Strom-based binary Bernoulli sampling largely improves the performance compared with the existing one by up to 1.8 times. To our best knowledge, this is the first significant attempt that solves the performance degradation problem of sampling algorithms by extending a single processing system to a distributed processing system. © 2017, ICIC Express Letters Office. All rights reserved.

키워드