Performance improvement of distributed processing in binary bernoulli sampling

초록

This paper addresses the problem of distributed processing in binary Bernoulli sampling (BBS in short). BBS is a stochastic sampling for the multi-source stream environment, and we need to use distributed processing of BBS to process large volumes of data streams generated from multiple input sources. Accordingly, a recent work proposed a distributed BBS model based on Apache Storm having the multiple coordinator structures. However, this technique has limitations in improving performance due to the coordinator waiting problem. In this paper, we solve the coordinator waiting problem by introducing multi-distribution and distributor separation structures. The multi-distribution structure minimizes the waiting time by participating multiple coordinators rather than only one in the distribution. The distributor separation structure maximizes processing performance by separating the distribution function from the coordinator. To experimentally evaluate the performance improvement, we apply the improved structures to Storm-based distributed BBS. Experimental results show that the multi-distribution and distributor separation structures improve the performance up to 90 times compared to the single distribution structure. ICIC International ©2021

키워드