APACHE STORM-BASED DISTRIBUTED SAMPLING LIBRARY FOR DATA STREAMS

초록

Recently, large data streams are rapidly generated in various fields such as Internet of Things (IoT) and social network services (SNS). Analyzing all such data streams is very inefficient, and we generally pre-process the data to reduce its huge value. In this paper, we first present a distributed processing-based sampling library for effective use of large data streams. We then apply and implement the proposed library to Apache Storm, a real-time distributed processing system. To validate the efficiency of the sampling library, we extract samples from real data streams and compare the graphs of the extracted samples with the original population. Experimental results show that the sample data using the proposed library presents a similar pattern reflecting the characteristics of the original data well. We believe that our library is an excellent result for easily sampling large data streams in a distributed environment. ©2023 ISSN 1881-803X.

키워드