TY  - JOUR
T1  - Big Data-based Log Collection and Analysis in IoT Environments
AU - Shin, Dong Jin AU - Eun, Jong Min AU - Lee, Ho Geun AU - Lee, Myoung Gyun AU - Park, Jeong Min AU - Kim, Jeong Joon 
JO  - Journal of Engineering and Applied Sciences
VL  - 13
IS  - 5
SP  - 1064
EP  - 1072
PY  - 2018
DA  - 2001/08/19
SN  - 1816-949x
DO  - jeasci.2018.1064.1072
UR  - https://makhillpublications.co/view-article.php?doi=jeasci.2018.1064.1072
KW  - Big data
KW  -IoT
KW  -sensor data
KW  -informal data
KW  -analysis
KW  -Korea
AB  - Recently, various new technologies such as Ai, IoT, cloud and big data are being developed in line
with the 4th industrial revolution. As the amount of various sensor data based on IoT is increased, many
techniques are required to collect and analyze the data. Therefore, we want to present the analysis results
through processing of big data. In IoT, sensor data can be various kinds and quantities such as ultrasonic
waves, infrared rays, cameras and vibrations. This type of informal data is difficult to obtain the desired
analytical results when applied to a general analysis program. In this study, we implemented a system that
processes informal data by collecting, storing, processing and analyzing data. We used Raspberry Pi in IoT and
generated web server log data. The generated web server log data is collected in real time using flume, a
collection solution of big data. Storage is stored in the HDFS of the hadoop solution and the unwanted
properties are refined through processing solutions Hive and Pig. At the end of the final refine process, we
analyzed the files with R programming and Spark.
ER  -