A METHOD FOR DISTRIBUTED CONCEPT DRIFT DETECTION
DOI:
https://doi.org/10.14529/cmse140110Keywords:
concept drift, data mining, distributed computations, iterative MapReduceAbstract
The paper introduces a method for distributed concept drift detection for data mining algorithms. Concept drift is understood as any unpredictable alteration in input data. There is an algorithm implementation proposed, based on MapReduce distributed computing technology. Proposed algorithm meant for concept drift detection in streaming data in online fashion. In order to provide iterative Map and Reduce phases a MapReduce framework is introduced. The algorithm is able to automatically detect input data alteration, which demands model parameters change and switching a new model online.
References
Andrzejak, A. Parallel Concept Drift Detection with Online Map-Reduce / A. Andrzejak, J.B. Gomes // International Workshop on Knowledge Discovery (KDCloud-2012). Dec. 2012. — P. 402–407.
Baena-Garcıa, M. Early drift detection method / M. Baena-Garcıa, J. Campo-Avila, R. Fidalgo, A. Bifet, R. Gavalda, R. Morales-Bueno // The 4th International Workshop on Knowledge Discovery from Data Streams. Sep. 2006. —P. 77–86.
Bose, J.-H. Beyond online aggregation: Parallel and incremental data mining with online mapreduce / J.-H. Bose, A. Andrzejak, M. Hogqvist // ACM Workshop on Massive Data Analytics over the Cloud (MDAC 2010). Apr. 2010.
Doulkeridis, C. A survey of large-scale analytical query processing in MapReduce / C. Doulkeridis, K. Nørvåg // The VLDB Journal. 2013. —P. 1–26.
Gama, J. Learning with drift detection / J. Gama, P. Medas, G. Castillo, P. Rodrigues // Advances in Artificial Intelligence. Nov. 2004. —Vol. 3171. — P. 286–295.
Sobhani, P. New Drift Detection Method for Data Streams / P. Sobhani, H. Beigy // Adaptive and Intelligent Systems. Sep. 2011. — Vol.6943. — P. 88–97.
