POSSIBILITIES OF USING NEURAL NETWORK INCREMENTAL LEARNING
DOI:
https://doi.org/10.14529/ctcr210402Keywords:
neural networks, incremental learning, machine learning, catastrophic forgettingAbstract
The present time is characterized by unprecedented growth in the volume of information flows. Information processing underlies the solution of many practical problems. The intelligent information systems applications range is extremely extensive: from managing continuous technological processes in real-time to solving commercial and administrative problems. Intelligent information systems should have such a main property, as the ability to quickly process dynamical incoming data in real-time. Also, intelligent information systems should be extracting knowledge from previously solved problems. Incremental neural network training has become one of the topical issues in machine learning in recent years. Compared to traditional machine learning, incremental learning allows assimilating new knowledge that comes in gradually and preserving old knowledge gained from previous tasks. Such training should be useful in intelligent systems where data flows dynamically. Aim. Consider the concepts, problems, and methods of incremental neural network training, as well as assess the possibility of using it in intelligent systems development. Materials and methods. The idea of incremental learning, obtained in the analysis of a person's learning during his life, is considered. The terms used in the literature to describe incremental learning are presented. The obstacles that arise in achieving the goal of incremental learning are described. A description of three scenarios of incremental learning, among which class-incremental learning is distinguished, is given. An analysis of the methods of incremental learning, grouped into a family of techniques by the solution of the catastrophic forgetting problem, is given. The possibilities offered by incremental learning versus traditional machine learning are presented. Results. The article attempts to assess the current state and the possibility of using incremental neural network learning, to identify differences from traditional machine learning. Conclusion. Incremental learning is useful for future intelligent systems, as it allows to maintain existing knowledge in the process of updating, avoid learning from scratch, and dynamically adjust the model's ability to learn according to new data available.References
Ade R.R., Deshmukh P.R. Methods for Incremental Learning: A Survey. International Journal of Data Mining & Knowledge Management Process, 2013, vol. 3, pp. 119–125. DOI: 10.5121/ijdkp.2013.3408
Luo Y., Yin L., Bai W., Mao K. An Appraisal of Incremental Learning Methods. Entropy, 2020, vol. 22, pp. 1–27. DOI: 10.3390/e22111190
Yacim J.A., Boshoff D.G.B. Impact of Artificial Neural Networks Training Algorithms on Accurate Prediction of Property Values. Journal of Real Estate Research, 2018, vol. 40, pp. 375–418. DOI: 10.1080/10835547.2018.12091505
Gu J., Wang Z., Kuen J., Ma L., Shahroudy A., Shuai B., Liu T., Wang X., Wang G., Cai J., Chen T. Recent Advances in Convolutional Neural Networks. Pattern Recognition, 2018, vol. 77,
pp. 354–377. DOI: 10.1016/j.patcog.2017.10.013
Baptista D., Morgado-Dias F. A Survey of Artificial Neural Network Training Tools. Neural Computing and Applications, 2013, vol. 23, pp. 609–615. DOI: 10.1007/s00521-013-1408-9
Ganguly S., Chatterjee A., Bhoumik D., Majumdar R. An Empirical Study of Incremental Learning in Neural Network with Noisy Training Set // Lecture Notes in Networks and Systems, 2021, pp. 72–77.
Wang J., Wang H., Chen Y., Liu C. A Constructive Algorithm for Unsupervised Learning with Incremental Neural Network. Journal of Applied Research and Technology, 2015, vol. 13, pp. 188–196. DOI: 10.1016/j.jart.2015.06.017
Gao F., Mei J., Sun J., Wang J., Yang E., Hussain A. A Novel Classification Algorithm Based
on Incremental Semi-Supervised Support Vector Machine. PLoS One, 2015, vol. 10. DOI: 10.1371/journal.pone.0135709
Dai X., Yin H., Jha N.K. Incremental Learning Using a Grow-and-Prune Paradigm with Efficient Neural Networks. IEEE Trans. Emerg. Top. Comput, 2020.
Geng X., Smith-Miles K. Incremental Learning. Encyclopedia of Biometrics, 2015. DOI: 10.1007/978-0-387-73003-5_304
Schlimmer J.C., Fisher D. A Case Study of Incremental Concept Induction. Am. Artif. Intel. Proc., 1986, pp. 496–501.
Aha D.W., Kibler D., Albert M.K. Instance-Based Learning Algorithms. Mach. Learn., 1991, vol. 6, pp. 37–66.
Sarwar S.S., Ankit A., Roy K. Incremental Learning in Deep Convolutional Neural Networks Using Partial Network Sharing. IEEE Access, 2019, vol. 8, pp. 4615–4628. DOI: 10.1109/ACCESS.2019.2963056
Dmitrienko V.D., Zakovorotniy A.Y. Architecture and Algorithms of Neural Networks Hamming and Hebb, Capable Learn and Identify New Information. Radio Electron. Comput. Sci. Control, 2014. DOI: 10.15588/1607-3274-2014-2-15
Coop R., Mishtal A., Arel I. Ensemble Learning in Fixed Expansion Layer Networks for Mitiga¬ting Catastrophic Forgetting. IEEE Trans. Neural Networks Learn. Syst., 2013, vol. 24, pp. 1623–1634. DOI: 10.1109/TNNLS.2013.2264952
Li H., Dong W., Hu B.-G. Incremental Concept Learning via Online Generative Memory Recall. IEEE Trans. Neural Networks Learn. Syst., 2020, vol. 32, pp. 3206–3216.
Lopez-Paz D., Ranzato M. Gradient Episodic Memory for Continual Learning. Advances in Neural Information Processing Systems. Neural Information Processing Systems Foundation, 2017,
pp. 6468–6477.
Mallya A., Davis D., Lazebnik S. Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2018, pp. 72–88. DOI: 10.1007/978-3-030-01225-0_5
Mallya A., Lazebnik S. PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, 2018. pp. 7765–7773. DOI: 10.1109/CVPR.2018.00810
Li P., Chen Z., Yang L.T., Gao J., Zhang Q., Deen M.J. An Incremental Deep Convolutional Computation Model for Feature Learning on Industrial Big Data. IEEE Trans. Ind. Informatics, 2019, vol. 15, pp. 1341–1349. DOI: 10.1109/TII.2018.2871084
Paik I., Oh S., Kwak T., Kim I. Overcoming Catastrophic Forgetting by Neuron-Level Plasticity Control. Proc. AAAI Conf. Artif. Intell, 2020, vol. 34, pp. 5339–5346. DOI: 10.1609/AAAI.V34I04.5981
Masse N.Y.; Grant G.D.; Freedman D.J. Alleviating Catastrophic Forgetting Using Context-Dependent Gating and Synaptic Stabilization. Proceedings of the National Academy of Sciences of
the United States of America, 2018, vol. 115. DOI: 10.1073/pnas.1803839115
Gaurav A., Abdelzad V., Vernekar S., Czarnecki K., Lee J., Sedwards S. Simple Continual Learning Strategies for Safer Classifers. CEUR Workshop Proceedings, 2020, vol. 2560, pp. 96–104.
Zenke F., Poole B., Ganguli S. Continual Learning Through Synaptic Intelligence. 34th International Conference on Machine Learning, ICML 2017, 2017, vol. 8, pp. 6072–6082.
Li Z., Hoiem D. Learning without Forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018. vol. 40, pp. 2935–2947. DOI: 10.1109/TPAMI.2017.2773081
Xiang Y., Miao Y., Chen J., Xuan Q. Efficient Incremental Learning Using Dynamic Correction Vector. IEEE Access, 2020, vol. 8, pp. 23090–23099. DOI: 10.1109/ACCESS.2019.2963461
Zhao B., Xiao X., Gan G., Zhang B., Xia S. Maintaining Discrimination and Fairness in Class Incremental Learning. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit, 2020, pp. 13205–13214. DOI: 10.1109/CVPR42600.2020.01322
Javed K., Shafait F. Revisiting Distillation and Incremental Classifier Learning. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), 2018, vol. 11366 LNCS, pp. 3–17. DOI: 10.1007/978-3-030-20876-9_1
Stojanov S., Mishra S., Thai N., Dhanda N., Humayun A., Yu C., Smith L.B., Rehg J.M. Incremental Object Learning from Contiguous Views. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019, pp. 8777–8786. DOI: 10.1109/CVPR.2019.00898
Xiong W., Wang Y., Cheng L. Fisher Discriminant Analysis Random Forest for Online Class Incremental Learning. 2018 IEEE Intl Conf Parallel Distrib. Process. with Appl. Ubiquitous Comput. Commun. Big Data Cloud Comput. Soc. Comput. Networking, Sustain. Comput. Commun, 2018,
pp. 597–604. DOI: 10.1109/BDCloud.2018.00093
Tasar O., Tarabalka Y., Alliez P. Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens, 2019, vol. 12, pp. 3524–3537. DOI: 10.1109/JSTARS.2019.2925416
Nakano S., Hattori M. Characteristics of Contrastive Hebbian Learning with Pseudorehearsal for Multilayer Neural Networks on Reduction of Catastrophic Forgetting. International Journal of Computational Intelligence Studies, 2018, vol. 7, p. 289. DOI: 10.1504/IJCISTUDIES.2018.096184
Rebuffi S.A., Kolesnikov A., Sperl G., Lampert C. iCaRL: Incremental Classifier and Representation Learning. Proceedings – 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017, pp. 5533–5542.
Guo L., Xie G., Xu X., Ren J. Exemplar-Supported Representation for Effective Class-Incremen¬tal Learning. IEEE Access, 2020, vol. 8, pp. 51276–51284. DOI: 10.1109/ACCESS.2020.2980386
Riemer M., Klinger T., Bouneffouf D., Franceschini M. Scalable Recollections for Continual Lifelong Learning. 33rd AAAI Conference on Artificial Intelligence, AAAI 2019, 31st Innovative Applications of Artificial Intelligence Conference, IAAI 2019 and the 9th AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, 2019, pp. 1352–1359. DOI: 10.1609/aaai.v33i01.33011352
Kim B., Lee J., Kim K., Kim S., Kim J. Collaborative Method for Incremental Learning on Classification and Generation. Proceedings – International Conference on Image Processing, ICIP, 2019, pp. 390–394.
Castro F.M., Marín-Jiménez M.J., Guil N., Schmid C., Alahari K. End-to-End Incremental Learning. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), 2018, vol. 11216 LNCS, pp. 241–257. DOI: 10.1007/978-3-030-01258-8_15
Chaudhry A., Marc'Aurelio R., Rohrbach M., Elhoseiny M. Efficient Lifelong learning with
A-GEM. 7th International Conference on Learning Representations, ICLR 2019, 2019.
Schlimmer J.C., Granger R.H. Incremental Learning from Noisy Data. Mach. Learn., 1986, vol. 1, pp. 317–354. DOI: 10.1023/A:1022810614389
Syed N.A., Liu H., Sung K.K. Handling Concept Drifts in Incremental Learning with Support Vector Machines. Association for Computing Machinery (ACM), 1999, pp. 317–321. DOI: 10.1145/312129.312267
Díaz-Rodríguez N., Lomonaco V., Filliat D., Maltoni D. Don’t Forget, There is More than Forgetting: New Metrics for Continual Learning, 2018.
She Q., Feng F., Hao X., Yang Q., Lan C., Lomonaco V., Shi X., Wang Z., Guo Y., Zhang Y., Qiao F., Chan R.H.M. OpenLORIS-Object: A Robotic Vision Dataset and Benchmark for Lifelong Deep Learning. Proceedings – IEEE International Conference on Robotics and Automation. Institute of Electrical and Electronics Engineers Inc., 2020, pp. 4767–4773. DOI: 10.1109/ICRA40945.2020.9196887
