Towards a loss function for training neural network models of time series imputation
DOI:
https://doi.org/10.14529/cmse240404Keywords:
time series, imputation of missing values, neural networks, loss function, PyTorchAbstract
In the article, we touch upon the problem of choosing a loss function for training neural network models for imputation of missing values of multidimensional time series and introduce a novel loss function called MPDE (Mean Profile Distance Error). The MPDE function for real and reconstructed m-length subsequences is calculated as the average of the distances between all pairs of ℓ-length sliding windows of these subsequences, where ℓ ⩽ m and above windows have the same starting points. The distance between two windows is a modification of the MPdist (matrix profile distance) similarity measure and is defined as the weighted sum of the Euclidean and z-normalized Euclidean distances between these windows. The above weights are taken from the range [0,1] and are the hyper-parameters of the loss function. When training a neural network model, MPDE allows taking into account the behavioral similarity of the compared subsequences through the presence of similar windows in them, regardless of the relative locations of these windows. Since MPDE has a high computational complexity, we implement a parallel algorithm for its calculation on a GPU to incorporate MPDE into deep learning frameworks. The algorithm is implemented using the PyTorch framework, where MPDE is represented as a sequence of automatically parallelizable operations with multidimensional tensors. Experiments over multidimensional time series from various subject domains showed that in 78% of cases state-of-the-art neural network models achieve their highest imputation accuracy (in terms of the RMSE metric) when using the proposed loss function; at the same time, the above models demonstrate imputation accuracy 40% higher than the average value achieved when using other loss functions.References
Majumdar S., Laha A.K. Clustering and classification of time series using topological data analysis with applications to finance. Expert Syst. Appl. 2020. Vol. 162. P. 113868. DOI: 10.1016/j.eswa.2020.113868.
Lara-Benítez P., Carranza-García M., Luna-Romera J.M., Riquelme J.C. Temporal convolutional networks applied to energy-related time series forecasting. Applied sciences. 2020.Vol. 10, no. 7. P. 2322. DOI: 10.3390/app10072322.
Gratius N., Wang Z., Hwang M.Y., et al. Digital Twin Technologies for Autonomous Environmental Control and Life Support Systems. J. Aerosp. Inf. Syst. 2024. Vol. 21, no. 4. P. 332–347. DOI: 10.2514/1.I011320.
Zhou Z., Tang W., Li M., et al. A Novel Hybrid Intelligent SOPDEL Model with Comprehensive Data Preprocessing for Long-Time-Series Climate Prediction. Remote. Sens. 2023. Vol. 15, no. 7. P. 1951. DOI: 10.3390/RS15071951.
Kazijevs M., Samad M.D. Deep imputation of missing values in time series health data: A review with benchmarking. J. Biomed. Informatics. 2023. Vol. 144. P. 104440. DOI: 10.1016/J.JBI.2023.104440.
Fang C., Wang C. Time Series Data Imputation: A Survey on Deep Learning Approaches. CoRR. 2020. Vol. abs/2011.11347. arXiv:2011.11347. URL: https://arxiv.org/abs/2011.11347.
Cao W., Wang D., Li J., et al. BRITS: Bidirectional Recurrent Imputation for Time Series. Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada / ed. by S. Bengio, H.M. Wallach, H. Larochelle, et al. 2018. P. 6776–6786. URL: https://proceedings.neurips.cc/paper/2018/hash/734e6bfcd358e25ac1db0a4241b95651-Abstract.html.
Yoon J., Zame W.R., Schaar M. van der. Estimating Missing Data in Temporal Data Streams Using Multi-Directional Recurrent Neural Networks. IEEE Trans. Biomed. Eng.2019. Vol. 66, no. 5. P. 1477–1490. DOI: 10.1109/TBME.2018.2874712.
Fortuin V., Baranchuk D., Rätsch G., Mandt S. GP-VAE: Deep Probabilistic Time Series Imputation. The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy]. Vol. 108 / ed. by S. Chiappa, R. Calandra. PMLR, 2020. P. 1651–1661. Proceedings of Machine Learning Research. URL: http://proceedings.mlr.press/v108/fortuin20a.html.
Du W., Côté D., Liu Y. SAITS: Self-attention-based imputation for time series. Expert Syst. Appl. 2023. Vol. 219. P. 119619. DOI: 10.1016/J.ESWA.2023.119619.
Oh E., Kim T., Ji Y., Khyalia S. STING: Self-attention based Time-series Imputation Networks using GAN. 2021. DOI: 10.1109/ICDM51629.2021.00155.
Kaya M., Bilge H.S. Deep Metric Learning: A Survey. Symmetry. 2019. Vol. 11, no. 9. P. 1066. DOI: 10.3390/SYM11091066.
Wang Q., Ma Y., Zhao K., Tian Y. A comprehensive survey of loss functions in machine learning. Annals of Data Science. 2022. Vol. 9. P. 187–212. DOI: 10.1007/s40745-020-00253-5.
Ciampiconi L., Elwood A., Leonardi M., et al. A survey and taxonomy of loss functions in machine learning. CoRR. 2023. Vol. abs/2301.05579. DOI: 10.48550/ARXIV.2301.05579. arXiv:2301.05579.
Netrapalli P. Stochastic gradient descent and its variants in machine learning. Journal of the Indian Institute of Science. 2019. Vol. 99, no. 2. P. 201–213. DOI: 10.1007/s41745-019-0098-4.
Xu J., Ren X., Lin J., Sun X. Diversity-Promoting GAN: A Cross-Entropy Based Generative Adversarial Network for Diversified Text Generation. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31–November 4, 2018 / ed. by E. Riloff, D. Chiang, J. Hockenmaier, J. Tsujii. Association for Computational Linguistics, 2018. P. 3940–3949. DOI: 10.18653/V1/D18-1428.
Chang Y., Wang X., Wang J., et al. A Survey on Evaluation of Large Language Models. ACM Trans. Intell. Syst. Technol. 2024. Vol. 15, no. 39. P. 1–45. DOI: 10.1145/3641289.
Chen W., Huang H., Peng S., et al. YOLO-face: a real-time face detector. Vis. Comput.2021. Vol. 37, no. 4. P. 805–813. DOI: 10.1007/S00371-020-01831-7.
Paszke A., Gross S., Massa F., et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada / ed. by H.M. Wallach, H. Larochelle, A. Beygelzimer, et al. 2019. P. 8024--8035. URL: https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html.
Jadon A., Patil A., Jadon S. A Comprehensive Survey of Regression Based Loss Functions for Time Series Forecasting. International Conference on Data Management, Analytics & Innovation. Springer. 2024. P. 117–147. DOI: 10.1007/978-981-97-3245-6_9.
Qi J., Du J., Siniscalchi S.M., et al. On Mean Absolute Error for Deep Neural Network Based Vector-to-Vector Regression. IEEE Signal Process. Lett. 2020. Vol. 27. P. 1485–1489. DOI: 10.1109/LSP.2020.3016837.
Chen X., Liu W., Mao X., Yang Z. Distributed High-dimensional Regression Under a Quantile Loss Function. J. Mach. Learn. Res. 2020. Vol. 21, no. 182 P. 1–43. URL: http://jmlr.org/papers/v21/20-297.html.
Saleh R.A., Saleh A.K.M.E. Statistical Properties of the log-cosh Loss Function Used in Machine Learning. CoRR. 2022. Vol. abs/2208.04564. DOI: 10.48550/ARXIV.2208.04564. arXiv:2208.04564.
Sun Q., Zhou W.-X., Fan J. Adaptive Huber regression. Journal of the American Statistical Association. 2020. Vol. 115, no. 529. P. 254–265. DOI: 10.1080/01621459.2018.1543124.
Cuturi M., Blondel M. Soft-DTW: a Differentiable Loss Function for Time-Series. Proceedings of the 34th International Conference on Machine Learning. Vol. 70 / ed. by D. Precup, Y.W. Teh. PMLR, 6-11 Aug/2017. P. 894–903. Proceedings of Machine Learning Research. URL: https://proceedings.mlr.press/v70/cuturi17a.html.
Berndt D.J., Clifford J. Using Dynamic Time Warping to find patterns in time series. KDD Workshop. 1994. P. 359–370. URL: https://cdn.aaai.org/Workshops/1994/WS-94-03/WS94-03-031.pdf.
Gharghabi S., Imani S., Bagnall A.J., et al. Matrix Profile XII: MPdist: A Novel Time Series Distance Measure to Allow Data Mining in More Challenging Scenarios. IEEE International Conference on Data Mining, ICDM 2018, Singapore, November 17-20, 2018. IEEE Computer Society, 2018. P. 965–970. DOI: 10.1109/ICDM.2018.00119.
Zhuang J., Tang T., Ding Y., et al. AdaBelief Optimizer: Adapting Step sizes by the Belief in Observed Gradients. Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual / ed. by H. Larochelle, M. Ranzato, R. Hadsell, et al. 2020. URL: https://proceedings.neurips.cc/paper/2020/hash/d9d4f495e875a2e075a1a4a6e1b9770f-Abstract.html.
Baydin A.G., Pearlmutter B.A., Radul A.A., Siskind J.M. Automatic Differentiation in Machine Learning: a Survey. J. Mach. Learn. Res. 2017. Vol. 18, no. 153. P. 1–153. URL: https://jmlr.org/papers/v18/17-468.html.
Bilenko R.V., Dolganina N.Y., Ivanova E.V., Rekachinsky A.I. High-performance Computing Resources of South Ural State University. Bulletin of the South Ural State University. Series: Computational Mathematics and Software Engineering. 2022. Vol. 11, no. 1. P. 15–30. DOI: 10.14529/cmse220102.
Trindade A. Electricity Load Diagrams 2011–2014. 2015. DOI: 10.24432/C58C86. UCI Machine Learning Repository.
Laña I., Olabarrieta I., Vélez M., Del Ser J. On the imputation of missing data for road traffic forecasting: New insights and novel techniques. Transportation Research Part C: Emerging Technologies. 2018. Vol. 90. P. 18–33. DOI: 10.1016/j.trc.2018.02.021.
Sheppy M., Beach A., Pless S. NREL RSF Measured Data 2011. 11/2014. Accessed: 2023-09-03 DOI: 10.25984/1845288.
Reiss A. PAMAP2 Physical Activity Monitoring. 2012. Accessed: 2023-09-03. DOI: 10.24432/C5NW2H. UCI Machine Learning Repository.
Yurtin A.A. Imputation of Multivariate Time Series Based on the Behavioral Patterns and Autoencoders. Bulletin of the South Ural State University. Series: Computational Mathematics and Software Engineering. 2024. Vol. 13, no. 2. P. 39–55. DOI: 10.14529/cmse240203.
BundesAmt Für Umwelt – Swiss Federal Office for the Environment. Accessed: 2023-09-03. https://www.hydrodaten.admin.ch/.
Lozano A.C., Li H., Niculescu-Mizil A., et al. Spatial-temporal causal modeling for climate change attribution. Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Paris, France, June 28 - July 1, 2009 / ed. by J.F. Elder, F. Fogelman-Soulié, P.A. Flach, M.J. Zaki. ACM, 2009. P. 587–596. DOI: 10.1145/1557019.1557086.
MeteoSwiss: Federal Office of Meteorology and Climatology. 2023. Accessed: 2023-09-03. https://www.meteoswiss.admin.ch/services-and-publications/service/open-government-data.html.
Weather Station Saaleaue, Max Planck Institute for Biogeochemistry, Germany. Accessed: 2023-09-03. https://www.bgc-jena.mpg.de/wetter/weather_data.html.
Khayati M., Lerner A., Tymchenko Z., Cudré-Mauroux P. Mind the Gap: An Experimental Evaluation of Imputation of Missing Values Techniques in Time Series. Proc. VLDB Endow.2020. Vol. 13, no. 5. P. 768–782. DOI: 10.14778/3377369.3377383.
Minor B.D., Doppa J.R., Cook D.J. Learning Activity Predictors from Sensor Data: Algorithms, Evaluation, and Applications. IEEE Trans. Knowl. Data Eng. 2017. Vol. 29, no. 12. P. 2744–2757. DOI: 10.1109/TKDE.2017.2750669.
Zymbler M.L., Polonsky V.A., Yurtin A.A. On One Method of Imputation Missing Values of a Streaming Time Series in Real Time. Bulletin of the South Ural State University. Series: Computational Mathematics and Software Engineering. 2021. Vol. 10, no. 4. P. 5–25. DOI: 10.14529/cmse210401.
Pontes F.J., F. de Amorim G. da, Balestrassi P.P., et al. Design of experiments and focused grid search for neural network parameter optimization. Neurocomputing. 2016. Vol. 186. P. 22–34. DOI: 10.1016/J.NEUCOM.2015.12.061.
