Integration of Missing Data Imputation Tools for Time Series in Real-Time Mode into a Relational DBMS
DOI:
https://doi.org/10.14529/cmse250102Keywords:
time series, DBMS, PostgreSQL, missing value imputation, neural networksAbstract
The article addresses the problem of integrating time series imputation into relational database management systems (RDBMS). A method called ImputeDB is proposed, which enables the real-time integration of neural network-based imputation models into the PostgreSQL RDBMS. The imputation of missing values is carried out through triggers (stored functions automatically executed by the RDBMS kernel when new data is inserted). When a trigger is activated, missing values are replaced by synthetic ones generated by a neural network model. Using the proposed method, a database application programmer can integrate the process of imputing missing values into the standard time series processing pipeline within the PostgreSQL RDBMS, without relying on external services. The proposed approach includes a set of components implemented as user-defined functions (UDFs) in Python and PL/Python: Trigger Constructor, Model Manager, Model Storage, and Imputer. The Trigger Constructor is used to create triggers that automatically perform imputation of missing values in inserted data. The Model Manager is responsible for training neural network models, while the Model Storage is used to save these models in a file-based repository. The Imputer, in turn, synthesizes missing values using the trained models. Experiments were conducted to evaluate the performance of the ImputeDB method. The experiments measured the processing time of data insertion with automatic gap imputation as a function of the time series dimensionality. Experiments were performed under two scenarios: single and multiple insertions. Neural network-based imputation models with various architectures, including recurrent neural networks, autoencoders, and transformers, were employed. The experimental results demonstrated that under conditions of increasing time series dimensionality and rising overhead from network requests and data transfer, ImputeDB exhibits superior performance. Specifically, the system achieved an efficiency gain of 22.5% compared to another approach, while maintaining the accuracy of the employed imputation methods.References
Majumdar S., Laha A.K. Clustering and classification of time series using topological data analysis with applications to finance. Expert Syst. Appl. 2020. Vol. 162. P. 113868. DOI: 10.1016/j.eswa.2020.113868.
Kumar S., Tiwari P., Zymbler M.L. Internet of Things is a revolutionary approach for future technology enhancement: a review. J. Big Data. 2019. Vol. 6. P. 111. DOI: 10.1186/S40537-019-0268-2.
Seyedan M., Mafakheri F. Predictive big data analytics for supply chain demand forecasting: methods, applications, and research opportunities. J. Big Data. 2020. Vol. 7, no. 1. P. 53. DOI: 10.1186/S40537-020-00329-2.
Jensen S.K., Pedersen T.B., Thomsen C. Time Series Management Systems: A Survey. IEEE Trans. Knowl. Data Eng. 2017. Vol. 29, no. 11. P. 2581–2600. DOI: 10.1109/TKDE.2017.2740932.
Ivanova E.V., Zymbler M.L. Overview of Modern Time Series Management Systems. Bulletin of the South Ural State University. Series: Computational Mathematics and Software Engineering. 2020. Vol. 9, no. 4. P. 79–97. DOI: 10.14529/cmse200406.
Shen C., Ouyang Q., Li F., et al. Lindorm TSDB: A Cloud-native Time-series Database for Large-scale Monitoring Systems. Proc. VLDB Endow. 2023. Vol. 16, no. 12. P. 3715–3727. DOI: 10.14778/3611540.3611559.
Ivanova E.V., Zymbler M.L. Embedding of the Matrix Profile Concept Into a Relational DBMS for Time Series Mining. Bulletin of the South Ural State University. Series: Computational Mathematics and Software Engineering. 2021. Vol. 10, no. 3. P. 72–87. DOI: 10.14529/cmse210305.
Khalefa M.E., Fischer U., Pedersen T.B., Lehner W. Model-based Integration of Past & Future in TimeTravel. Proc. VLDB Endow. 2012. Vol. 5, no. 12. P. 1974–1977. DOI: 10.14778/2367502.2367551.
Fischer U., Rosenthal F., Lehner W. F2DB: The Flash-Forward Database System. IEEE 28th International Conference on Data Engineering (ICDE 2012), Washington, DC, USA (Arlington, Virginia), 1-5 April, 2012 / ed. by A. Kementsietsidis, M.A.V. Salles. IEEE Computer Society, 2012. P. 1245–1248. DOI: 10.1109/ICDE.2012.117.
Agarwal A., Alomar A., Shah D. tspDB: Time Series Predict DB. NeurIPS 2020 Competition and Demonstration Track, 6-12 December 2020, Virtual Event / Vancouver, BC, Canada. Vol. 133 / ed. by H.J. Escalante, K. Hofmann. PMLR, 2020. P. 27–56. Proceedings of Machine Learning Research. URL: http://proceedings.mlr.press/v133/agarwal21a.html.
Arous I., Khayati M., Cudré-Mauroux P., et al. RecovDB: Accurate and Efficient Missing Blocks Recovery for Large Time Series. 35th IEEE International Conference on Data Engineering, ICDE 2019, Macao, China, April 8-11, 2019. IEEE, 2019. P. 1976–1979. DOI: 10.1109/ICDE.2019.00218.
Ariyo A.A., Adewumi A.O., Ayo C.K. Stock Price Prediction Using the ARIMA Model. UKSim-AMSS 16th International Conference on Computer Modelling and Simulation, UKSim 2014, Cambridge, United Kingdom, March 26-28, 2014 / ed. by D. Al-Dabass, A. Orsoni, R.J. Cant, et al. IEEE, 2014. P. 106–112. DOI: 10.1109/UKSIM.2014.67.
Salinas D., Flunkert V., Gasthaus J., Januschowski T. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting. 2020. Vol. 36, no. 3. P. 1181–1191. DOI: https://doi.org/10.1016/j.ijforecast.2019.07.001.
Lim B., Arik S.Ó., Loeff N., Pfister T. Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting. CoRR. 2019. Vol. abs/1912.09363. arXiv: 1912.09363. URL: http://arxiv.org/abs/1912.09363.
Cao W., Wang D., Li J., et al. BRITS: Bidirectional Recurrent Imputation for Time Series. Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada / ed. by S. Bengio, H.M. Wallach, H. Larochelle, et al. 2018. P. 6776–6786. URL: https://proceedings.neurips.cc/paper/2018/hash/734e6bfcd358e25ac1db0a4241b95651-Abstract.html.
Yoon J., Zame W.R., Schaar M. van der. Estimating Missing Data in Temporal Data Streams Using Multi-Directional Recurrent Neural Networks. IEEE Trans. Biomed. Eng. 2019. Vol. 66, no. 5. P. 1477–1490. DOI: 10.1109/TBME.2018.2874712.
Fortuin V., Baranchuk D., Rätsch G., Mandt S. GP-VAE: Deep Probabilistic Time Series Imputation. The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy]. Vol. 108 / ed. by S. Chiappa, R. Calandra. PMLR, 2020. P. 1651–1661. Proceedings of Machine Learning Research. URL: http://proceedings.mlr.press/v108/fortuin20a.html.
Du W., Côte D., Liu Y. SAITS: Self-attention-based imputation for time series. Expert Syst. Appl. 2023. Vol. 219. P. 119619. DOI: 10.1016/J.ESWA.2023.119619.
Oh E., Kim T., Ji Y., Khyalia S. STING: Self-attention based Time-series Imputation Networks using GAN. CoRR. 2022. Vol. abs/2209.10801. DOI: 10.48550/ARXIV.2209.10801. arXiv: 2209.10801.
Fang C., Wang C. Time Series Data Imputation: A Survey on Deep Learning Approaches. CoRR. 2020. Vol. abs/2011.11347. arXiv: 2011.11347. URL: https://arxiv.org/abs/2011.11347.
Wang J., Du W., Cao W., et al. Deep Learning for Multivariate Time Series Imputation: A Survey. CoRR. 2024. Vol. abs/2402.04059. DOI: 10.48550/ARXIV.2402.04059. arXiv: 2402.04059.
Silberschatz A., Korth H.F., Sudarshan S. Database System Concepts, Seventh Edition. McGraw-Hill Book Company, 2020. URL: https://www.db-book.com/.
Yurtin A.A. Imputation of Multivariate Time Series Based on the Behavioral Patterns and Autoencoders. Bulletin of the South Ural State University. Series: Computational Mathematics and Software Engineering. 2024. Vol. 13, no. 2. P. 39–55. DOI: 10.14529/cmse240203.
Zymbler M., Goglachev A. Fast summarization of long time series with graphics processor. Mathematics. 2022. Vol. 10, no. 10. P. 1781. DOI: 10.3390/math10101781.
Imani S., Madrid F., Ding W., et al. Introducing time series snippets: A new primitive for summarizing long time series. Data Min. Knowl. Discov. 2020. Vol. 34, no. 6. P. 1713–1743. DOI: 10.1007/s10618-020-00702-y.
Bilenko R.V., Dolganina N.Y., Ivanova E.V., Rekachinsky A.I. High-performance Computing Resources of South Ural State University. Bulletin of the South Ural State University. Series: Computational Mathematics and Software Engineering. 2022. Vol. 11, no. 1. P. 15–30. DOI: 10.14529/cmse220102.
Zymbler M.L., Polonsky V.A., Yurtin A.A. On One Method of Imputation Missing Values of a Streaming Time Series in Real Time. Bulletin of the South Ural State University. Series: Computational Mathematics and Software Engineering. 2021. Vol. 10, no. 4. P. 5–25. DOI: 10.14529/cmse210401.
