Recognition of Human Fatigue Based on Speech Analysis Using Neural Network Technologies
DOI:
https://doi.org/10.14529/cmse230103Keywords:
human fatigue recognition, speech database, instrumental complex, cardio-respiratory test, machine learning, deep neural networkAbstract
Qualitative psychophysiological research studies are associated with the creation of accessible and well-organized databases that require a lot of preliminary work on the development of measuring complexes, including not only tools for measuring the psychophysiological parameters of a human, but also their emotional state, which is displayed in facial expression, speech and behavioral patterns. Measuring systems should also include the means of experimental material processing. The purpose of the study was to conduct an experiment on creating a prototype of the Speech Data Base of Russian-speaking respondents and to obtain answers to some methodological questions that arise among specialists when they use the database for the task of recognizing the state of human fatigue. A hardware and software complex has been developed that allows to synchronously register psychophysiological parameters, video recordings of behavioral reactions and audio recordings of human speech. As a model of physical fatigue, a cardio-respiratory test with physical activity (load) was used. Before and after completing the test, volunteers read out a set of standard phonetically representative texts. The obtained audio recordings were processed using a specialized neural network capable of analyzing the integral spectral characteristics of sound. The results of the experiment showed the possibility of recognizing the state of human fatigue based on speech analysis, which makes it possible to proceed to the creation of a large bank of audio recordings and the improvement of algorithms for recognizing the state of fatigue.References
Zhang X.-J., Gu J.-H., Tao Z. Research of detecting fatigue from speech by PNN. 2010 International Conference on Information, Networking and Automation (ICINA). Vol. 2. 2010. P. V2278–V2281. DOI: 10.1109/ICINA.2010.5636509.
Krajewski J., Batliner A., Golz M. Acoustic sleepiness detection: Framework and validation of a speech-adapted pattern recognition approach. Behavior Research Methods. 2009. Vol. 41, no. 3. P. 795–804. DOI: 10.3758/BRM.41.3.795.
Krajewski J., Trutschel U., Golz M., et al. Estimating Fatigue from Predetermined Speech Samples Transmitted by Operator Communication Systems. Proceedings of the 5th International Driving Symposium on Human Factors in Driver Assessment, Training, and Vehicle Design: Driving Assessment 2009. University of Iowa, 2009. DOI: 10.17077/drivingassessment.1359.
Greeley H., Berg J., Friets E., et al. Fatigue estimation using voice analysis. Behavior Research Methods. 2007. Vol. 39, no. 3. P. 610–619. DOI: 10.3758/BF03193033.
openSMILE 3.0 - audEERING. Homepage. URL: https://www.audeering.com/research/opensmile/A (accessed: 15.11.2022).
Baykaner K., Huckvale M., Whiteley I., et al. The Prediction of Fatigue Using Speech as a Biosignal. Statistical Language and Speech Processing. Vol. 9449 / ed. by A.-H. Dediu, C. Martín-Vide, K. Vicsi. Cham: Springer, 2015. P. 8–17. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). DOI: 10.1007/978-3-319-25789-1_2.
Eyben F., Scherer K., Schuller B., et al. The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing. IEEE Transactions on Affective Computing. 2016. Vol. 7, no. 2. P. 190–202. DOI: 10.1109/TAFFC.2015.2457417.
Parada-Cabaleiro E., Costantini G., Batliner A., et al. DEMoS: an Italian emotional speech corpus: Elicitation methods, machine learning, and perception. Language Resources and Evaluation. 2020. Vol. 54, no. 2. P. 341–383. DOI: 10.1007/s10579-019-09450-y.
Freitag M., Amiriparian S., Pugachevskiy S., et al. auDeep: Unsupervised learning of representations from audio with deep recurrent neural networks. Journal of Machine Learning Research. 2018. Vol. 18. P. 1–5. URL: http://jmlr.org/papers/v18/17-406.html.
Yakovlev A.V. Development of a distributed software system for synchronized collection of speech, video and psychophysiological data about a volunteer in the process of experimental research. Processing, Transmission and Protection of Information in Computer Systems ’22: Proceedings of the Second International Scientific Conference, St. Petersburg, Russia. St. Petersburg: Publishing of the St. Petersburg State University of Aerospace Instrumentation, 2022. P. 95–100. (in Russian).
Hidalgo-Gadea G., Kreuder A., Krajewski J., Vorstius C. Towards better microsleep predictions in fatigued drivers: exploring benefits of personality traits and IQ. Ergonomics. 2021. Vol. 64, no. 6. P. 778–792. DOI: 10.1080/00140139.2021.1882707.
Fan X., Zhao C., Luo H., Zhang W. An event-related potential objective evaluation study of mental fatigue based on 2-back task. Journal of biomedical engineering. 2018. Vol. 35, no. 6. P. 837–844. DOI: 10.7507/1001-5515.201801064.
Trejo L.J., Kochavi R., Kubitz K., et al. Measures and models for predicting cognitive fatigue. Biomonitoring for Physiological and Cognitive Performance during Military Operations. Vol. 5797 / ed. by J.A. Caldwell, N.J. Wesensten. International Society for Optics, Photonics. SPIE, 2005. P. 105–115. DOI: 10.1117/12.604286.
Yamada Y., Kobayashi M. Detecting mental fatigue from eye-tracking data gathered while watching video: Evaluation in younger and older adults. Artificial Intelligence in Medicine. 2018. Vol. 91. P. 39–48. DOI: 10.1016/j.artmed.2018.06.005.
Matsumoto T., Ito K., Moritani T. The relationship between anaerobic threshold and electromyographic fatigue threshold in college women. European Journal of Applied Physiology and Occupational Physiology. 1991. Vol. 63, no. 1. P. 1–5. DOI: 10.1007/BF00760792.
Solberg G., Robstad B., Skjønsberg O., Borchsenius F. Respiratory gas exchange indices for estimating the anaerobic threshold. Journal of Sports Science and Medicine. 2005. Vol. 4, no. 1. P. 29–36. URL: https://pubmed.ncbi.nlm.nih.gov/24431958/.
Yakovlev A.V. The use of multilayer networks-autoencoders for the recognition of human fatigue on the basis of speech data. Processing, Transmission and Protection of Information in Computer Systems ’22: Proceedings of the Second International Scientific Conference, St. Petersburg, Russia. St. Petersburg: Publishing of the St. Petersburg State University of Aerospace Instrumentation, 2022. P. 87–94. (in Russian).
