Coordinated Checkpointing with Sender-based Logging and Asynchronous Recovery from Failure
DOI:
https://doi.org/10.14529/cmse190205Keywords:
MPI, ULFM extension, coordinated checkpointing, asynchronous recovery, fault toleranceAbstract
The increasing growth in the number of components of supercomputers leads HPC specialists to unfavorable estimates for future supercomputers: “the range of the mean time between failures will be from 1 hour to 9 hours.” This estimate leads to the problem of long calculations on supercomputers. In this paper, we propose a recovery method from failure which does not require rollback for all processes. This method can reduce overhead costs for some computational algorithms. The standard fault tolerance method consists of two phases: coordinated checkpointing and rollback of all processes to the last checkpoint in the case of a failure. The proposed method includes coordinated checkpointing with sender-based logging and asynchronous recovery when most processes wait and several processes recalculate the lost data. We developed parallel programs to solve the problem of heat transfer in the thin plate which computation algorithm has a small amount of data for logging. In these programs, failures occur by calling the function raise(SIGKILL), coordinated or asynchronous recovery is performed by ULFM functions. In order to obtain theoretical estimates of overhead costs, we propose a simulation model of program execution with failures. This model assumes that failures strike during the computations, checkpointing and recovery. We made a comparison of recovery methods with different failure rates. The comparison showed that the use of asynchronous recovery results in a reduction of overhead costs by theoretical estimates from 22% to 40%, and by computational experiments from 13% to 53%.References
Schroeder B., Gibson G.A. Understanding Failures in Petascale Computers. Journal of Physics: Conference Series. 2007. vol. 78, no. 1. pp. 12–22. DOI: 10.1088/1742-6596/78/1/012022
Hsu C.-H., Feng W.-C. A Power-aware Run-time System for High-performance Computing. Proceedings of the 2005 ACM/IEEE Conference on Supercomputing (Seattle, WA, USA, November 12 – 18, 2005). IEEE, 2005. pp. 1–9. DOI: 10.1109/sc.2005.3
Martino C.D., Kalbarczyk Z., Iyer R.K., Baccanico F., Fullop J., Kramer W. Lessons Learned from the Analysis of System Failures at Petascale: The Case of Blue Waters. 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (Atlanta, Georgia, USA, June 23 – 26, 2014). IEEE, 2014. pp. 610–621. DOI: 10.1109/dsn.2014.62
Dongarra J., Herault T., Robert Y. Fault-tolerance Techniques for High-performance Computing. Springer, Cham, 2015. 320 p. DOI: 10.1007/978-3-319-20943-2
Berkeley Lab Checkpoint/Restart (BLCR) for LINUX. Available at: http://crd.lbl.gov/departments/computer-science/CLaSS/research/BLCR/ (accessed: 03.11.2018)
Cappello F., Geist A., Gropp W., Kale S., Kramer B., Snir M., Toward Exascale Resilience: 2014 Update. Supercomputing Frontiers and Innovations. 2014. vol. 1, no. 1. pp. 5–28. DOI: 10.14529/jsfi140101
Elnozahy E.N. M., Alvisi L., Wang Y.-M., Johnson D. B. A Survey of Rollback-recovery Protocols in Message-passing Systems. ACM Computing Surveys. 2002. vol. 34, no. 3. pp. 375–408. DOI: 10.1145/568522.568525
Bouteiller A., Herault T., Bosilca G., Du P., Dongarra J. Algorithm-based Fault Tolerance for Dense Matrix Factorizations, Multiple Failures and Accuracy. ACM Transactions on Parallel Computing. 2015. vol. 1, no. 2. pp. 1–28. DOI: 10.1145/2686892
Engelmann C., Vallee G.R., Naughton T., Scott S.L. Proactive Fault Tolerance Using Preemptive Migration. 17th Euromicro International Conference on Parallel, Distributed and Network-based Processing (Weimar, Germany, February 18 – 20, 2009). IEEE, 2009. pp. 252–257. DOI: 10.1109/PDP.2009.31.
Bondarenko A.A., Yakobovskiy M.V. Fault Tolerance for HPC by Using Local Checkpoints. Vestnik Yuzho-Uralskogo gosudarstvennogo universiteta. Seriya Vychislitel'naya matematika i informatika [Bulletin of South Ural State University. Series: Computational Mathematics and Software Engineering]. 2014. vol. 3, no. 3. pp. 20–36. DOI: 10.14529/cmse140302 (in Russian)
Di S., Bouguerra M.S., Bautista-Gomez L., Cappello F. Optimization of Multi-level Checkpoint Model for Large Scale HPC Applications. 28th International Parallel and Distributed Processing Symposium (Phoenix, Arizona, USA, May 19 – 23, 2014). IEEE, 2014. pp. 1181–1190. DOI: 10.1109/IPDPS.2014.122.
Benoit A., Cavelan A., Le Fèvre V., Robert Y., Sun H. Towards Optimal Multi-level Checkpointing. IEEE Transactions on Computers. 2016. vol. 66, no. 7. pp. 1212–1226. DOI: 10.1109/TC.2016.2643660.
Di S., Robert Y., Vivien F., Cappello F. Toward an Optimal Online Checkpoint Solution under a Two-level HPC Checkpoint Model. IEEE Transactions on Parallel and Distributed Systems. 2016. vol. 28, no. 1. pp. 244–259. DOI: 10.1109/TPDS.2016.2546248.
Fault Tolerance Research Hub. Available at: http://fault-tolerance.org/ (accessed: 03.11.2018)
Bondarenko A.A., Lyakhov P.A., Yakobovskiy M.V. The Overheads Associated with Multi-level Coordinated Checkpointing. Parallelnye vychislitelnye tekhnologii (PaVT'2017): Trudy mezhdunarodnoj nauchnoj konferentsii (Kazan', 3 – 7 aprelya 2017) [Parallel Computational Technologies (PCT'2017): Proceedings of the International Scientific Conference (Kazan, Russia, 3 – 7 April, 2017)]. Chelyabinsk, Publishing of the South Ural State University, 2017. pp. 262–270. (in Russian)
