Multimodal Dynamic Graph CNN for 3D Semantic Segmentation
DOI:
https://doi.org/10.14529/cmse240202Keywords:
segmentation of 3D objects, graph convolutional neural networks, point clouds registrationAbstract
In this paper, a semantic segmentation method of point clouds in the form of terrain using a new multimodal convolutional neural network architecture based on a regular dynamic weighted graph, which allows to obtain an accurate solution to the segmentation problem based on a fusion of geometric and color features. The method can be effectively used for sparse, noisy, inhomogeneous and non convex point clouds. The computer modeling of state-of-the-art methods for 3D semantic segmentation was carried out using the reference data collection ModelNet 40 and a data set of archaeological sites of the Bronze Age of the Southern Trans-Urals, namely data obtained as a result of a total station survey (the Trimble 3300 total station) of a complex of archaeological sites in the valley of the Sintashta river. A comparative analysis of the proposed method and state-of-the-art methods for 3D semantic segmentation with different combinations of input features of point clouds was carried out, and the method influence of forming a point cloud on the accuracy of 3D semantic segmentation was also investigated: in the first case, a point cloud from a reference dataset was studied, in the second case, variants using 3D registration based on NICP and FICP algorithms were applied.References
Su H., Maji S., Kalogerakis E., Learned–Miller E. Multi-view Convolutional Neural Networks for 3D Shape Recognition. IEEE Proceedings of International Conference on Computer Vision (ICCV) (Santiago, Chile, December 7–13, 2015). P. 945–953. DOI: 10.1109/ICCV.2015.114.
Charles R.Q., Su H., Kaichun M., Guibas L.J. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (Honolulu, HI, USA, July 21–26, 2017). P. 77–85. DOI: 10.1109/CVPR.2017.16.
Charles R.Q., Li Y., Hao S., Leonidas J.G. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. Proceedings of 31st Conference on Neural Information Processing Systems (NIPS) (Long Beach, USA, December 4–9, 2017). P. 5099–5108. DOI: 10.48550/arXiv.1706.02413.
Zhang Y., Rabbat M. A Graph-CNN for 3D Point Cloud Classification. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (Calgary, AB, Canada, April 15–20, 2018). P. 6279–6283. DOI: 10.1109/ICASSP.2018.8462291.
Wang Y., Sun. Y., Liu Z., et al. Dynamic Graph CNN for Learning on Point Clouds. ACM Transactions on Graphics. 2019. Vol. 38. no. 5. Article 146. P. 1–12. DOI: 10.1145/3326362.
Te G., Hu W., Zheng A., Guo Z. RGCNN: Regularized Graph CNN for Point Cloud Segmentation. Proceedings of the 26th ACM international conference on Multimedia (MM ’18) (Seoul, Republic of Korea, October 22–26, 2018). ACM, 2018. P. 746–754. DOI: 10.1145/3240508.3240621.
Boulch A. ConvPoint: Continuous convolutions for point cloud processing. Computers and Graphics. 2020. Vol. 88. P. 24–34. DOI: 10.1016/j.cag.2020.02.005.
Hugues T., Charles R.Q., Deschaud J.E., et al. KPConv: Flexible and Deformable Convolution for Point Clouds. IEEE/CVF International Conference on Computer Vision (ICCV) (Seoul, Republic of Korea, October 27 – November 2, 2019). P. 6410–6419. DOI: 10.1109/ICCV.2019.00651.
Zhang Z., Hua B.S., Yeung S.K. ShellNet: Efficient Point Cloud Convolutional Neural Networks Using Concentric Shells Statistics. IEEE/CVF International Conference on Computer Vision (ICCV) (Seoul, Republic of Korea, October 27 – November 2, 2019). P. 1607–1616. DOI: 10.1109/ICCV.2019.00169.
Damien R., Hugo R., Loic L. Efficient 3D semantic segmentation with superpoint transformer. IEEE/CVF International Conference on Computer Vision (ICCV) (Paris, France, October 1–6, 2023). P. 17149–17158. DOI: 10.1109/ICCV51070.2023.0157.
D Semantic Segmentation on DALES. URL: https://paperswithcode.com/sota/3dsemantic-segmentation-on-dales (accessed: 31.03.2024).
Vokhmintcev A.V., Melnikov A.V., Pachganov S.A. Simultaneous localization and mapping method in three-dimensional space based on the combined solution of the point-point variation problem icp for an affine transformation. Informatics and Applications. 2020. Vol. 14, no. 1. P. 101–112. DOI: 10.14357/19922264200114.
Vokhmintcev A.V., Sochenkov I.V., Kuznetsov V.V., Tikhonkikh D.V. Face recognition based on matching algorithm with recursive calculation of local oriented gradient histogram. Doklady Mathematics. 2016. Vol. 93, no. 1. P. 37–41. DOI: 10.1134/S1064562416010178.
Vokhmintcev A.V., Khristodulo O.I, Melnikov A.V., Romanov M.A. Application of Dynamic Graph CNN* and FICP for Detection and Research Archaeology Sites. Analysis of Images, Social Networks and Texts (AIST 2023). Vol. 14486 / eds. by D.I. Ignatov, et al. Cham: Springer, 2024. Lecture Notes in Computer Science. DOI: 10.1007/978-3-031-54534-4_21.
Day W.H.E., Edelsbrunner H. Efficient algorithms for agglomerative hierarchical clustering methods. Journal of Classification. 1984. Vol. 1, no. 1. P. 7–24. DOI: 10.1007/BF01890115.
Qian G., Abualshour A., Li G., et al. PU-GCN: Point Cloud Upsampling using Graph Convolutional Networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (Nashville, TN, USA, June 20–25, 2021). DOI: 10.1109/CVPR46437.2021.01151.
Vokhmintsev A.V., Khristodulo O.I., Romanov M.A. Semantic Classification and Segmentation of Archaeological Sites Based on a Fusion of Object Detector and 3DEF. 2023 International Russian Automation Conference (RusAutoCon) (Sochi, Russian Federation, September 10–16, 2023). P. 122–127. DOI: 10.1109/RusAutoCon58002.2023.10272916
