A deep learning network for semantic labeling of large-scale urban point clouds

doi:10.11947/j.AGCS.2021.20210093

Abstract

Abstract: In recent years, point cloud has become an important type of 3D spatial data. How to improve the understanding abilities of point cloud using artificial intelligence for correct semantic labeling and accurate detection of objects is an urgent and difficult problem. This paper hence proposes an end-to-end 3D point cloud deep learning network, which effectively guarantees the efficiencies of point cloud sampling, the accuracy of feature extraction and the optimization of the overall network performance by the up-down sampling strategy of irregular distribution point cloud, multi-layer aggregation and propagation of features and the loss function for uneven samples. The studies on the large-scale 3D point cloud benchmark data show that it achieves excellent performance in semantic labeling for large-scale outdoor scenes of point clouds, better than those of the state-of-art deep learning networks of point cloud, providing a strong support for the high-performance extraction of 3D geospatial information.

Key words: Deep learning, artificial intelligence, point cloud, semantic labeling, feature extraction

CLC Number:

P207

YANG Bisheng, HAN Xu, DONG Zhen. A deep learning network for semantic labeling of large-scale urban point clouds[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(8): 1059-1067.

References

[1] 杨必胜, 梁福逊, 黄荣刚. 三维激光扫描点云数据处理研究进展、挑战与趋势[J]. 测绘学报, 2017, 46(10):1509-1516.DOI:10.11947/j.AGCS.2017.20170351. YANG Bisheng, LIANG Fuxun, HUANG Ronggang. Progress, challenges and perspectives of 3D LiDAR point cloud processing[J]. Acta Geodaetica et Cartographica Sinica, 2017, 46(10):1509-1516.DOI:10.11947/j.AGCS.2017.20170351.
[2] 廖小罕. 中国对地观测20年科技进步和发展[J]. 遥感学报, 2021, 25(1):267-275. LIAO Xiaohan. Scientific and technological progress and development prospect of the earth observation in China in the past 20 years[J]. National Remote Sensing Bulletin, 2021, 25(1):267-275.
[3] 杨必胜, 董震. 点云智能研究进展与趋势[J]. 测绘学报, 2019, 48(12):1575-1585. DOI:10.11947/j.AGCS.2019.20190465. YANG Bisheng, DONG Zhen. Progress and perspective of point cloud intelligence[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(12):1575-1585. DOI:10.11947/j.AGCS.2019.20190465.
[4] 杨必胜, 董震. 点云智能处理[M]. 北京:科学出版社, 2020. YANG Bisheng, DONG Zhen. Intelligent processing of point cloud[M]. Beijing:Science Press, 2020.
[5] MA J W, CZERNIAWSKI T, LEITE F. Semantic segmentation of point clouds of building interiors with deep learning:Augmenting training datasets with synthetic BIM-based point clouds[J]. Automation in Construction, 2020, 113:103144.
[6] DONG Zhen, YANG Bisheng, HU Pingbo, et al. An efficient global energy optimization approach for robust 3D plane segmentation of point clouds[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 137:112-133.
[7] YI Li, SU Hao, GUO Xingwen, et al. SyncSpecCNN:synchronized spectral CNN for 3D shape segmentation[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Honolulu, HI, USA:IEEE, 2017:6584-6592.
[8] WU Zhirong, SONG Shuran, KHOSLA A, et al. 3D ShapeNets:a deep representation for volumetric shapes[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA:IEEE, 2015:1912-1920.
[9] DAI A, CHANG A X, SAVVA M, et al. ScanNet:richly-annotated 3D reconstructions of indoor scenes[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA:IEEE, 2017:2432-2443.
[10] HACKEL T, SAVINOV N, LADICKY L, et al. Semantic3d.net:a new large-scale point cloud classification benchmark[J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2017, IV-1/W1:91-98.
[11] GEIGER A, LENZ P, URTASUN R. Are we ready for autonomous driving? The KITTI vision benchmark suite[C]//2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, RI, USA:IEEE, 2012:3354-3361.
[12] DONG Zhen, LIANG Fuxun, YANG Bisheng, et al. Registration of large-scale terrestrial laser scanner point clouds:a review and benchmark[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 163:327-342.
[13] 杨必胜, 韩旭, 董震. 点云深度学习基准数据集[J]. 遥感学报, 2021, 25(1):231-240. YANG Bisheng, HAN Xu, DONG Zhen. Point Cloud Benchmark Dataset WHU-TLS and WHU-MLS for Deep Learning[J]. Journal of Remote Sensing, 2021, 25(1):231-240.
[14] FANG Hao, LAFARGE F. Pyramid scene parsing network in 3D:Improving semantic segmentation of point clouds with multi-scale contextual information[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 154:246-258.
[15] HU Qingyong, YANG Bo, XIE Linhai, et al. RandLA-net:efficient semantic segmentation of large-scale point clouds[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA:IEEE, 2020:11105-11114.
[16] CHEN Xiaozhi, MA Huimin, WAN Ji, et al. Multi-view 3D object detection network for autonomous driving[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA:IEEE, 2017:6526-6534.
[17] YANG Bin, LUO Wenjie, URTASUN R. PIXOR:real-time 3D object detection from point clouds[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA:IEEE, 2018:7652-7660.
[18] LANG A H, VORA S, CAESAR H, et al. PointPillars:fast encoders for object detection from point clouds[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA, USA:IEEE, 2019:12689-12697.
[19] GRAHAM B, ENGELCKE M, MAATEN L V D. 3D semantic segmentation with submanifold sparse convolutional networks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Salt Lake City, UT, USA:IEEE, 2018:9224-9232.
[20] MENG H Y, GAO Lin, LAI Yukun, et al. VV-net:voxel VAE net with group convolutions for point cloud segmentation[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South):IEEE, 2019:8499-8507.
[21] CHARLES R, SU H, KAICHUN M, et al. PointNet:Deep Learning on Point Sets for 3D Classification and Segmentation[C/OL]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Honolulu, HI, USA:IEEE,2017.
[22] QI C R, YI Li, SU Hao, et al. PointNet++:deep hierarchical feature learning on point sets in a metric space[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook, NY, USA:Curran Associates Inc., 2017:5105-5114.
[23] WANG Yue, SUN Yongbin, LIU Ziwei, et al. Dynamic graph CNN for learning on point clouds[J]. ACM Transactions on Graphics, 2019, 38(5):1-12.
[24] JIANG Li, ZHAO Hengshuang, LIU Shu, et al. Hierar chical point-edge interaction network for point cloud semantic segmentation[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South):IEEE, 2019:10432-10440. DOI:10.1109/ICCV.2019.01053.
[25] HERMOSILLA P, RITSCHEL T, VÁZQUEZ P P, et al. Monte Carlo convolution for learning on non-uniformly sampled point clouds[J]. ACM Transactions on Graphics, 2019, 37(6):1-12.
[26] XU Yifan, FAN Tianqi, XU Mingye, et al. SpiderCNN:deep learning on point sets with parameterized convolutional filters[M]//FERRARI V, HEBERT M, SMINCHISESCU C, et al.Computer Vision-ECCV 2018. Cham:Springer International Publishing, 2018:90-105. DOI:10.1007/978-3-030-01237-3_6.
[27] WU Wenxuan, QI Z, FUXIN Li. PointConv:deep convolutional networks on 3D point clouds[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Long Beach, CA, USA:IEEE, 2019:9613-9622.
[28] ZHANG Rui, LI Guangyun, LI Minglei, et al. Fusion of images and point clouds for the semantic segmentation of large-scale 3D scenes based on deep learning[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 143:85-96.
[29] LUO Zhipeng, LIU Di, LI J, et al. Learning sequential slice representation with an attention-embedding network for 3D shape recognition and retrieval in MLS point clouds[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 161:147-163.