A LiDAR point cloud hierarchical semantic segmentation method combining CNN and MRF

doi:10.11947/j.AGCS.2021.20200095

Abstract

Abstract: The result of point cloud semantic segmentation includes the recognition of multiple objects in the scene, which is an important part of 3D scene information extraction. It also plays a key role in many fields such as smart cities. Since the large amount of data and high scene complexity, however, most existing methods can only extract a limited type of objects with a relatively low recognition rate from laser scanning data. This paper proposes a hierarchical multiple object automatic extraction framework combining residual learning and Markov random field (MRF) optimization in a 3D LiDAR point cloud scene. The framework first filters raw data into ground points and non-ground points; buildings are extracted from non-ground points to reduce scene complexity; then the residual learning network is adopted to pointwise classify the remaining point clouds; finally, a MRF-based optimization is used for post-processing and improving the accuracy of point cloud classification. In order to evaluate the effectiveness and robustness of the proposed method, experiments were performed on three outdoor large-scale point cloud scenarios. Experimental results show that the proposed method can perform effective semantic segmentation on various types of point cloud scenes, with (94.6%, 96.8%, 95.7%), (88.5%, 90.5%, 89.2%) and (95.3%, 95.2%, 95.3%), respectively. In addition, compared with existing advanced methods, it is shown that our method significantly improves the performance of semantic segmentation.

Key words: LiDAR point cloud, semantic segmentation, hierarchical extraction, residual learning, MRF

CLC Number:

P237

JIANG Tengping, WANG Yongjun, ZHANG Linqi, LIANG Chong, SUN Jian. A LiDAR point cloud hierarchical semantic segmentation method combining CNN and MRF[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(2): 215-225.

References

[1] 杨必胜, 董震. 点云智能研究进展与趋势[J]. 测绘学报, 2019, 48(12):1575-1585. DOI:10.11947/j.AGCS.2019.20190465. YANG Bisheng, DONG Zhen. Progress and perspective of point cloud intelligence[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(12):1575-1585. DOI:10.11947/j.AGCS.2019.20190465.
[2] 杨必胜, 梁福逊, 黄荣刚. 三维激光扫描点云数据处理研究进展、挑战与趋势[J]. 测绘学报, 2017, 46(10):1509-1516. DOI:10.11947/j.AGCS.2017.20170351. YANG Bisheng, LIANG Fuxun, HUANG Ronggang. Progress, challenges and perspectives of 3D LiDAR point cloud processing[J]. Acta Geodaetica et Cartographica Sinica, 2017, 46(10):1509-1516. DOI:10.11947/j.AGCS.2017.20170351.
[3] WEINMANN M, JUTZI B, HINZ S, et al. Semantic point cloud interpretation based on optimal neighborhoods, relevant features and efficient classifiers[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2015, 105:286-304.
[4] LANDRIEU L, RAGUET H, VALLET B, et al. A structured regularization framework for spatially smoothing semantic labelings of 3D point clouds[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2017, 132:102-118.
[5] 熊艳, 高仁强, 徐战亚. 机载LiDAR点云数据降维与分类的随机森林方法[J]. 测绘学报, 2018, 47(4):508-518. DOI:10.11947/j.AGCS.2018.20170417. XIONG Yan, GAO Renqiang, XU Zhanya. Random forest method for dimension reduction and point cloud classification based on airborne LiDAR[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(4):508-518. DOI:10.11947/j.AGCS.2018.20170417.
[6] ZHU Zan, GAN Shu,WANG Jianqi, et al. A Novel airborne 3D laser point cloud hole repair algorithm considering topographic features[J]. Journal of Geodesy and Geoinformation Science, 2020, 3(3):29-38.
[7] 赵传, 郭海涛, 卢俊, 等. 基于深度残差网络的机载LiDAR点云分类[J]. 测绘学报, 2020, 49(2):202-213. DOI:10.11947/j.AGCS.2020.20190004. ZHAO Chuan, GUO Haitao, LU Jun, et al. Airborne LiDAR point cloud classification based on deep residual network[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(2):202-213. DOI:10.11947/j.AGCS.2020.20190004.
[8] HUA B S, TRAN M K, YEUNG S K. Pointwise convolutional neural networks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City:IEEE, 2018:984-993.
[9] ZHANG Dejun, HE Fazhi, TU Zhigang, et al. Pointwise geometric and semantic learning network on 3D point clouds[J]. Integrated Computer-Aided Engineering, 2020, 27(1):57-75.
[10] LUO Huan, WANG Cheng, WEN Chenglu, et al. Patch-based semantic labeling of road scene using colorized mobile LiDAR point clouds[J]. IEEE Transactions on Intelligent Transportation Systems, 2016, 17(5):1286-1297.
[11] 罗海峰, 方莉娜, 陈崇成, 等. 基于DBN的车载激光点云路侧多目标提取[J]. 测绘学报, 2018, 47(2):234-246. DOI:10.11947/j.AGCS.2018.20170524. LUO Haifeng, FANG Lina, CHEN Chongcheng, et al. Roadside multiple objects extraction from mobile laser scanning point cloud based on DBN[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(2):234-246. DOI:10.11947/j.AGCS.2018.20170524.
[12] WANG Hanyun, WANG Cheng, LUO Huan, et al. 3-D point cloud object detection based on supervoxel neighborhood with Hough forest framework[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2015, 8(4):1570-1581.
[13] 董震, 杨必胜. 车载激光扫描数据中多类目标的层次化提取方法[J]. 测绘学报, 2015, 44(9):980-987. DOI:10.11947/j.AGCS.2015.20140339. DONG Zhen, YANG Bisheng. Hierarchical extraction of multiple objects from mobile laser scanning data[J]. Acta Geodaetica et Cartographica Sinica, 2015, 44(9):980-987. DOI:10.11947/j.AGCS.2015.20140339.
[14] 李明磊, 刘少创, 杨欢, 等. 双层优化的激光雷达点云场景分割方法[J]. 测绘学报, 2018, 47(2):269-274. DOI:10.11947/j.AGCS.2018.20170493. LI Minglei, LIU Shaochuang, YANG Huan, et al. Bilevel optimization for scene segmentation of LiDAR point cloud[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(2):269-274. DOI:10.11947/j.AGCS.2018.20170493.
[15] XU Yusheng, YE Zhen, YAO Wei, et al. Classification of LiDAR point clouds using supervoxel-based detrended feature and perception-weighted graphical model[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13:72-88.
[16] BABAHAJIANI P, FAN Lixin, GABBOUJ M. Object recognition in 3D point cloud of urban street scene[C]//Computer Vision-ACCV Workshops. Singapore:Springer, 2014:177-190.
[17] YANG Bisheng, DONG Zhen, LIU Yuan, et al. Computing multiple aggregation levels and contextual features for road facilities recognition using mobile laser scanning data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2017, 126:180-194.
[18] YANG Bisheng, DONG Zhen, ZHAO Gang, et al. Hierarchical extraction of urban objects from mobile laser scanning data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2015, 99:45-57.
[19] TEO T A, CHIU C M. Pole-like road object detection from mobile LiDAR system using a coarse-to-fine approach[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2015, 8(10):4805-4818.
[20] LIU Xiaoqiang, CHEN Yanming, LI Shuyi, et al. Hierarchical classification of urban ALS data by using geometry and intensity information[J]. Sensors, 2019, 19(20):4583.
[21] WANG Yongjun, JIANG Tengping, YU Min, et al. Semantic-based building extraction from LiDAR point clouds using contexts and optimization in complex environment[J]. Sensors, 2020, 20:3386.
[22] LI Nan, LIU Chun, PFEIFER N. Improving LiDAR classification accuracy by contextual label smoothing in post-processing[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 148:13-31.
[23] WANG Y, JIANG T, LIU J, et al. Hierarchical instance recognition of individual roadside trees in environmentally complex urban areas from UAV laser scanning point clouds[J]. ISPRS International Journal of Geo-Information. 2020; 9(10):595.
[24] ZHANG Liangpei,ZHANG Yun, CHEN Zhenzhong, et al. Splitting and merging based multi-model fitting for point cloud segmentation[J]. Journal of Geodesy and Geoinformation Science, 2019, 2(2):78-89.
[25] RUSU R B, COUSINS S. 3D is here:Point cloud library (PCL)[C]//2011 IEEE International Conference on Robotics and Automation. Shanghai, China:IEEE, 2011:1-4.
[26] ZHANG Wuming, QI Jianbo, WAN Peng, et al. An easy-to-use airborne LiDAR data filtering method based on cloth simulation[J]. Remote Sensing, 2016, 8(6):501.
[27] XIA Shanbo, WANG Ruisheng. Extraction of residential building instances in suburban areas from mobile LiDAR data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 144:453-468.
[28] SCHNABEL R, WAHL R, KLEIN R. Efficient RANSAC for point-cloud shape detection[J]. Computer Graphics Forum, 2007, 26(2):214-226.
[29] KOLMOGOROV V, ZABIN R. What energy functions can be minimized via graph cuts?[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(2):147-159.
[30] QI C R, SU Hao, MO Kaichun, et al. PointNet:deep learning on point sets for 3D classification and segmentation[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI:IEEE, 2017:77-85.
[31] ARSHAD S, SHAHZAD M, RIAZ Q, et al. DPRNet:Deep 3D point based residual network for semantic segmentation and classification of 3D point clouds[J]. IEEE Access, 2019, 7:68892-68904.
[32] KLAMBAUER G, UNTERTHINER T, MAYR A, et al. Self-normalizing neural networks[C]//31st Conference on Neural Information Processing Systems. Long Beach, CA:[s.n.], 2017:971-980.
[33] XIANG Binbin, YAO Jian, LU Xiaohu, et al. Segmentation-based classification for 3D point clouds in the road environment[J]. International Journal of Remote Sensing, 2018, 39(19):6182-6212.
[34] HACKEL T, SAVINOV N, LADICKY L, et al. SEMANTIC3D.NET:A new large-scale point cloud classification benchmark[J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2017, IV-1/W1:91-98.