复杂城市动态图卷积网络三维场景语义分割法

doi:10.11947/j.AGCS.2023.20220466

摘要/Abstract

摘要： 在摄影测量与遥感领域,三维网格数据是最终用户产品之一,已广泛应用于城市规划、导航等任务中。但针对以三维网格表示的复杂城市场景的智能化语义分割的研究较少。为此,本文提出复杂城市动态图卷积网络三维场景语义分割方法(3Dcity-net)。利用三维网格固有的三维空间坐标信息和纹理信息构建的复合特征向量来表示三维网格中的三角面片。为降低纹理信息中噪声和冗余信息对语义分割结果的影响,提出在3Dcity-net网络结构中嵌入主成分分析模块。为缓解样本数据不平衡引起的语义分割精度下降的问题,采用焦点损失函数替代交叉熵损失函数。利用Hessigheim三维网格数据进行了语义分割试验。试验结果表明,3Dcity-net能够获得具有竞争力的三维网格语义分割结果,其中总体精度OA、Kappa系数、平均准确率mP、平均召回率mR、平均F₁值(F₁ score)和平均交并比mIoU分别为81.5%、0.776、73.0%、58.4%、62.6%和49.8%。与先进方法相比,本文方法总体精度OA分别提高了0.9%和8.3%。

关键词: 实景三维, 语义分割, 图卷积网络, 三维表示, 三维网格

Abstract: In photogrammetry and remote sensing community, 3D mesh is one of the final user products, which is widely applied in urban planning, navigation, etc. However, there are few works on semantic complex 3D mesh urban scene segmentation based on deep learning methods. Thus, a semantic segmentation method of 3D scenes using dynamic graph CNN for complex city (3Dcity-net) is proposed. By using mesh-inherent features containing 3D spatial information and texture information, a composite feature vector is proposed to represent each face in 3D mesh. To reduce the influence on semantic segmentation by the noise and redundant information in texture information, a principal component analysis (PCA) module is embedded in to the proposed 3D city-net. In order to alleviate the problem of semantic segmentation precision decrease caused by the unbalanced sample data, the focal loss function is used to replace the cross-entropy loss function. The Hessigheim 3D mesh data are utilized to perform experiments. The results of experiments show that the proposed method can obtain competitive semantic segmentation results on 3D mesh. The overall accuracy, Kappa coefficient, mean precision, mean recall, mean F₁ score, and mean IoU is 81.5%, 0.776, 73.0%, 58.4%, 62.6%, and 49.8%, respectively. Comparing to two state-of-the-art methods, the overall accuracy increases by 0.9%, and 8.3%, respectively.

Key words: 3D real scene, semantic segmentation, graph CNN network, 3D representation, 3D mesh

中图分类号:

张荣庭, 张广运, 尹继豪. 复杂城市动态图卷积网络三维场景语义分割法[J]. 测绘学报, 2023, 52(10): 1703-1713.

ZHANG Rongting, ZHANG Guangyun, YIN Jihao. Semantic segmentation method of 3D scenes using dynamic graph CNN for complex city[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(10): 1703-1713.

参考文献

[1] 张广运,张荣庭,戴琼海,等.测绘地理信息与人工智能2. 0融合发展的方向[J].测绘学报, 2021, 50(8):1096-1108. DOI:10.11947/j.AGCS.2021. 20210200. ZHANG Guangyun, ZHANG Rongting, DAI Qionghai, et al. The direction of integration surveying and mapping geographic information and artificial intelligence 2. 0[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(8):1096-1108. DOI:10.11947/j.AGCS.2021. 20210200.
[2] 张永军,万一,史文中,等.多源卫星影像的摄影测量遥感智能处理技术框架与初步实践[J].测绘学报, 2021, 50(8):1068-1083. DOI:10.11947/j.AGCS.2021. 20210079. ZHANG Yongjun, WAN Yi, SHI Wenzhong, et al. Technical framework and preliminary practices of photogrammetric remote sensing intelligent processing of multi-source satellite images[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(8):1068-1083. DOI:10.11947/j.AGCS.2021. 20210079.
[3] 蒋腾平,王永君,张林淇,等.融合CNN和MRF的激光点云层次化语义分割方法[J].测绘学报, 2021, 50(2):215-225. DOI:10.11947/j.AGCS.2021. 20200095. JIANG Tengping, WANG Yongjun, ZHANG Linqi, et al. A LiDAR point cloud hierarchical semantic segmentation method combining CNN and MRF[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(2):215-225. DOI:10.11947/j.AGCS.2021. 20200095.
[4] 黄刚.基于深度学习的移动激光点云数据自动分类方法研究[J].测绘学报, 2022, 51(5):786. DOI:10.11947/j.AGCS.2022. 20200599. HUANG Gang. Research on automatic classification method of mobile laser point cloud data based on deep learning[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(5):786. DOI:10.11947/j.AGCS.2022. 20200599.
[5] 赵传,郭海涛,卢俊,等.结合区域增长与RANSAC的机载LiDAR点云屋顶面分割[J].测绘学报, 2021, 50(5):621-633. DOI:10.11947/j.AGCS.2021. 20200270. ZHAO Chuan, GUO Haitao, LU Jun, et al. Roof segmentation from airborne LiDAR by combining region growing with random sample consensus[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(5):621-633. DOI:10.11947/j.AGCS.2021. 20200270.
[6] HU Xiangyun, YUAN Yi. Deep-learning-based classification for DTM extraction from ALS point cloud[J]. Remote Sensing, 2016, 8(9):730.
[7] RIZALDY A, PERSELLO C, GEVAERT C, et al. Ground and multi-class classification of airborne laser scanner point clouds using fully convolutional networks[J]. Remote Sensing, 2018, 10(11):1723.
[8] CHARLES R Q, HAO Su, MO Kaichun, et al. PointNet:deep learning on point sets for 3 D classification and segmentation[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu:IEEE, 2017:77-85.
[9] QI C R, YI Li, SU Hao, et al. PointNet++:deep hierarchical feature learning on point sets in a metric space[C]//Proceedings of the 31 st International Conference on Neural Information Processing Systems. Long Beach:ACM Press, 2017:5105-5114.
[10] WANG Yue, SUN Yongbin, LIU Ziwei, et al. Dynamic graph CNN for learning on point clouds[J]. ACM Transactions on Graphics, 38(5):146,.
[11] GUPTA A, BYRNE J, MOLONEY D, et al. Tree annotations in LiDAR data using point densities and convolutional neural networks[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(2):971-981.
[12] YE Maosheng, XU Shuangjie, CAO Tongyi. HVNet:hybrid voxel network for LiDAR based 3 D object detection[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle:IEEE, 2020:1628-1637.
[13] GUO Menghao, CAI Junxiong, LIU Zhengning, et al. PCT:point cloud transformer[J]. Computational Visual Media, 2021, 7(2):187-199.
[14] 吴军,崔玥,赵雪梅,等. SSA-PointNet++:空间自注意力机制下的3 D点云语义分割网络[J].计算机辅助设计与图形学学报, 2022, 34(3):437-448. WU Jun, CUI Yue, ZHAO Xuemei, et al. SSA-PointNet++:a space self-attention CNN for the semantic segmentation of 3 D point cloud[J]. Journal of Computer-Aided Design&Computer Graphics, 2022, 34(3):437-448.
[15] ZHU Jianfeng, SUI Lichun, ZANG Yufu, et al. Classification of airborne laser scanning point cloud using point-based convolutional neural network[J]. ISPRS International Journal of Geo-Information, 2021, 10(7):444.
[16] WANG Y, LI S, WANG M, et al. A simple deep learning network for classification of 3 D mobile LiDAR point cloud[J]. Journal of Geodesy and Geoinformation Science, 2021, 4(3):49-59.
[17] ZUO Renxiang, ZHANG Guangyun, ZHANG Rongting, et al. A deformable attention network for high-resolution remote sensing images semantic segmentation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:1-14.
[18] SAHA S, MOU Lichao, QIU Chunping, et al. Unsupervised deep joint segmentation of multitemporal high-resolution images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(12):8780-8792.
[19] KOTARIDIS I, LAZARIDOU M. Remote sensing image segmentation advances:a metaanalysis[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 173:309-322.
[20] HE Yunqian, XIA Guihua, LUO Yongkang, et al. DVFENet:dual-branch voxel feature extraction network for 3 D object detection[J]. Neurocomputing, 2021, 459:201-211.
[21] KÖLLE M, LAUPHEIMER D, SCHMOHL S, et al. The hessigheim 3 D (H3 D) benchmark on semantic segmentation of high-resolution 3 D point clouds and textured meshes from UAV LiDAR and multi-view-stereo[EB/OL].[2022-07-21]. https://arxiv.org/abs/2102.05346.
[22] SHILANE P, MIN P, KAZHDAN M, et al. The princeton shape benchmark[C]//Proceedings of 2004 Shape Modeling Applications. Genova:IEEE, 2004:167-178.
[23] WANG Yunhai, ASAFI S, VAN KAICK O, et al. Active co-analysis of a set of shapes[J]. ACM Transactions on Graphics, 2012, 31(6):1-10.
[24] BOGO F, ROMERO J, LOPER M, et al. FAUST:dataset and evaluation for 3 D mesh registration[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus:IEEE, 2014:3794-3801.
[25] BRONSTEIN A M, BRONSTEIN M M, CASTELLANI U, et al. SHREC 2010:robust largescale shape retrieval benchmark[J]. Processing 3 DOR, 2010, 5(4):1-8.
[26] MASCI J, BOSCAINI D, BRONSTEIN M M, et al. Geodesic convolutional neural networks on Riemannian manifolds[C]//Proceedings of 2015 IEEE International Conference on Computer Vision Workshop. Santiago:IEEE, 2016:832-840.
[27] BOSCAINI D, MASCI J, RODOIÀ E, et al. Learning shape correspondence with anisotropic convolutional neural networks[C]//Proceedings of the 30 th International Conference on Neural Information Processing Systems. Barcelona:ACM Press, 2016:3197-3205.
[28] MONTI F, BOSCAINI D, MASCI J, et al. Geometric deep learning on graphs and manifolds using mixture model CNNs[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu:IEEE, 2017:5425-5434.
[29] HANOCKA R, HERTZ A, FISH N, et al. MeshCNN:a network with an edge[J]. ACM Transactions on Graphics, 38(4):90.
[30] LIN T Y, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(2):318-327.
[31] LAUPHEIMER D, SHAMSEDDIN M H, HAALA N. The importance of radiometric feature quality for semantic mesh segmentation[C]//Proceedings of 2020 Computer Science, Environmental Science. Stuttgart:DGPF, 2020.
[32] TUTZAUER P, LAUPHEIMER D, HAALA N. Semantic urban mesh enhancement utilizing a hybrid model[J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2019, 4:175-182