结合多尺度共享编码的半监督网络航空影像语义分割

doi:10.11947/j.AGCS.2022.20200522

测绘学报 ›› 2022, Vol. 51 ›› Issue (11): 2355-2364.doi: 10.11947/j.AGCS.2022.20200522

结合多尺度共享编码的半监督网络航空影像语义分割

李佳田¹, 杨汝春¹, 姚彦吉¹, 贺日兴^2,3, 阿晓荟¹, 吕少云¹

1. 昆明理工大学国土资源工程学院, 云南昆明 650093;
2. 首都师范大学资源环境与旅游学院, 北京 100048;
3. 首都师范大学三维数据获取与应用教育部重点实验室, 北京 100048

收稿日期:2020-10-23 修回日期:2022-05-05 发布日期:2022-11-30
通讯作者: 杨汝春 E-mail:1572112413@qq.com
作者简介:李佳田(1975—),男,教授,博士生导师,研究方向为数值最优化方法与机器场景理解。 E-mail: ljtwcx@163.com
基金资助:
国家自然科学基金(41561082)

Semantic segmentation of aerial image based on semi-supervised network with multi-scale shared coding

LI Jiatian¹, YANG Ruchun¹, YAO Yanji¹, HE Rixing^2,3, A Xiaohui¹, LÜ Shaoyun¹

1. Faculty of Land and Resources Engineering, Kunming University of Science and Technology, Kunming 650093, China;
2. College of Resources Environment and Tourism, Capital Normal University, Beijing 100048, China;
3. Key Laboratory of 3D Information Acquisition and Application, MOE, Capital Normal University, Beijing 100048, China

Received:2020-10-23 Revised:2022-05-05 Published:2022-11-30
Supported by:
The National Natural Science Foundation of China (No. 41561082)

摘要/Abstract

摘要： 在半监督语义分割中,主要采用编码-主从解码器结构使无标签样本参与计算以提高分割精度,但编码器的连续下采样操作易丢失浅层细节特征,从而导致地物边界分割不完整。为此,本文提出结合多尺度共享编码的半监督网络架构对航空影像进行语义分割,该网络的编码器采用ResNet-50获取影像浅层特征,并通过在ResNet-50末端嵌入多尺度共享编码模块来链接浅层特征,以构建密集特征金字塔和扩大感受野,从而获取目标地物多尺度细节信息。将本文网络与UNet、DeepLabv3+、FCN监督网络和CCT、XModalNet、VLCNet半监督网络在LandCover.ai和DroneDeploy数据集上分别进行了对比试验和精度评估。结果表明:本文网络在标签数量与精度方面均具有明显优势,对于LandCover.ai数据集,在6000张标签样本和6500张无标签样本的前提下,整体mIoU提升1.15%,对于DroneDeploy数据集,在30张标签样本和5张无标签样本的前提下,整体mIoU提升0.94%,同时显著提升影像地物的分割精度,得到更清晰、完整的地物边界。

关键词: 半监督, 语义分割, 多尺度共享编码器, 主-从解码器, 航空影像

Abstract: In semi-supervised semantic segmentation, the segmentation accuracy of aerial images is mainly improved by using the structure of encoder—master-auxiliary decoder which applies the unlabeled samples to the calculation. However, the loss of shallow detail features which is caused by continuous downsampling in the process of encoding makes the boundary of ground objects incomplete. Therefore, a semi-supervised network combining multi-scale shared encoding is proposed for semantic segmentation of aerial images. The encoder uses ResNet-50 to obtain the shallow features of the image, and links the shallow features by embedding a multi-scale shared coding module at the end of ResNet-50 to build a dense feature pyramid and expand the receptive field, thereby obtaining multi-scale detailed information of the target feature. The effectiveness of the proposed method is verified by compared with UNet, DeepLabv3+, FCN and CCT, XModalNet, VLCNet on the two datasets of LandCover.ai and DroneDeploy, and the result shows that our network has obvious advantages in terms of label number and accuracy. For the LandCover.ai dataset, under the premise of 6000 labeled samples and 6500 unlabeled samples, the overall mIoU increased by 1.15%. For the DroneDeploy dataset, under the premise of 30 labeled samples and 5 unlabeled samples, the overall mIoU increased by 0.94%, while significantly improving the segmentation accuracy of ground objects to obtain a clear and complete ground boundary.

Key words: semi-supervised, semantic segmentation, multiscale shared encoder, master-auxiliary decoder, aerial images

中图分类号:

P227

李佳田, 杨汝春, 姚彦吉, 贺日兴, 阿晓荟, 吕少云. 结合多尺度共享编码的半监督网络航空影像语义分割[J]. 测绘学报, 2022, 51(11): 2355-2364.

LI Jiatian, YANG Ruchun, YAO Yanji, HE Rixing, A Xiaohui, LÜ Shaoyun. Semantic segmentation of aerial image based on semi-supervised network with multi-scale shared coding[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(11): 2355-2364.

参考文献

[1] LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation [C]// Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Boston, MA,USA:IEEE, 2015:3431-3440.
[2] 李道纪, 郭海涛, 卢俊, 等. 遥感影像地物分类多注意力融和U型网络法[J]. 测绘学报, 2020, 49(8): 1051-1064.DOI: 10.11947/j.AGCS.2020.20190407. LI Daoji, GUO Haitao, LU Jun, et al. A remote sensing image classification procedure based on multilevel attention fusion U-Net[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(8): 1051-1064.DOI: 10.11947/j.AGCS.2020.20190407.
[3] HUANG Bo, ZHAO Bei, SONG Yimeng. Urban land-use mapping using a deep convolutional neural network with high spatial resolution multispectral remote sensing imagery[J]. Remote Sensing of Environment, 2018, 214: 73-86.
[4] 党宇, 张继贤, 邓喀中, 等. 基于深度学习AlexNet的遥感影像地表覆盖分类评价研究[J]. 地球信息科学学报, 2017, 19(11): 1530-1537. DANG Yu, ZHANG Jixian, DENG Kazhong, et al. Study on the evaluation of land cover classification using remote sensing images based on AlexNet[J]. Journal of Geo-Information Science, 2017, 19(11): 1530-1537.
[5] 杨军,于茜子.结合空洞卷积的FuseNet变体网络高分辨率遥感影像语义分割[J].武汉大学学报(信息科学版),2022,47(7):1071-1080. YANG Jun, YU Xizi. Semantic segmentation of high-resolution remote sensing images based on improved FuseNet combined with atrous convolution[J]. Geomatics and Information Science of Wuhan University, 2022, 47(7): 1071-1080.
[6] HAZIRBAS C, MA L, DOMOKOS C, et al. Fusenet: incorporating depth into semantic segmentation via fusion-based CNN architecture[C]//Proceedings of 2016 Asian Conference on Computer Vision. Cham,Gernamy:Springer,2016: 213-228.
[7] AUDEBERT N, SAUX B L, LEFÈVRE S. Semantic segmentation of earth observation data using multimodal and multi-scale deep networks[C]//Proceedings of 2016 Asian Conference on Computer Vision. Cham,Gernamy:Springer,2016:180-196.
[8] ZUO Zongcheng, ZHANG Wen, ZHANG Dongying. A remote sensing image semantic segmentation method by combining deformable convolution with conditional random fields[J]. Journal of Geodesy and Geoinformation Science, 2020, 3(3): 39-49.
[9] INTERDONATO R, IENCO D, GAETANO R, et al. DuPLO: a DUal view point deep learning architecture for time series classification[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 149: 91-104.
[10] HU Ting, HUANG Xin, LI Jiayi, et al. A semi-supervised approach towards land cover mapping with Sentinel-2 desnse time-series imagery[C]//Proceedings of 2019 IEEE International Geoscience and Remote Sensing Symposium. Yokohama, Japan: IEEE, 2019: 2423-2426.
[11] BREIMAN L. Random forests[J].Machine learning, 2001, 45(1): 5-32.
[12] 冯文卿, 眭海刚, 涂继辉, 等. 高分辨率遥感影像的随机森林变化检测方法[J]. 测绘学报, 2017, 46(11): 1880-1890.DOI: 10.11947/j.AGCS.2017.20170074. FENG Wenqing, SUI Haigang, TU Jihui, et al. Change detection method for high resolution remote sensing images using random forest[J]. Acta Geodaetica et Cartographica Sinica, 2017, 46(11): 1880-1890.DOI: 10.11947/j.AGCS.2017.20170074.
[13] HASTIE T, ROSSET S, ZHU J, et al. Multi-class AdaBoost[J]. Statistics and its Interface, 2009, 2(3): 349-360.
[14] CAI T T, ZHOU W X. Matrix completion via max-norm constrained optimization[J]. Electronic Journal of Statistics, 2016, 10(1): 1493-1525
[15] SUN He, REN Jinchang, ZHAO Huimin, et al. Superpixel based feature specific sparse representation for spectral-spatial classification of hyperspectral images[J]. Remote Sensing, 2019, 11(5): 536.
[16] HONG Danfeng, YOKOYA N, XIA Guisong, et al. X-ModalNet: a semi-supervised deep cross-modal network for classification of remote sensing data[J].ISPRS Journal of Photogrammetry and Remote Sensing,2020, 167: 12-23.
[17] ZHANG B, ZHANG Y, LI Y, et al. Semi-supervised semantic segmentation network via learning consistency for remote sensing land-cover classification[J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2020,2:609-615.
[18] 耿艳磊, 陶超, 沈靖, 等. 高分辨率遥感影像语义分割的半监督全卷积网络法[J]. 测绘学报, 2020, 49(4): 499-508.DOI: 10.11947/j.AGCS.2020.20190044. GENG Yanlei, TAO Chao, SHEN Jing, et al. High-resolution remote sensing image semantic segmentation based on semi-supervised full convolution network method[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(4): 499-508.DOI: 10.11947/j.AGCS.2020.20190044.
[19] OUALI Y, HUDELOT C, TAMI M. Semi-supervised semantic segmentation with cross-consistency training[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, USA: IEEE,2020: 12671-12681.
[20] ZHOU Jingchun, HAO Mingliang, ZHANG Dehuan, et al. Fusion PSPnet image segmentation based method for multi-focus image fusion[J]. IEEE Photonics Journal, 2019, 11(6): 1-12.
[21] BOGUSZEWSKI A, BATORSKI D, ZIEMBA-JANKOWSKA N, et al. LandCover.ai: dataset for automatic mapping of buildings, woodlands, water and roads from aerial imagery[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Nashville, TN, USA:IEEE,2021: 1102-1110.
[22] GONG Han, COSKER D. Interactive removal and ground truth for difficult shadow scenes[J]. Journal of the Optical Society of America A, Optics, Image Science, and Vision, 2016, 33(9): 1798-1811.
[23] CHEN L C, ZHU Yukun, PAPANDREOU G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation [C]//Proceedings of the 15th European Conference on Computer Vision(ECCV). Munich, Germany: Springer, 2018, 833-851.
[24] RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation [C]//Proceedings of 2015 International Conference on Medical Image Computing and Computer-assisted Intervention. Munich, Germany:Springer,2015: 234-241.
[25] 赵斌, 王春平, 付强, 等. 基于深度注意力机制的多尺度红外行人检测[J]. 光学学报, 2020, 40(5): 47-58. ZHAO Bin, WANG Chunping, FU Qiang, et al. Multi-scale infrared pedestrian detection based on deep attention mechanism[J]. Acta Optica Sinica, 2020, 40(5): 47-58.
[26] BLAGA B C Z, NEDEVSCHI S. A critical evaluation of aerial datasets for semantic segmentation[C]//Proceedings of the 16th IEEE International Conference on Intelligent Computer Communication and Processing(ICCP). Cluj-Napoca, Romania: IEEE, 2020:3-5.

结合多尺度共享编码的半监督网络航空影像语义分割

Semantic segmentation of aerial image based on semi-supervised network with multi-scale shared coding

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	白坤, 慕晓冬, 陈雪冰, 朱永清, 尤轩昂. 融合半监督学习的无监督遥感影像场景分类[J]. 测绘学报, 2022, 51(5): 691-702.
[2]	金飞, 官恺, 刘智, 韩佳容, 芮杰, 李庆高. 无监督密集匹配特征提取网络性能分析[J]. 测绘学报, 2022, 51(3): 426-436.
[3]	杨秋莲, 刘艳飞, 丁乐乐, 孟凡效. 高分影像场景分类的半监督深度卷积神经网络学习方法[J]. 测绘学报, 2021, 50(7): 930-938.
[4]	蒋腾平, 王永君, 张林淇, 梁冲, 孙剑. 融合CNN和MRF的激光点云层次化语义分割方法[J]. 测绘学报, 2021, 50(2): 215-225.
[5]	邵晓航, 吴杭彬, 刘春, 陈晨, 蔡天池, 程帆瑾. 语素关联约束的动态环境视觉定位优化[J]. 测绘学报, 2021, 50(11): 1478-1486.
[6]	李道纪, 郭海涛, 卢俊, 赵传, 林雨准, 余东行. 遥感影像地物分类多注意力融和U型网络法[J]. 测绘学报, 2020, 49(8): 1051-1064.
[7]	耿艳磊, 陶超, 沈靖, 邹峥嵘. 高分辨率遥感影像语义分割的半监督全卷积网络法[J]. 测绘学报, 2020, 49(4): 499-508.
[8]	刘嗣超, 武鹏达, 赵占杰, 李成名. 交通监控视频图像语义分割及其拼接方法[J]. 测绘学报, 2020, 49(4): 522-532.
[9]	王忠武, 王志盼, 尤淑撑, 雷帆, 曹里, 杨凯钧. Landsat影像冰川提取的上下文感知语义分割网络法[J]. 测绘学报, 2020, 49(12): 1575-1582.
[10]	徐旺, 游雄, 张威巍, 邓晨. 2D地图的建筑物场景结构提取方法及其在城市增强现实中的应用[J]. 测绘学报, 2020, 49(12): 1619-1629.
[11]	李雪, 张力, 王庆栋, 艾海滨. 多时相遥感影像语义分割色彩一致性对抗网络[J]. 测绘学报, 2020, 49(11): 1473-1484.
[12]	刘瑾, 季顺平. 基于深度学习的航空遥感影像密集匹配[J]. 测绘学报, 2019, 48(9): 1141-1150.
[13]	左宗成, 张文, 张东映. 融合可变形卷积与条件随机场的遥感影像语义分割方法[J]. 测绘学报, 2019, 48(6): 718-726.
[14]	季顺平, 魏世清. 遥感影像建筑物提取的卷积神经元网络与开源数据集方法[J]. 测绘学报, 2019, 48(4): 448-459.
[15]	贺浩, 王仕成, 杨东方, 王舒洋, 刘星. 基于Encoder-Decoder网络的遥感影像道路提取方法[J]. 测绘学报, 2019, 48(3): 330-338.