融合可变形卷积与条件随机场的遥感影像语义分割方法

doi:10.11947/j.AGCS.2019.20170740

测绘学报 ›› 2019, Vol. 48 ›› Issue (6): 718-726.doi: 10.11947/j.AGCS.2019.20170740

融合可变形卷积与条件随机场的遥感影像语义分割方法

左宗成^1,2, 张文¹, 张东映³

1. 武汉大学遥感信息工程学院, 湖北武汉 430079;
2. 欧特克(中国)软件研发有限公司, 上海 200122;
3. 郑州大学水利与环境学院, 河南郑州 450002

收稿日期:2017-12-23 修回日期:2018-08-01 出版日期:2019-06-20 发布日期:2019-07-09
通讯作者: 张文 E-mail:wen_zhang@whu.edu.cn
作者简介:左宗成(1988-),男,硕士,工程师,研究方向为高分辨遥感影像处理及信息提取、模式识别与机器学习。E-mail:jason.zuo@autodesk.com
基金资助:
国家重点研发计划（2017YFC0405806）

A remote sensing image semantic segmentation method by combining deformable convolution with conditional random fields

ZUO Zongcheng^1,2, ZHANG Wen¹, ZHANG Dongying³

1. School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China;
2. Autodesk(China) Software Research and Development Co. Ltd., Shanghai 200122, China;
3. College of Water Conservancy & Environmental Engineering, Zhengzhou University, Zhengzhou 450002, China

Received:2017-12-23 Revised:2018-08-01 Online:2019-06-20 Published:2019-07-09
Supported by:
The National Key Research and Development Program of China (No.2017YFC0405806)

摘要/Abstract

摘要： 当前，深度卷积神经网络在遥感影像语义分割领域取得了长足的发展。标准的卷积神经网络由于卷积核的几何形状是固定的，导致对几何变换的模拟能力受到限制。本文引入一种可变形卷积来增强卷积网络对空间变换的适应能力。由于神经网络架构中使用了池化层操作，这会导致在输出层未能充分地对局部对象进行准确的分割。为了克服这种特性，本文将神经网络输出层的粗糙预测分割结果通过全连接的条件随机场来进行处理，以此来提高对影像细节的分割能力。本文方法易于采用标准的反向传播算法进行端到端的方式训练。ISPRS数据集上的测试试验结果表明本文方法可以有效地克服遥感影像中分割对象的复杂结构对分割结果的影响，并在该数据集上获得了当前最好的语义分割结果。

关键词: 高分辨率遥感影像, 语义分割, 可变形卷积网络, 条件随机场

Abstract: Currently, deep convolutional neural networks have made great progress in the field of semantic segmentation. Because of the fixed convolution kernel geometry, standard convolution neural networks have been limited the ability to simulate geometric transformations. Therefore, a deformable convolution is introduced to enhance the adaptability of convolutional networks to spatial transformation. Considering that the deep convolutional neural networks cannot adequately segment the local objects at the output layer due to using the pooling layers in neural networks architecture. To overcome this shortcoming, the rough prediction segmentation results of the neural network output layer will be processed by fully connected conditional random fields to improve the ability of image segmentation. The proposed method can easily be trained by end-to-end using standard backpropagation algorithms. Finally, the proposed method is tested on the ISPRS dataset. The results show that the proposed method can effectively overcome the influence of the complex structure of the segmentation object and obtain state-of-the-art accuracy on the ISPRS Vaihingen 2D semantic labeling dataset.

Key words: high-resolution remote sensing image, semantic segmentation, deformable convolution network, conditions random fields

中图分类号:

P237

左宗成, 张文, 张东映. 融合可变形卷积与条件随机场的遥感影像语义分割方法[J]. 测绘学报, 2019, 48(6): 718-726.

ZUO Zongcheng, ZHANG Wen, ZHANG Dongying. A remote sensing image semantic segmentation method by combining deformable convolution with conditional random fields[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(6): 718-726.

参考文献

[1] 刘婧, 李培军. 结合结构和光谱特征的高分辨率影像分割方法[J]. 测绘学报, 2014, 43(5):466-473. DOI:10.13485/j.cnki.112089.2014.0087. LIU Jing, LI Peijun. A high resolution image segmentation method by combined structural and spectral characteristics[J]. Acta Geodaetica et Cartographica Sinica, 2014, 43(5):466-473. DOI:10.13485/j.cnki.11-2089.2014.0087.
[2] 周成虎, 骆剑承. 高分辨率卫星遥感影像地学计算[M]. 北京:科学出版社, 2009. ZHOU Chenghu, LUO Jiancheng. Geo-computing of high resolution satellite remote sensing image[M]. Beijing:Science Press, 2009.
[3] LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, USA:IEEE, 2015:3431-3440.
[4] WANG Hongzhen, WANG Ying, ZHANG Qian, et al. Gated convolutional neural network for semantic segmentation in high-resolution images[J]. Remote Sensing, 2017, 9(5):446.
[5] GARCIA-GARCIA A, ORTS-ESCOLANO S, OPREA S, et al. A review on deep learning techniques applied to semantic segmentation[J]. arXiv:1704.06857, 2017.
[6] HUBEL D H. The visual cortex of the brain[J]. Scientific American, 1963, 209:54-62. DOI:10.1038/scientificamerican1163-54.
[7] KRÄHENBVHL P, KOLTUN V. Efficient inference in fully connected crfs with gaussian edge potentials[J]. arXiv:1210.5644, 2012:109-117.
[8] SHOTTON J, WINN J, ROTHER C, et al. Textonboost for image understanding:multi-class object recognition and segmentation by jointly modeling texture, layout, and context[J]. International Journal of Computer Vision, 2009, 81(1):2-23.
[9] DAI Jifeng, QI Haozhi, XIONG Yuwen, et al. Deformable convolutional networks[J]. arXiv:1703.06211, 2017:764-773.
[10] LUO Wenjie, LI Yujia, URTASUN R, et al. Understanding the effective receptive field in deep convolutional neural networks[J]. arXiv:1701.04128, 2017.
[11] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv:1409.1556, 2014.
[12] HOLSCHNEIDER M, KRONLAND-MARTINET R, MORLET J, et al. A real-time algorithm for signal analysis with the help of the wavelet transform[M]. COMBES J M, GROSSMANN A, TCHAMITCHIAN P. Wavelets:Time-Frequency Methods and Phase Space. Berlin:Springer, 1990:286-297.
[13] DENG Jia, DONG Wei, SOCHER R, et al. Imagenet:a large-scale hierarchical image database[C]//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, FL, USA:IEEE, 2009.
[14] DAI Jifeng, LI Yi, HE Kaiming, et al. R-FCN:object detection via region-based fully convolutional networks[C]//Proceedings of the 30th International Conference on Neural Information Processing Systems. Barcelona, Spain:ACM, 2016:379-387.
[15] BADRINARAYANAN V, KENDALL A, CIPOLLA R. SegNet:a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12):2481-2495.
[16] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. Semantic image segmentation with deep convolutional nets and fully connected CRFs[C]//Proceedings of the International Conference on Learning Representations. San Diego, CA:Computational and Biological Learning Society, 2015.
[17] CHATFIELD K, SIMONYAN K, VEDALDI A, et al. Return of the devil in the details:delving deep into convolutional nets[C]//Proceedings of the British Machine Vision Conference. Dundee, Britain:BMVA Press, 2014.
[18] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv:1409.1556, 2014.
[19] IOFFE S, SZEGEDY C. Batch normalization:accelerating deep network training by reducing internal covariate shift[C]//Proceedings of the 32nd International Conference on Machine Learning. Lille, France:JMLR, 2015:448-456.
[20] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Delving deep into rectifiers:surpassing human-level performance on imagenet classification[C]//Proceedings of the IEEE International Conference on Computer Vision. Santiago, Chile:IEEE, 2015:1026-1034.
[21] CLEVERT D A, UNTERTHINER T, HOCHREITER S. Fast and accurate deep network learning by exponential linear units (ELUs)[C]//Proceedings of the International Conference on Learning Representations. San Diego, CA:Computational and Biological Learning Society, 2015.
[22] EIGEN D, FERGUS R. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture[C]//Proceedings of the IEEE International Conference on Computer Vision. Santiago, Chile:IEEE, 2015:2650-2658.
[23] MOSTAJABI M, YADOLLAHPOUR P, SHAKHNAROVICH G. Feedforward semantic segmentation with zoom-out features[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, USA:IEEE, 2015:3376-3385.
[24] ROTHER C, KOLMOGOROV V, BLAKE A. Grabcut:interactive foreground extraction using iterated graph cuts[C]//Proceedings of the ACM SIGGRAPH 2004. Los Angeles, California:ACM, 2004:309-314.
[25] KOHLI P, LADICKY L, TORR P H S. Robust higher order potentials for enforcing label consistency[C]//Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage, AK, USA:IEEE, 2009.
[26] KRÄHENBÜHL P, KOLTUN V. Efficient inference in fully connected CRFs with Gaussian edge potentials[C]//Proceedings of the 25th annual conference on Neural Information Processing Systems. Granada:NIF, 2011.
[27] ADAMS A, BAEK J, DAVIS M A. Fast high-dimensional filtering using the permutohedral lattice[J]. Computer Graphics Forum, 2010,29(2):753-762.
[28] GERKE M, ROTTENSTEINER F, WEGNER J D, et al. ISPRS semantic labeling contest[C]//Proceedings of PCV-Photogrammetric Computer Vision.[S.l.]:ISPRS, 2014.
[29] GERKE M. Use of the stair vision library within the ISPRS 2D semantic labeling benchmark (Vaihingen)[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Boston, MA, USA:IEEE, 2015.
[30] PAISITKRIANGKRAI S, SHERRAH J, JANNEY P, et al. Effective semantic pixel labelling with convolutional networks and conditional random fields[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Boston, MA, USA:IEEE, 2015:36-43.
[31] AUDEBERT N, LE SAUX B, LEFÈVRE S. Semantic segmentation of earth observation data using multimodal and multi-scale deep networks[M]//LAI S H, LEPETIT V, NISHINO K, et al. Computer Vision-ACCV 2016. Cham:Springer, 2017.
[32] ZHOU Hao, ZHANG Jun, LEI Jun, et al. Image semantic segmentation based on FCN-CRF model[C]//Proceedings of International Conference on Image, Vision and Computing. Portsmouth, UK:IEEE, 2016:9-14.

融合可变形卷积与条件随机场的遥感影像语义分割方法

A remote sensing image semantic segmentation method by combining deformable convolution with conditional random fields

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	何直蒙, 丁海勇, 安炳琪. 高分辨率遥感影像建筑物提取的空洞卷积E-Unet算法[J]. 测绘学报, 2022, 51(3): 457-467.
[2]	张玉鑫, 颜青松, 邓非. 高分辨率遥感影像建筑物提取多路径RSU网络法[J]. 测绘学报, 2022, 51(1): 135-144.
[3]	蒋腾平, 王永君, 张林淇, 梁冲, 孙剑. 融合CNN和MRF的激光点云层次化语义分割方法[J]. 测绘学报, 2021, 50(2): 215-225.
[4]	邵晓航, 吴杭彬, 刘春, 陈晨, 蔡天池, 程帆瑾. 语素关联约束的动态环境视觉定位优化[J]. 测绘学报, 2021, 50(11): 1478-1486.
[5]	张涛, 丁乐乐, 史芙蓉. 高分辨率遥感影像城中村提取的景观语义指数方法[J]. 测绘学报, 2021, 50(1): 97-104.
[6]	李道纪, 郭海涛, 卢俊, 赵传, 林雨准, 余东行. 遥感影像地物分类多注意力融和U型网络法[J]. 测绘学报, 2020, 49(8): 1051-1064.
[7]	耿艳磊, 陶超, 沈靖, 邹峥嵘. 高分辨率遥感影像语义分割的半监督全卷积网络法[J]. 测绘学报, 2020, 49(4): 499-508.
[8]	刘嗣超, 武鹏达, 赵占杰, 李成名. 交通监控视频图像语义分割及其拼接方法[J]. 测绘学报, 2020, 49(4): 522-532.
[9]	魏立飞, 余铭, 钟燕飞, 袁自然, 黄灿. 空-谱融合的条件随机场高光谱影像分类方法[J]. 测绘学报, 2020, 49(3): 343-354.
[10]	王忠武, 王志盼, 尤淑撑, 雷帆, 曹里, 杨凯钧. Landsat影像冰川提取的上下文感知语义分割网络法[J]. 测绘学报, 2020, 49(12): 1575-1582.
[11]	徐旺, 游雄, 张威巍, 邓晨. 2D地图的建筑物场景结构提取方法及其在城市增强现实中的应用[J]. 测绘学报, 2020, 49(12): 1619-1629.
[12]	李雪, 张力, 王庆栋, 艾海滨. 多时相遥感影像语义分割色彩一致性对抗网络[J]. 测绘学报, 2020, 49(11): 1473-1484.
[13]	赵泉华, 谢凯浪, 王光辉, 李玉. 全卷积网络和条件随机场相结合的全极化SAR土地覆盖分类[J]. 测绘学报, 2020, 49(1): 65-78.
[14]	季顺平, 魏世清. 遥感影像建筑物提取的卷积神经元网络与开源数据集方法[J]. 测绘学报, 2019, 48(4): 448-459.
[15]	贺浩, 王仕成, 杨东方, 王舒洋, 刘星. 基于Encoder-Decoder网络的遥感影像道路提取方法[J]. 测绘学报, 2019, 48(3): 330-338.