A remote sensing image semantic segmentation method by combining deformable convolution with conditional random fields

doi:10.11947/j.AGCS.2019.20170740

Abstract

Abstract: Currently, deep convolutional neural networks have made great progress in the field of semantic segmentation. Because of the fixed convolution kernel geometry, standard convolution neural networks have been limited the ability to simulate geometric transformations. Therefore, a deformable convolution is introduced to enhance the adaptability of convolutional networks to spatial transformation. Considering that the deep convolutional neural networks cannot adequately segment the local objects at the output layer due to using the pooling layers in neural networks architecture. To overcome this shortcoming, the rough prediction segmentation results of the neural network output layer will be processed by fully connected conditional random fields to improve the ability of image segmentation. The proposed method can easily be trained by end-to-end using standard backpropagation algorithms. Finally, the proposed method is tested on the ISPRS dataset. The results show that the proposed method can effectively overcome the influence of the complex structure of the segmentation object and obtain state-of-the-art accuracy on the ISPRS Vaihingen 2D semantic labeling dataset.

Key words: high-resolution remote sensing image, semantic segmentation, deformable convolution network, conditions random fields

CLC Number:

P237

ZUO Zongcheng, ZHANG Wen, ZHANG Dongying. A remote sensing image semantic segmentation method by combining deformable convolution with conditional random fields[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(6): 718-726.

References

[1] 刘婧, 李培军. 结合结构和光谱特征的高分辨率影像分割方法[J]. 测绘学报, 2014, 43(5):466-473. DOI:10.13485/j.cnki.112089.2014.0087. LIU Jing, LI Peijun. A high resolution image segmentation method by combined structural and spectral characteristics[J]. Acta Geodaetica et Cartographica Sinica, 2014, 43(5):466-473. DOI:10.13485/j.cnki.11-2089.2014.0087.
[2] 周成虎, 骆剑承. 高分辨率卫星遥感影像地学计算[M]. 北京:科学出版社, 2009. ZHOU Chenghu, LUO Jiancheng. Geo-computing of high resolution satellite remote sensing image[M]. Beijing:Science Press, 2009.
[3] LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, USA:IEEE, 2015:3431-3440.
[4] WANG Hongzhen, WANG Ying, ZHANG Qian, et al. Gated convolutional neural network for semantic segmentation in high-resolution images[J]. Remote Sensing, 2017, 9(5):446.
[5] GARCIA-GARCIA A, ORTS-ESCOLANO S, OPREA S, et al. A review on deep learning techniques applied to semantic segmentation[J]. arXiv:1704.06857, 2017.
[6] HUBEL D H. The visual cortex of the brain[J]. Scientific American, 1963, 209:54-62. DOI:10.1038/scientificamerican1163-54.
[7] KRÄHENBVHL P, KOLTUN V. Efficient inference in fully connected crfs with gaussian edge potentials[J]. arXiv:1210.5644, 2012:109-117.
[8] SHOTTON J, WINN J, ROTHER C, et al. Textonboost for image understanding:multi-class object recognition and segmentation by jointly modeling texture, layout, and context[J]. International Journal of Computer Vision, 2009, 81(1):2-23.
[9] DAI Jifeng, QI Haozhi, XIONG Yuwen, et al. Deformable convolutional networks[J]. arXiv:1703.06211, 2017:764-773.
[10] LUO Wenjie, LI Yujia, URTASUN R, et al. Understanding the effective receptive field in deep convolutional neural networks[J]. arXiv:1701.04128, 2017.
[11] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv:1409.1556, 2014.
[12] HOLSCHNEIDER M, KRONLAND-MARTINET R, MORLET J, et al. A real-time algorithm for signal analysis with the help of the wavelet transform[M]. COMBES J M, GROSSMANN A, TCHAMITCHIAN P. Wavelets:Time-Frequency Methods and Phase Space. Berlin:Springer, 1990:286-297.
[13] DENG Jia, DONG Wei, SOCHER R, et al. Imagenet:a large-scale hierarchical image database[C]//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, FL, USA:IEEE, 2009.
[14] DAI Jifeng, LI Yi, HE Kaiming, et al. R-FCN:object detection via region-based fully convolutional networks[C]//Proceedings of the 30th International Conference on Neural Information Processing Systems. Barcelona, Spain:ACM, 2016:379-387.
[15] BADRINARAYANAN V, KENDALL A, CIPOLLA R. SegNet:a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12):2481-2495.
[16] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. Semantic image segmentation with deep convolutional nets and fully connected CRFs[C]//Proceedings of the International Conference on Learning Representations. San Diego, CA:Computational and Biological Learning Society, 2015.
[17] CHATFIELD K, SIMONYAN K, VEDALDI A, et al. Return of the devil in the details:delving deep into convolutional nets[C]//Proceedings of the British Machine Vision Conference. Dundee, Britain:BMVA Press, 2014.
[18] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv:1409.1556, 2014.
[19] IOFFE S, SZEGEDY C. Batch normalization:accelerating deep network training by reducing internal covariate shift[C]//Proceedings of the 32nd International Conference on Machine Learning. Lille, France:JMLR, 2015:448-456.
[20] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Delving deep into rectifiers:surpassing human-level performance on imagenet classification[C]//Proceedings of the IEEE International Conference on Computer Vision. Santiago, Chile:IEEE, 2015:1026-1034.
[21] CLEVERT D A, UNTERTHINER T, HOCHREITER S. Fast and accurate deep network learning by exponential linear units (ELUs)[C]//Proceedings of the International Conference on Learning Representations. San Diego, CA:Computational and Biological Learning Society, 2015.
[22] EIGEN D, FERGUS R. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture[C]//Proceedings of the IEEE International Conference on Computer Vision. Santiago, Chile:IEEE, 2015:2650-2658.
[23] MOSTAJABI M, YADOLLAHPOUR P, SHAKHNAROVICH G. Feedforward semantic segmentation with zoom-out features[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, USA:IEEE, 2015:3376-3385.
[24] ROTHER C, KOLMOGOROV V, BLAKE A. Grabcut:interactive foreground extraction using iterated graph cuts[C]//Proceedings of the ACM SIGGRAPH 2004. Los Angeles, California:ACM, 2004:309-314.
[25] KOHLI P, LADICKY L, TORR P H S. Robust higher order potentials for enforcing label consistency[C]//Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage, AK, USA:IEEE, 2009.
[26] KRÄHENBÜHL P, KOLTUN V. Efficient inference in fully connected CRFs with Gaussian edge potentials[C]//Proceedings of the 25th annual conference on Neural Information Processing Systems. Granada:NIF, 2011.
[27] ADAMS A, BAEK J, DAVIS M A. Fast high-dimensional filtering using the permutohedral lattice[J]. Computer Graphics Forum, 2010,29(2):753-762.
[28] GERKE M, ROTTENSTEINER F, WEGNER J D, et al. ISPRS semantic labeling contest[C]//Proceedings of PCV-Photogrammetric Computer Vision.[S.l.]:ISPRS, 2014.
[29] GERKE M. Use of the stair vision library within the ISPRS 2D semantic labeling benchmark (Vaihingen)[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Boston, MA, USA:IEEE, 2015.
[30] PAISITKRIANGKRAI S, SHERRAH J, JANNEY P, et al. Effective semantic pixel labelling with convolutional networks and conditional random fields[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Boston, MA, USA:IEEE, 2015:36-43.
[31] AUDEBERT N, LE SAUX B, LEFÈVRE S. Semantic segmentation of earth observation data using multimodal and multi-scale deep networks[M]//LAI S H, LEPETIT V, NISHINO K, et al. Computer Vision-ACCV 2016. Cham:Springer, 2017.
[32] ZHOU Hao, ZHANG Jun, LEI Jun, et al. Image semantic segmentation based on FCN-CRF model[C]//Proceedings of International Conference on Image, Vision and Computing. Portsmouth, UK:IEEE, 2016:9-14.