多时相遥感影像语义分割色彩一致性对抗网络

doi:10.11947/j.AGCS.2020.20190439

摘要/Abstract

摘要： 利用深度卷积神经网络智能化地提取遥感图像中的建筑物对于数字城市构建、灾害侦查、土地管理等具有重要意义。多时相遥感图像之间的色彩差异会导致建筑物语义分割模型泛化能力下降。针对此，本文提出了注意力引导的色彩一致生成对抗网络（attention-guided color consistency adversarial network，ACGAN）。该算法以参考色彩风格图像及相同区域、不同时相的待纠正图像作为训练集，采用加入了U型注意力机制的循环一致生成对抗网络训练得到色彩一致模型。在预测阶段，该模型将待纠正图像的色调转换为参考色彩风格图像的色调，这一阶段基于深度学习模型的推理能力，而不再需要相应的参考色彩风格图像。为了验证算法的有效性，首先，将本文算法与传统的图像处理算法及其他循环一致生成对抗网络做了对比试验。结果表明，ACGAN色彩一致后的图像与参考色彩风格图像的色调更加相似。其次，将以上不同的色彩一致性算法处理后的结果图像进行建筑物语义分割试验，证明本文方法更加有利于多时相遥感图像语义分割模型泛化能力的提升。

关键词: 多时相遥感图像, 色彩一致性, 生成对抗网络, 注意力机制, 语义分割

Abstract: Using deep convolutional neural network (CNN) to intelligently extract buildings from remote sensing images is of great significance for digital city construction, disaster detection and land management. The color difference between multi-temporal remote sensing images will lead to the decrease of generalization ability of building semantic segmentation model. In view of this, this paper proposes the attention-guided color consistency adversarial network (ACGAN). The algorithm takes the reference color style images and the images to be corrected in the same area and different phases as the training set and adopts the consistency adversarial network with the U-shaped attention mechanism to train the color consistency model. In the prediction stage, this model converts the hue of the images to that of the reference color style image, which is based on the reasoning ability of the deep learning model, instead of the corresponding reference color style image. This model transforms the hue of the images to be corrected into that of the reference color style images. This stage is based on the reasoning ability of the deep learning model, and the corresponding reference color style image is no longer needed. In order to verify the effectiveness of the algorithm, firstly, we compare the algorithm of this paper with the traditional image processing algorithm and other consistency adversarial network. The results show that the images after ACGAN color consistency processing are more similar to that of the reference color style images. Secondly, we carried out the building semantic segmentation experiment on the images processed by the above different color consistency algorithms, which proved that the method in this paper is more conducive to the impro-vement of the generalization ability of multi-temporal remote sensing image semantic segmentation model.

Key words: multi-temporal remote sensing imagery, color consistency, generative adversarial networks, semantic segmentation, attention mechanism

中图分类号:

P237

李雪, 张力, 王庆栋, 艾海滨. 多时相遥感影像语义分割色彩一致性对抗网络[J]. 测绘学报, 2020, 49(11): 1473-1484.

LI Xue, ZHANG Li, WANG Qingdong, AI Haibin. Multi-temporal remote sensing imagery semantic segmentation color consistency adversarial network[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(11): 1473-1484.

参考文献

[1] JI Shunping, WEI Shiqing, LU Meng. Fully convolutional networks for multisource building extraction from an open aerial and satellite imagery data set[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(1):574-586.
[2] 左宗成, 张文, 张东映. 融合可变形卷积与条件随机场的遥感影像语义分割方法[J]. 测绘学报, 2019, 48(6):718-726. DOI:10.11947/j.AGCS.2019.20170740. ZUO Zongcheng, ZHANG Wen, ZHANG Dongying. A remote sensing image semantic segmentation method by combining deformable convolution with conditional random fields[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(6):718-726. DOI:10.11947/j.AGCS.2019.20170740.
[3] ZHANG Hongyi, CISSE M, DAUPHIN Y N, et al. Mixup:beyond empirical risk minimization[J]. arXiv preprint arXiv:1710.09412, 2017.
[4] INOUE H. Data augmentation by pairing samples for images classification[J]. arXiv preprint arXiv:1801.02929, 2018.
[5] CUBUK E D, ZOPH B, MANE D, et al. AutoAugment:learning augmentation strategies from data[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA:IEEE, 2019:113-123.
[6] CANTY M J, NIELSEN A A. Automatic radiometric normalization of multitemporal satellite imagery with the iteratively re-weighted MAD transformation[J]. Remote Sensing of Environment, 2008, 112(3):1025-1036.
[7] HUANG T W, CHEN H T. Landmark-based sparse color representations for color transfer[C]//Proceedings of 2009 IEEE International Conference on Computer Vision. Kyoto, Japan:IEEE, 2009:199-204.
[8] 李德仁, 王密, 潘俊. 光学遥感影像的自动匀光处理及应用[J]. 武汉大学学报(信息科学版), 2006, 31(9):753-756. LI Deren, WANG Mi, PAN Jun. Auto-dodging processing and its application for optical RS images[J]. Geomatics and Information Science of Wuhan University, 2006, 31(9):753-756.
[9] HORN B K P, WOODHAM R J. Destriping Landsat MSS images by histogram modification[J]. Computer Graphics and Image Processing, 1979, 10(1):69-83.
[10] PITIÉ F, KOKARAM A. The linear Monge-Kantorovitch colour mapping for example-based colour transfer[C]//Proceedings of the 4th European Conference on Visual Media Production. London:IET, 2007:27-28.
[11] YOO J D, PARK M K, CHO J H, et al. Local color transfer between images using dominant colors[J]. Journal of Electronic Imaging, 2013, 22(3):033003.
[12] MOULON P, DUISIT B, MONASSE P. Global multiple-view color consistency[J]. International Conference on Visual Media Production, 2013.
[13] HACOHEN Y, SHECHTMAN E, GOLDMAN D B, et al. Non-rigid dense correspondence with applications for image enhancement[J]. ACM Transactions on Graphics, 2011, 30(4):70.
[14] GATYS L A, ECKER A S, BETHGE M. A neural algorithm of artistic style[C]. arXiv:1508.06576, 2016:326.
[15] GATYS L A, ECKER A S, BETHGE M, et al. Image style transfer using convolutional neural networks[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV:IEEE, 2016:2414-2423.
[16] LI Chuan, WAND M. Combining markov random fields and convolutional neural networks for image synthesis[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV:IEEE, 2016:2479-2486.
[17] LUAN Fujun, PARIS S, SHECHTMAN E, et al. Deep photo style transfer[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI:IEEE, 2017:6997-7005.
[18] WANG Li, WANG Zhao, YANG Xiaosong, et al. Photographic style transfer[J]. The Visual Computer, 2020, 36(2):317-331.
[19] LI Yijun, LIU Mingyu, LI Xueting, et al. A closed-form solution to photorealistic image stylization[C]//Proceedings of the 16th European Conference on Computer Vision. Munich:Springer, 2018:453-468.
[20] JOHNSON J, ALAHI A, LI Feifei. Perceptual losses for real-time style transfer and super-resolution[C]//Proceedings of the 14th European Conference on Computer Vision. Amsterdam:Springer, 2016:694-711.
[21] ZHU Junyan, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy:IEEE, 2017:2242-2251.
[22] MNIH V, HEESS N, GRAVES A. Recurrent models of visual attention[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal:NIPS, 2014:2204-2212.
[23] KIM J, KIM M, KANG H, et al. U-GAT-IT:unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation[J]. arXiv preprint arXiv:1907.10830, 2019.
[24] PUMAROLA A, AGUDO A, MARTINEZ A M, et al. Ganimation:anatomically-aware facial animation from a single image[C]//Proceedings of the 16th European Conference on Computer Vision. Munich, Germany:Springer, 2018:818-833.
[25] MEJJATI Y A, RICHARDT C, TOMPKIN J, et al. Unsupervised attention-guided image-to-image translation[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. Montreal:NIPS, 2018:3693-3703.
[26] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV:IEEE, 2016:770-778.
[27] ISOLA P, ZHU Junyan, ZHOU Tinghui, et al. Image-to-image translation with conditional adversarial networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI:IEEE, 2017:1125-1134.
[28] 邱锡鹏. 神经网络与深度学习[EB/OL].[2017-04-21]. https://nndl.github.io/. QIU Xipeng. Neural network and deep learning[EB/OL].[2017-04-21]. https://nndl.github.io/.
[29] PAUSANG S M C, FAJARDO-LIM Y. Nash equilibria in a multiple stage inspection game[J]. Electronic Notes in Discrete Mathematics, 2016, 56:49-57.
[30] 季顺平, 魏世清. 遥感影像建筑物提取的卷积神经元网络与开源数据集方法[J]. 测绘学报, 2019, 48(4):448-459. DOI:10.11947/j.AGCS.2019.20180206. JI Shunping, WEI Shiqing. Building extraction via convolutional neural networks from an open remote sensing building dataset[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(4):448-459. DOI:10.11947/j.AGCS.2019.20180206.
[31] BADRINARAYANAN V, KENDALL A, CIPOLLA R. U-Net:a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12):2481-2495.
[32] LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston:IEEE, 2015:3431-3440.