Unsupervised domain adaptation alignment method for cross-domain semantic segmentation of remote sensing images

doi:10.11947/j.AGCS.2023.20220483

Abstract

Abstract: Deep learning models rely on a large number of homogenous labeled samples, i.e., limiting the training and testing data to obey the same data distribution. However, when facing large-scale and diverse remote sensing data, it is difficult to guarantee the requirement of homogeneous distribution among data, and the segmentation accuracy of deep learning models decreases significantly. To address this problem, this paper proposes an unsupervised domain adaptation (UDA) method for semantic segmentation of remote sensing images. When the distributions of training data (source domain) and testing data (target domain) are different, the proposed method improves the accuracy of semantic segmentation in the target domain by training deep learning models using only source-domain labels. The method introduces optimal transport theory and global alignment in image, feature and output spaces to mitigate the domain shift between the source and target domains. The experiments employ the Potsdam and Vaihingen datasets provided by the International Society for Photogrammetry and Remote Sensing (ISPRS) to validate the performance. The results show that the method in this paper achieves higher accuracy compared with existing methods. Based on the ablation study, the effectiveness of the optimal transport theory is demonstrated in the UDA framework for semantic segmentation driven by deep learning.

Key words: remote sensing imagery, semantic segmentation, domain adaptation, optimal transport

CLC Number:

P237

SHEN Ziyang, NI Huan, GUAN Haiyan. Unsupervised domain adaptation alignment method for cross-domain semantic segmentation of remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(12): 2115-2126.

References

[1] 陈杰, 邓敏, 肖鹏峰, 等. 结合支持向量机与粒度计算的高分辨率遥感影像面向对象分类[J]. 测绘学报, 2011, 40(2): 135-141, 147. CHEN Jie, DENG Min, XIAO Pengfeng, et al. Object-oriented classification of high resolution imagery combining support vector machine with granular computing[J]. Acta Geodaetica et Cartographica Sinica, 2011, 40(2): 135-141, 147.
[2] AWAD M, CHEHDI K, NASRI A. Multicomponent image segmentation using a genetic algorithm and artificial neural network[J]. IEEE Geoscience and Remote Sensing Letters, 2007, 4(4): 571-575.
[3] LALIBERTE A S, FREDRICKSON E L, RANGO A. Combining decision trees with hierarchical object-oriented image analysis for mapping arid rangelands[J]. Photogrammetric Engineering & Remote Sensing, 2007, 73(2): 197-207.
[4] 王猛, 张新长, 王家耀, 等. 结合随机森林面向对象的森林资源分类[J]. 测绘学报, 2020, 49(2): 235-244. DOI: 10.11947/j.AGCS.2020.20190272. WANG Meng, ZHANG Xinchang, WANG Jiayao, et al. Forest resource classification based on random forest and object oriented method[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(2): 235-244. DOI: 10.11947/j.AGCS.2020.20190272.
[5] DOU Peng, CHEN Yangbo, YUE Haiyun. Remote-sensing imagery classification using multiple classification algorithm-based AdaBoost[J]. International Journal of Remote Sensing, 2018, 39(3): 619-639.
[6] 刘帅, 李笑迎, 于梦, 等. 高分辨率遥感图像双解耦语义分割网络模型[J]. 测绘学报, 2023,52(4): 638-647. DOI: 10.11947/j.AGCS.2023.20210455. LIU Shuai, LI Xiaoying, YU Meng, et al. Dual decoupling semantic segmentation model for high-resolution remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2023,52(4): 638-647. DOI: 10.11947/j.AGCS.2023.20210455.
[7] ZUO Zongcheng, ZHANG Wen, ZHANG Dongying. A remote sensing image semantic segmentation method by combining deformable convolution with conditional random fields[J]. Journal of Geodesy and Geoinformation Science, 2020, 3(2): 114.
[8] HONG Danfeng, GAO Lianru, YAO Jing, et al. Graph convolutional networks for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(7): 5966-5978.
[9] XU Zhiyong, ZHANG Weicun, ZHANG Tianxiang, et al. Efficient transformer for remote sensing image segmentation[J]. Remote Sensing, 2021, 13(18): 3585.
[10] HONG Danfeng, GAO Lianru, YOKOYA N, et al. More diverse means better: multimodal deep learning meets remote-sensing imagery classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(5): 4340-4354.
[11] BENJDIRA B, BAZI Y, KOUBAA A, et al. Unsupervised domain adaptation using generative adversarial networks for semantic segmentation of aerial images[J]. Remote Sensing, 2019, 11(11): 1369.
[12] GONG Jianya, JI Shunping. Photogrammetry and deep learning[J]. Journal of Geodesy and Geoinformation Science, 2018, 1(1): 1-15.
[13] 刘玮. 面向高分辨率遥感图像分类的域适应算法研究[D]. 哈尔滨: 哈尔滨工业大学, 2020. LIU Wei. Domain adaptation for high resolution remote sensing image classification[D]. Harbin: Harbin Institute of Technology, 2020.
[14] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11): 139-144.
[15] TZENG E, HOFFMAN J, ZHANG Ning, et al. Deep domain confusion: maximizing for domain invariance[EB/OL]. [2022-08-08].https://arxiv.org/abs/1412.3474.pdf.
[16] LONG Mingsheng, CAO Yue, WANG Jianmin, et al. Learning transferable features with deep adaptation networks[C]//Proceedings of 2015 International Conference on Machine Learning. Lille:ICML, 2015: 97-105.
[17] SUN Baochen, FENG Jiashi, SAENKO K. Return of frustratingly easy domain adaptation[C]//Proceedings of 2016 AAAI Conference on Artificial Intelligence. New York: ACM Press, 2016: 2058-2065.
[18] CHEN Chao, CHEN Zhihong, JIANG Boyuan, et al. Joint domain alignment and discriminative feature learning for unsupervised deep domain adaptation[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33(1): 3296-3303.
[19] ZELLINGER W, GRUBINGER T, LUGHOFER E, et al. Central moment discrepancy (CMD) for domain-invariant representation learning[EB/OL]. [2022-08-08]. https://arxiv.org/abs/1702.08811.pdf.
[20] ZHU Junyan, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice:IEEE, 2017: 2223-2232.
[21] TASAR O, HAPPY S L, TARABALKA Y, et al. ColorMapGAN: unsupervised domain adaptation for semantic segmentation using color mapping generative adversarial networks[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(10): 7178-7193.
[22] ZHAO Yang, GUO Peng, SUN Zihao, et al. ResiDualGAN: resize-residual DualGAN for cross-domain remote sensing images semantic segmentation[EB/OL].[2022-08-08]. https://arxiv.org/abs/2201.11523.pdf.
[23] CHEN Y H, CHEN Weiyu, CHEN Yuting, et al. No more discrimination: cross city adaptation of road scene segmenters[C]//Proceeding of 2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 2011-2020.
[24] TSAI Y H, HUNG W C, SCHULTER S, et al. Learning to adapt structured output space for semantic segmentation[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7472-7481.
[25] WANG Zhonghao, YU Mo, WEI Yunchao, et al. Differential treatment for stuff and things: a simple unsupervised domain adaptation method for semantic segmentation[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Seattle: IEEE, 2020: 12635-12644.
[26] XU Qingsong, YUAN Xin, OUYANG Chaojun. Class-aware domain adaptation for semantic segmentation of remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-17.
[27] CHEN Jie, ZHU Jingru, GUO Ya, et al. Unsupervised domain adaptation for semantic segmentation of high-resolution remote sensing imagery driven by category-certainty attention[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-15.
[28] COURTY N, FLAMARY R, TUIA D, et al. Optimal transport for domain adaptation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(9): 1853-1865.
[29] XIE Yujia, CHEN Minshuo, JIANG Haoming, et al. On scalable and efficient computation of large scale optimal transport[EB/OL].[2022-08-08]. https://arxiv.org/abs/1905.00158.pdf.
[30] XU Renjun, LIU P, WANG Liyan, et al. Reliable weighted optimal transport for unsupervised domain adaptation[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 4394-4403.
[31] LI Mengxue, ZHAI Yiming, LUO Youwei, et al. Enhanced transport distance for unsupervised domain adaptation[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 13936-13944.
[32] COURTY N, FLAMARY R, HABRARD A, et al. Joint distribution optimal transportation for domain adaptation[EB/OL].[2022-08-08]. https://arxiv.org/abs/1705.08848.pdf.
[33] DAMODARAN B B, KELLENBERGER B, FLAMARY R, et al. DeepJDOT: deep joint distribution optimal transport for unsupervised domain adaptation[C]//Proceedings of 2018 European Conference on Computer Vision. Cham: Springer, 2018: 467-483.
[34] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
[35] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848.
[36] VILLANI C. Optimal transport: old and new[M]. Berlin: Springer, 2009.
[37] RUBNER Y, TOMASI C, GUIBAS L J. A metric for distributions with applications to image databases[C]//Proceedings of 1998 International Conference on Computer Vision. Bombay:Narosa Publishing House, 1998: 59-66.
[38] 刘轲. 基于局部特征匹配的颜色传递方法研究[D]. 赣州: 江西理工大学, 2021. LIU Ke. Research on color transfer method based on local feature matching[D]. Ganzhou: Jiangxi University of Science and Technology, 2021.