面向多源异质遥感影像地物分类的自监督预训练方法

doi:10.11947/j.AGCS.2024.20220553

摘要/Abstract

摘要： 近年来,深度学习改变了遥感图像处理的方法。由于标注高质量样本费时费力,标签样本数量不足的现实问题会严重影响深层神经网络模型的性能。为解决这一突出矛盾,本文提出了用于多源异质遥感影像地物分类的自监督预训练和微调分类方案,旨在缓解模型对于标签样本的严重依赖。具体来讲,生成式自监督学习模型由非对称的编码器-解码器结构组成,其中深度编码器从多源遥感数据中学习高阶关键特征,任务特定的解码器用于重建原始遥感影像。为提升特性表示能力,交叉注意力机制模型用于融合异源特征中的信息,进而从多源异质遥感影像中学习更多的互补信息。在微调分类阶段,预训练好的编码器作为无监督特征提取器,基于Transformer结构的轻量级分类器将学习到的特征与光谱信息结合并用于地物分类。这种自监督预训练方案能够从多源异质遥感影像中学习到刻画原始数据的高级关键特征,并且此过程不需要任何人工标注信息,从而缓解了对标签样本的依赖。与现有的分类范式相比,本文提出的自监督预训练和微调方案在多源遥感影像地物分类中能够取得更优的分类结果。

关键词: 遥感, 多源异质数据, 预训练, 自监督学习, 土地覆盖分类

Abstract: Deep learning has revolutionized the remote sensing image processing techniques over the past few years. Nevertheless, it is laborious to annotate high quality samples, thus limiting the performance of deep networks because of insufficient supervision information. To resolve this contradiction, we investigate the self-supervised pre-training and fine-tuning paradigm for multi-source heterogeneous remote sensing image land cover classification, aiming to relieve the urgent need for manually annotated data. Specifically, the proposed generative feature learning model consists of asymmetric encoder-decoder structure, in which the deep encoder extracts high-level key characteristics contained in multi-source data and task-specific lightweight decoders are developed to reconstruct original data. To further improve the feature representation capability, the cross-attention layers are utilized to exchange information contained in heterogeneous characteristics, thus learning more complementary information from multi-source remote sensing data. In fine-tuning stage, the trained encoder is employed as unsupervised feature extractor, and learned features are utilized for land cover classification through the designed lightweight Transformer based classifier. This self-supervised pre-training architecture is capable of learning high-level key features from multi-source heterogenous remote sensing images, and this process does not require any labeled information, thus relieving the urgent need for labeled samples. Compared with existing classification paradigms, the proposed multimodal self-supervised pre-training and fine-tuning scheme achieves superior performance for remote sensing image classification.

Key words: remote sensing, multi-source heterogeneous data, pre-training, self-supervised learning, land cover classification

中图分类号:

P237

薛志祥, 余旭初, 刘景正, 杨国鹏, 刘冰, 余岸竹, 周嘉男, 金上鸿. 面向多源异质遥感影像地物分类的自监督预训练方法[J]. 测绘学报, 2024, 53(3): 512-525.

XUE Zhixiang, YU Xuchu, LIU Jingzheng, YANG Guopeng, LIU Bing, YU Anzhu, ZHOU Jianan, JIN Shanghong. A self-supervised pre-training scheme for multi-source heterogeneous remote sensing image land cover classification[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(3): 512-525.

参考文献

[1] HONG Danfeng, HU Jingliang, YAO Jing, et al. Multimodal remote sensing benchmark datasets for land cover classification with a shared and specific feature learning model[J]. ISPRS Journal of Photogrammetry and Remote Sensing:Official Publication of the International Society for Photogrammetry and Remote Sensing (ISPRS), 2021, 178:68-80.
[2] LEI Lei, WANG Xinyu, ZHONG Yanfei, et al. DOCC:deep one-class crop classification via positive and unlabeled learning for multi-modal satellite imagery[J]. International Journal of Applied Earth Observation and Geoinformation, 2021, 105:102598.
[3] 张良培, 何江, 杨倩倩, 等. 数据驱动的多源遥感信息融合研究进展[J]. 测绘学报, 2022, 51(7):1317-1337.DOI:10.11947/j.AGCS.2022.20220171. ZHANG Liangpei, HE Jiang, YANG Qianqian, et al. Data-driven multi-source remote sensing data fusion:progress and challenges[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(7):1317-1337.DOI:10.11947/j.AGCS.2022.20220171.
[4] 施蓓琦, 刘春, 孙伟伟, 等. 应用稀疏非负矩阵分解聚类实现高光谱影像波段的优化选择[J]. 测绘学报, 2013, 42(3):351-358, 366. SHI Beiqi, LIU Chun, SUN Weiwei, et al. Sparse nonnegative matrix factorization for hyperspectral optimal band selection[J]. Acta Geodaetica et Cartographica Sinica, 2013, 42(3):351-358, 366.
[5] 余岸竹, 刘冰, 邢志鹏, 等. 面向高光谱影像分类的显著性特征提取方法[J]. 测绘学报, 2019, 48(8):985-995.DOI:10.11947/j.AGCS.2019.20180499. YU Anzhu, LIU Bing, XING Zhipeng, et al. Salient feature extraction method for hyperspectral image classification[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(8):985-995. DOI:10.11947/j.AGCS.2019.20180499.
[6] 张良培, 李家艺. 高光谱图像稀疏信息处理综述与展望[J]. 遥感学报, 2016, 20(5):1091-1101. ZHANG Liangpei, LI Jiayi. Development and prospect of sparse representation-based hyperspectral image processing and analysis[J]. Journal of Remote Sensing, 2016, 20(5):1091-1101.
[7] CHEN Yushi, JIANG Hanlu, LI Chunyang, et al. Deep feature extraction and classification of hyperspectral images based on convolutional neural networks[J]. IEEE Transactions on Geoscience and Remote Sensing, 2016, 54(10):6232-6251.
[8] LEE H, KWON H. Going deeper with contextual CNN for hyperspectral image classification[J]. IEEE Transactions on Image Processing:a Publication of the IEEE Signal Processing Society, 2017, 26(10):4843-4855.
[9] WANG Kexian, ZHENG Shunyi, LI Rui, et al. A deep double-channel dense network for hyperspectral image classification[J]. Journal of Geodesy and Geoinformation Science, 2021, 4(4):46-62.
[10] MOU Lichao, GHAMISIP, ZHU Xiaoxiang. Deep recurrent neural networks for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(7):3639-3655.
[11] HONG Danfeng, HAN Zhu, YAO Jing, et al. SpectralFormer:rethinking hyperspectral image classification with transformers[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60:5518615.
[12] HANG Renlong, LI Zhu, LIU Qingshan, et al. Hyperspectral image classification with attention-aided CNNs[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(3):2281-2293.
[13] YUE Jun, FANG Leyuan, RAHMANI H, et al. Self-supervised learning with adaptive distillation for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60:5501813.
[14] YANG Lina, ZHANG Fengqi, WANGP S P, et al. Multi-scale spatial-spectral fusion based on multi-input fusion calculation and coordinate attention for hyperspectral image classification[J]. Pattern Recognition, 2022, 122:108348.
[15] XUE Zhixiang, YU Xuchu, TAN Xiong, et al. Multiscale deep learning network with self-calibrated convolution for hyperspectral and LiDAR data collaborative classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60:5514116.
[16] LI Wei, WANG Junjie, GAO Yunhao, et al. Graph-feature-enhanced selective assignment network for hyperspectral and multispectral data classification[J]. IEEE Transactions on Geoscience and Remote Sensing,2022, 60:5526914.
[17] WANG Junjie, LI Wei, GAO Yunhao, et al. Hyperspectral and SAR image classification via multiscale interactive fusion network[J]. IEEE Transactions on Neural Networks and Learning Systems,2023, 34(12):10823-10837.
[18] 张兵, 杨晓梅, 高连如, 等. 遥感大数据智能解译的地理学认知模型与方法[J]. 测绘学报, 2022, 51(7):1398-1415.DOI:10.11947/j.AGCS.2022.20220279. ZHANG Bing, YANG Xiaomei, GAO Lianru, et al. Geo-cognitive models and methods for intelligent interpretation of remotely sensed big data[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(7):1398-1415.DOI:10.11947/j.AGCS.2022.20220279.
[19] 左溪冰, 刘冰, 余旭初, 等. 高光谱影像小样本分类的图卷积网络方法[J]. 测绘学报, 2021, 50(10):1358-1369. DOI:10.11947/j.AGCS.2021.20200155. ZUO Xibing, LIU Bing, YU Xuchu, et al. Graph convolutional network method for small sample classification of hyperspectral images[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(10):1358-1369.DOI:10.11947/j.AGCS.2021.20200155.
[20] ZHU Lin, CHEN Yushi, GHAMISI P, et al. Generative adversarial networks for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(9):5046-5063.
[21] 刘冰, 左溪冰, 谭熊, 等. 高光谱影像分类的深度少样例学习方法[J]. 测绘学报, 2020, 49(10):1331-1342.DOI:10.11947/j.AGCS.2020.20190486. LIU Bing, ZUO Xibing, TAN Xiong, et al. A deep few-shot learning algorithm for hyperspectral image classification[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(10):1331-1342. DOI:10.11947/j.AGCS.2020.20190486.
[22] 陶超, 阴紫薇, 朱庆, 等. 遥感影像智能解译:从监督学习到自监督学习[J]. 测绘学报, 2021, 50(8):1122-1134.DOI:10.11947/j.AGCS.2021.20210089. TAO Chao, YIN Ziwei, ZHU Qing, et al. Remote sensing image intelligent interpretation:from supervised learning to self-supervised learning[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(8):1122-1134.DOI:10.11947/j.AGCS.2021.20210089.
[23] ZHU Mingzhen, FAN Jiayuan, YANG Qihang, et al. SC-EADNet:a self-supervised contrastive efficient asymmetric dilated network for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60:5519517.
[24] LI Xiaomin, SHI Daqian, DIAO Xiaolei, et al. SCL-MLNet:boosting few-shot remote sensing scene classification via self-supervised contrastive learning[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60:5801112.
[25] ZHANG Lamei, ZHANG Siyu, ZOU Bin, et al. Unsupervised deep representation learning and few-shot classification of PolSAR images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:5100316.
[26] CHEN Yuxing, BRUZZONE L. Self-supervised change detection in multiview remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:5402812.
[27] XUE Zhixiang, YU Xuchu, YU Anzhu, et al. Self-supervised feature learning for multimodal remote sensing image land cover classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 1809, 60:5533815.
[28] HE Kaiming, CHEN Xinlei, XIE Saining, et al. Masked autoencoders are scalable vision learners[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans:IEEE, 2022:15979-15988.
[29] XIE Zhenda, ZHANG Zheng, CAO Yue, et al. SimMIM:a simple framework for masked image modeling[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans:IEEE, 2022:9643-9653.
[30] FEICHTENHOFER C, FAN Haoqi, LI Yanghao, et al. Masked autoencoders as spatiotemporal learners[C]//Proceedings of the 35th International Conference on Neural Information Processing Systems. New Orleans:ACM Press, 2022:35946-35958.
[31] BACHMANNR, MIZRAHI D, ATANOV A, et al. MultiMAE:multi-modal multi-task masked autoencoders[C]//Proceedigns of 2022 European Conference on Computer Vision. Cham:Springer, 2022:348-367.
[32] XUE Zhixiang, LIU Bing, YU Anzhu, et al. Self-supervised feature representation and few-shot land cover classification of multimodal remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:1-118.
[33] MELGANIF, BRUZZONE L. Classification of hyperspectral remote sensing images with support vector machines[J]. IEEE Transactions on Geoscience and Remote Sensing, 2004, 42(8):1778-1790.
[34] BRUZZONEL, CHI M, MARCONCINI M. A novel transductive SVM for semisupervised classification of remote-sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2006, 44(11):3363-3373.
[35] HONG Danfeng, WU Xin, GHAMISI P, et al. Invariant attribute profiles:a spatial-frequency joint feature extractor for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(6):3791-3808.
[36] XU Yonghao, ZHANG Liangpei, DU Bo, et al. Spectral-spatial unified networks for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(10):5893-5909.
[37] WANG Di, DU Bo, ZHANG Liangpei, et al. Adaptive spectral-spatial multiscale contextual feature extraction for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(3):2461-2477.
[38] YANG Yi, ZHU Daoye, QU Tengteng, et al. Single-stream CNN with learnable architecture for multisource remote sensing data[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:5409218.
[39] LIU Bing, YU Anzhu, YU Xuchu, et al. Deep multiview learning for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(9):7758-7772.