基于Swin Transformer-CNN的单目遥感影像高程估计方法及其在公路建设场景中的应用

doi:10.11947/j.AGCS.2024.20220607

摘要/Abstract

摘要： 目前，在遥感影像几何条件和辐射质量良好的情况下，通过多视遥感影像的逐像素立体密集匹配对场景进行高程估计的技术相对比较成熟，无论是精度还是效率均达到了较高水平。然而，当具有良好几何条件和辐射质量的多视遥感影像难以获取时，经典摄影测量和计算机视觉的几何处理方法可能会面临较大的挑战。本文对该问题进行了研究，针对大幅遥感图像中各部分高程分布差异大，模型训练难度大的问题，提出了一种基于Swin Transformer和卷积神经网络（convolutional neural network，CNN）的单目遥感影像高程估计方法。一方面Swin Transformer利用滑动窗口和层级设计，兼具了卷积神经网络处理大尺寸图像和提取多尺度特征的能力及Transformer的全局信息交互能力。另一方面针对大幅遥感图像中各部分高程分布差异大带来的训练不稳定问题，本文方法能针对每张输入图像自适应地划分高程值，将高程估计问题转化为分类-回归问题，最终图像各像素点的高程值由划分的高程值及其分布概率得到。试验结果表明：本文所提出的基于Swin Transformer-CNN的遥感影像高程估计方法无论是定性还是定量的结果都取得了很好的效果，且能应用于公路建设施工场景中，具有良好的泛化性。

关键词: 遥感影像智能解译, 深度学习, 单目高程预测, 全局信息, 卷积神经网络

Abstract: At present, under the good condition of image geometry and radiation quality, the technology of 3D scene reconstruction by intensive matching of multi-view aerospace image is relatively mature, which has achieved good results both in accuracy and efficiency. However, when multi-view aerospace images with good geometric conditions are difficult to obtain, the geometric processing methods of classical photogrammetry and computer vision may face great challenges. In this paper, we study this problem and propose a monocular height estimation method of remote sensing image based on Swin Transformer and convolutional neural network (CNN). Swin Transformer is a hierarchical transformer structure with shifted windows. It combines the ability of convolutional neural network to process large scale image and extract multi-scale features, as well as the global information interaction ability of transformer. In addition, our method reformulates the height estimation problem into a classification-regression problem to improve model performance. Specifically, for each input image, our model classifies the height range into several discrete bins adaptively, where continuous height value is estimated via a linear combination of predicted discrete bins and height distribution probability. In experiments, we qualitatively and quantitatively demonstrate that the proposed method outperforms the state-of-the-art approaches, and it can also be applied to highway road construction sites with good generalization.

Key words: intelligent interpretation of remote sensing image, deep learning, monocular height estimation, global information, convolutional neural networks

中图分类号:

P237

廖钊宏, 张依晨, 杨飚, 林明春, 孙文博, 高智. 基于Swin Transformer-CNN的单目遥感影像高程估计方法及其在公路建设场景中的应用[J]. 测绘学报, 2024, 53(2): 344-352.

LIAO Zhaohong, ZHANG Yichen, YANG Biao, LIN Mingchun, SUN Wenbo, GAO Zhi. Monocular height estimation method of remote sensing image based on Swin Transformer-CNN and its application in highway road construction sites[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(2): 344-352.

参考文献

[1] GEHRKE S, MORIN K, DOWNEY M, et al. Semi-global matching:an alternative to LiDAR for dsm generation?[J]. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2008, XXXVIII-B1:1-6.
[2] DE FRANCHIS C, MEINHARDT-LLOPIS E, MICHEL J, et al. An automatic and modular stereo pipeline for pushbroom images[J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2014, II-3:49-56.
[3] ALBANWAN H, QIN Rongjun. An adaptive and image-guided fusion for stereo satellite image derived digital surface models[J]. Journal of Geodesy and Geoinformation Science, 2022, 5(4):1-9.
[4] 余东行, 徐青, 赵传, 等. 注意力引导特征融合与联合学习的遥感影像场景分类[J]. 测绘学报, 2023, 52(4):624-637.DOI:10.11947/J.AGCS.2023.20210659. YU Donghang, XU Qing, ZHAO Chuan, et al. Attention-guided feature fusion and joint learning for remote sensing image scene classification[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(4):624-637.DOI:10.11947/J.AGCS.2023.20210659.
[5] 胡明洪, 李佳田, 姚彦吉, 等. 结合多路径的高分辨率遥感影像建筑物提取SER-UNet算法[J]. 测绘学报, 2023, 52(5):808-817.DOI:10.11947/J.AGCS.2023.20210691. HU Minghong, LI Jiatian, YAO Yanji, et al. SER-UNet algorithm for building extraction from high-resolution remote sensing image combined with multipath[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(5):808-817.DOI:10.11947/J.AGCS.2023.20210691.
[6] WANG Yanjun, LI Shaochun, WANG Mengjie, et al. A simple deep learning network for classification of 3D mobile LiDAR point clouds[J]. Journal of Geodesy and Geoinformation Science, 2021, 4(3):49-59.
[7] HONG Danfeng, GAO Lianru, YAO Jing, et al. Graph convolutional networks for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(7):5966-5978.
[8] HONG Danfeng, HAN Zhu, YAO Jing, et al. Spectral former:rethinking hyperspectral image classification with transformers[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60:1-15.
[9] YAO Jing, CAO Xiangyong, HONG Danfeng, et al. Semi-active convolutional neural networks for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 61:1-15.
[10] XIONG Zhitong, CHEN Sining, SHI Yilei, et al. Disentangled latent transformer for interpretable monocular height estimation[EB/OL].[2023-07-10]. https://arxiv.org/abs/2201.06357.pdf.
[11] MOU Lichao, ZHU Xiaoxiang. IM2HEIGHT:height estimation from single monocular imagery via fully residual convolutional-deconvolutional network[EB/OL].[2023-10-25]. https://arxiv.org/abs/1802.10249.pdf.
[12] GHAMISI P, YOKOYA N. IMG2DSM:height simulation from single imagery using conditional generative adversarial net[J]. IEEE Geoscience and Remote Sensing Letters, 2018, 15(5):794-798.
[13] PAOLETTI M E, HAUT J M, GHAMISI P, et al. U-IMG2DSM:unpaired simulation of digital surface models with generative adversarial networks[J]. IEEE Geoscience and Remote Sensing Letters, 2021, 18(7):1288-1292.
[14] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words:transformers for image recognition at scale[EB/OL].[2023-05-20]. https://arxiv.org/abs/2010.11929.pdf.
[15] LIU Ze, LIN Yutong, CAO Yue, et al. Swin transformer:hierarchical vision transformer using shifted windows[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision.Montreal:IEEE, 2021:10012-10022.
[16] EIGEN D, PUHRSCH C, FERGUS R. Depth map prediction from a single image using a multi-scale deep network[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal:ACM Press, 2014:2366-2374.
[17] 张聪, 马燕新, 万建伟, 等. 基于通道注意力机制的单目深度估计[J]. 信号处理, 2022,38(11):2332-2341. ZHANG Cong, MA Yanxin, WAN Jianwei, et al.Multi-scale monocular depth estimation network based on channel attention[J]. Journal of Signal Processing, 2022, 38(11):2332-2341.
[18] 蓝玲玲. 基于深度学习的单目深度估计研究[D]. 昆明:云南师范大学,2022. LAN Lingling. Research on monocular depth estimation based on deep learning[D]. Kunming:Yunnan Normal University,2022.
[19] 朱孟飞. 基于深度学习的水下单目深度估计方法研究[D]. 上海:上海海洋大学,2022. ZHU Mengfei. Research on estimation method of underwater monocular depth based on deep learning[D]. Shanghai:Shanghai Ocean University,2022.
[20] JOHNSTON A, CARNEIRO G. Self-supervised monocular trained depth estimation using self-attention and discrete disparity volume[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Seattle:IEEE, 2020:4756-4765.
[21] FAROOQ BHAT S, ALHASHIM I, WONKA P. AdaBins:depth estimation using adaptive bins[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Nashville:IEEE, 2021:4009-4018.
[22] LI Zhenyu, WANG Xuyang, LIU Xianming, et al. Bins former:revisiting adaptive bins for monocular depth estimation[EB/OL].[2023-10-25]. https://arxiv.org/abs/2204.00987.pdf.
[23] ZHAO Qi, LIU Jiahui, LI Yuewen, et al. Semantic segmentation with attention mechanism for remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 61:1-13.
[24] WANG Yufeng, DING Wenrui, ZHANG Ruiqian, et al. Boundary-aware multitask learning for remote sensing imagery[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14:951-963.
[25] XING Siyuan, DONG Qiulei, HU Zhanyi. SCE-net:self- and cross-enhancement network for single-view height estimation and semantic segmentation[J]. Remote Sensing, 2022, 14(9):2252.
[26] SRIVASTAVA S, VOLPI M, TUIA D. Joint height estimation and semantic labeling of monocular aerial images with CNNS[C]//Proceedings of 2017 IEEE International Geoscience and Remote Sensing Symposium. Fort Worth:IEEE, 2017:5173-5176.
[27] CARVALHO M, LE SAUX B, TROUVE-PELOUX P, et al. On regression losses for deep depth estimation[C]//Proceedings of the 25th IEEE International Conference on Image Processing. Athens:IEEE, 2018:2915-2919.
[28] ZHANG Yiteng, CHEN Xuejin. Multi-path fusion network for high-resolution height estimation from a single orthophoto[C]//Proceedings of 2019 IEEE International Conference on Multimedia & Expo Workshops. Shanghai:IEEE, 2019:186-191.
[29] AMIRKOLAEE H A, AREFI H. Height estimation from single aerial images using a deep convolutional encoder-decoder network[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 149:50-66.
[30] ALIDOOST F, AREFI H, TOMBARI F. 2D image-to-3D model:knowledge-based 3D building reconstruction (3DBR) using single aerial images and convolutional neural networks (CNNs)[J]. Remote Sensing, 2019, 11(19):2219.
[31] LIU C J, KRYLOV V A, KANE P, et al. IM2ELEVATION:building height estimation from single-view aerial imagery[J]. Remote Sensing, 2020, 12(17):2719.
[32] CARVALHO M, LE SAUX B, TROUVE-PELOUX P, et al. Multitask learning of height and semantics from aerial images[J]. IEEE Geoscience and Remote Sensing Letters, 2020, 17(8):1391-1395.