基于投影转换与道路分割的街景-卫星影像跨视角定位方法

doi:10.11947/j.AGCS.2025.20240254

摘要/Abstract

摘要：

地面街景影像与卫星影像之间巨大的视角差异，使得两者之间的匹配极为困难，这给跨视角图像定位的研究与应用带来巨大挑战。针对街景-卫星影像跨视角定位中视角差异造成的定位困难问题，本文提出了一种基于投影转换与道路分割的跨视角定位方法。首先，通过在街景影像与卫星影像之间建立投影关系，实现地面图像视角向卫星图像视角之间的转换，以减小地面图像与卫星图像之间的视角差异。同时，为了更好地学习图像中的视点不变性特征，本文从自监督学习范式出发，利用具有强大零样本泛化能力的视觉基础模型进行道路分割，在训练过程中引入道路先验信息的辅助训练分支，从而在不改变模型架构和模型推理速度的情况下提高模型的性能。结合本文方法后，SAFA、GeoDTR、SAIG、TransGeo 4种跨视角定位方法在CVACT数据集上的平均Recall@1精度提升了0.55个百分点，在CVWU数据集上的平均Recall@1提升了2.84个百分点。试验结果表明本文方法可以与其他模型架构进行有机结合，具有很好的适用性。

关键词: 投影转换, 自监督学习, 道路分割, 辅助训练分支, 跨视角图像定位

Abstract:

The vast differences between street view and satellite images make it extremely difficult to match them, which brings great challenges to the research and application of cross-view image geo-localization in this study, we propose a cross-view geo-localization framework based on projective transform and road segmentation to address the difficulties caused by the viewpoint differences in cross-view image geo-localization. Firstly, we establish a geometric projection relationship between the street view and satellite images to achieve the viewpoint transformation from the ground images to satellite images, in order to reduce the viewpoint difference between the street view and satellite images. Meanwhile, to better learn the viewpoint invariant features in the images, we are inspired by self-supervised learning and then use a visual foundation model with strong zero-shot generalization capability for road segmentation, and introduce an auxiliary training branch for road prior information during the training process, to improve the performance of the model without changing the model architecture and the model inference speed. After using our method, the average Recall@1 accuracy of the four methods, SAFA, GeoDTR, SAIG, and TransGeo, is improved by 0.55% on the CVACT dataset and 2.84% on the CVWU dataset. The experimental results show that the proposed cross-view geo-localization method, which combines geometric projective transform with self-supervised learning, can be organically combined with other model architectures.

Key words: projective transform, self-supervised learning, road segmentation, auxiliary training branch, cross-view image geo-localization

中图分类号:

P237

甘文建, 周杨, 胡校飞, 赵璐颖, 黄高爽, 侯铭波. 基于投影转换与道路分割的街景-卫星影像跨视角定位方法[J]. 测绘学报, 2025, 54(8): 1489-1500.

Wenjian GAN, Yang ZHOU, Xiaofei HU, Luying ZHAO, Gaoshuang HUANG, Mingbo HOU. Combining projective transform and road segmentation for street view-satellite images cross-view geo-localization[J]. Acta Geodaetica et Cartographica Sinica, 2025, 54(8): 1489-1500.

图/表 12

图1

图2

图3

图4

表1

图5

图6

表2

图7

图8

表3

图9

参考文献 35

[1]	ZHANG Xiwu, WANG Lei, SU Yan. Visual place recognition: a survey from deep learning perspective[J]. Pattern Recognition, 2021, 113: 107760.
[2]	MASONE C, CAPUTO B. A survey on deep visual place recognition[J]. IEEE Access, 2021, 9: 19516-19547.
[3]	DOAN D, LATIF Y, CHIN T J, et al. Scalable place recognition under appearance change for autonomous driving[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 9318-9327.
[4]	黄高爽, 周杨, 胡校飞, 等. 图像地理定位研究进展[J]. 地球信息科学学报, 2023, 25(7): 1336-1362.
	HUANG Gaoshuang, ZHOU Yang, HU Xiaofei, et al. A survey of the research progress in image geo-localization[J]. Journal of Geo-information Science, 2023, 25(7): 1336-1362.
[5]	杨元喜, 王建荣. 泛在感知与航天测绘[J]. 测绘学报, 2023, 52(1): 1-7. DOI: . doi: 10.11947/j.AGCS.2023.20220405
	YANG Yuanxi, WANG Jianrong. Ubiquitous perception and space mapping[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(1): 1-7. DOI: . doi: 10.11947/j.AGCS.2023.20220405
[6]	LIN T Y, BELONGIE S, HAYS J. Cross-view image geolocalization[C]//Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland: IEEE, 2013: 891-898.
[7]	VISWANATHAN A, PIRES B R, HUBER D. Vision based robot localization by ground to satellite matching in GPS-denied situations[C]//Proceedings of 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. Chicago: IEEE, 2014: 192-198.
[8]	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: Transformers for image recognition at scale[EB/OL]. [2021-06-03]. https://arxiv.org/abs/2010.11929
[9]	SHI Yujiao, LIU Liu, YU Xin, et al. Spatial-aware feature aggregation for image based cross-view geo-localization[C]//Proceedings of the 33rd International Conference on Neural Information Processing Systems. Vancouver: ACM Press, 2019: 10090-10100.
[10]	LI Songlian, TU Zhigang, CHEN Yujin, et al. Multi-scale attention encoder for street-to-aerial image geo-localization[J]. CAAI Transactions on Intelligence Technology, 2023, 8(1): 166-176.
[11]	饶子昱, 卢俊, 郭海涛, 等. 利用视角转换的跨视角影像匹配方法[J]. 地球信息科学学报, 2023, 25(2): 368-379.
	RAO Ziyu, LU Jun, GUO Haitao, et al. A cross-view image matching method with viewpoint conversion[J]. Journal of Geo-information Science, 2023, 25(2): 368-379.
[12]	SHI Yujiao, YU Xin, LIU Liu, et al. Accurate 3-DoF camera geo-localization via ground-to-satellite image matching[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(3): 2682-2697.
[13]	GOODFELLOW I, ABADIE J, MIRZA M, et al. Conditional generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montréal: ACM Press, 2014: 2672-2680.
[14]	REGMI K, SHAH M. Bridging the domain gap for ground-to-aerial image matching[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 470-479.
[15]	何思瑾. 基于深度学习的跨视角图像地理定位技术研究[D]. 武汉: 华中科技大学, 2021.
	HE Sijin. Research on cross-view image geo-localization technology based on deep learning[D]. Wuhan: Huazhong University of Science and Technology, 2021.
[16]	TOKER A, ZHOU Qunjie, MAXIMOV M, et al. Coming down to earth: satellite-to-street view synthesis for geo-localization[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 6484-6493.
[17]	SHI Yujiao, CAMPBELL D, YU Xin, et al. Geometry-guided street-view panorama synthesis from satellite imagery[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(12): 10009-10022.
[18]	HU Sixing, FENG Mengdan, NGUYEN R M H, et al. CVM-net: cross-view matching network for image-based ground-to-aerial geo-localization[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7258-7267.
[19]	SHI Yujiao, YU Xin, LIU Liu, et al. Optimal feature transport for cross-view image geo-localization[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 11990-11997.
[20]	SUN Bin, CHEN Chen, ZHU Yingying, et al. GEOCAPSNET: ground to aerial view image geo-localization using capsule network[C]//Proceedings of 2019 IEEE International Conference on Multimedia and Expo. Shanghai: IEEE, 2019: 742-747.
[21]	ZHANG Xiaohan, LI Xingyu, SULTANI W, et al. Cross-view geo-localization via learning disentangled geometric layout correspondence[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37(3): 3480-3488.
[22]	ZHU Sijie, SHAH M, CHEN Chen. TransGeo: transformer is all you need for cross-view image geo-localization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 1152-1161.
[23]	ZHU Yingying, YANG Hongji, LU Yuxin, et al. Simple, effective and general: a new backbone for cross-view image geo-localization[EB/OL]. [2023-02-03]. https://arxiv.org/abs/2302.01572.
[24]	VON RUEDEN L, MAYER S, BECKH K, et al. Informed machine learning-a taxonomy and survey of integrating prior knowledge into learning systems[J]. IEEE Transactions on Knowledge and Data Engineering, 2023, 35(1): 614-633.
[25]	CHEN Xinlei, HE Kaiming. Exploring simple Siamese representation learning[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 15745-15753.
[26]	JEAN G, FLORIAN S, FLORENT A, et al. Bootstrap your own latent a new approach to self-supervised learning[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Online: ACM, 2020: 21271-21284.
[27]	XIE Zhenda, ZHANG Zheng, CAO Yue, et al. SimMIM: a simple framework for masked image modeling[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 9643-9653.
[28]	OQUAB M, DARCET T, MOUTAKANNI T, et al. DINOv2: learning robust visual features without supervision[EB/OL]. [2024-02-02]. https://arxiv.org/abs/2304.07193.
[29]	SCHROFF F, KALENICHENKO D, PHILBIN J. FaceNet: a unified embedding for face recognition and clustering[C]//Proceedings of 2015 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 815-823.
[30]	HU Sixing, LEE G H. Image-based geo-localization using satellite imagery[J]. International Journal of Computer Vision, 2020, 128(5): 1205-1219.
[31]	HE Kaiming, FAN Haoqi, WU Yuxin, et al. Momentum contrast for unsupervised visual representation learning[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 9726-9735.
[32]	MINAEE S, BOYKOV Y Y, PORIKLI F, et al. Image segmentation using deep learning: a survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(7): 3523-3542.
[33]	ZOU Zhengxia, CHEN Keyan, SHI Zhenwei, et al. Object detection in 20 years: a survey[J]. Proceedings of the IEEE, 2023, 111(3): 257-276.
[34]	CHENG Bowen, MISRA I, SCHWING A G, et al. Masked-attention mask Transformer for universal image segmentation[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 1280-1289.
[35]	LIU Liu, LI Hongdong. Lending orientation to neural networks for cross-view geo-localization[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 5617-5626.

方法	CVACT				CVWU
方法	R@1	R@5	R@10	R@1%	R@1	R@5	R@10	R@1%
SAFA	81.25	92.46	94.42	97.75	75.23	88.51	91.76	95.05
SAFA^†	81.83	92.72	94.27	97.84	77.49	89.73	92.20	95.62
GeoDTR	83.27	93.30	94.84	97.83	73.76	89.01	92.67	96.84
GeoDTR^†	83.91	93.45	94.78	97.72	78.12	90.67	93.17	96.52
SAIG	84.24	93.94	95.57	98.32	84.03	95.55	97.24	99.25
SAIG^†	84.67	93.81	95.49	98.30	87.79	95.93	97.59	99.15
TransGeo	86.63	95.37	96.56	98.49	94.18	98.87	99.47	99.81
TransGeo^†	87.19	95.44	96.45	98.59	95.15	99.00	99.47	99.81

转换方法	CVACT
转换方法	R@1	R@5	R@10	R@1%
无转换	68.61	85.54	89.71	96.92
极坐标	75.84	89.52	92.39	97.25
逆极坐标	55.52	75.56	81.72	94.50
几何投影	79.39	91.42	93.63	97.38
本文方法	81.25	92.46	94.42	97.75

方法	FLOPs	Params/MB	CVACT						CVWU
方法	FLOPs	Params/MB	0	0.1	0.2	0.3	0.4	0.5	0	0.1	0.2	0.3	0.4	0.5
SAFA	4.15×10⁹	2.78×10⁶	81.25	81.19	81.54	81.66	81.83	81.50	75.23	76.86	75.49	75.89	77.49	76.30
GeoDTR	1.089×10¹⁰	1.235×10⁷	83.27	83.18	83.45	83.91	83.42	83.42	73.76	75.08	77.11	78.12	75.42	75.20
SAIG	1.245×10¹⁰	1.552×10⁷	84.24	84.50	84.38	84.67	84.39	84.19	84.03	85.13	86.88	86.76	86.76	87.79
TransGeo	1.247×10¹⁰	2.236×10⁷	86.63	86.70	87.19	86.72	86.52	85.99	94.18	94.83	94.90	95.12	95.15	95.08