基于场景图的遥感场景检索方法

doi:10.11947/j.AGCS.2025.20230439

测绘学报 ›› 2025, Vol. 54 ›› Issue (1): 123-135.doi: 10.11947/j.AGCS.2025.20230439

基于场景图的遥感场景检索方法

唐佳怡(), 童晓冲(), 邱春平, 雷亚现, 雷毅, 宋好帅

信息工程大学地理空间信息学院，河南　郑州　450001

收稿日期:2023-10-07 修回日期:2024-12-17 出版日期:2025-02-17 发布日期:2025-02-17
通讯作者: 童晓冲 E-mail:tangjiayi113769@163.com;txchr@aliyun.com
作者简介:唐佳怡（2000—），女，博士生，研究方向为地理空间智能。　E-mail：tangjiayi113769@163.com
基金资助:
国家重点研发计划(2024YFF1400804);嵩山实验室项目(221100211000-03);河南省自然科学基金优秀青年基金(212300410096);国家自然科学基金(42201513)

Remote sensing scene retrieval method based on scene graph

Jiayi TANG(), Xiaochong TONG(), Chunping QIU, Yaxian LEI, Yi LEI, Haoshuai SONG

Institute of Geospatial Information, Information Engineering University, Zhengzhou 450001, China

Received:2023-10-07 Revised:2024-12-17 Online:2025-02-17 Published:2025-02-17
Contact: Xiaochong TONG E-mail:tangjiayi113769@163.com;txchr@aliyun.com
About author:TANG Jiayi (2000—), female, PhD candidate, majors in geospatial intelligence.　E-mail: tangjiayi113769@163.com
Supported by:
The National Key Research and Development Program of China(2024YFF1400804);Program of Songshan Laboratory(221100211000-03);The Excellent Youth Foundation of Henan Municipal Natural Science Foundation(212300410096);The National Natural Science Foundation of China(42201513)

摘要/Abstract

摘要：

当前，大部分的遥感场景检索都是基于遥感图像的深度特征的相似度匹配实现的，难以直接表征场景实体之间的关系信息，并且缺乏直接表达空间结构和语义的方式，因此无法满足对遥感场景的更加复杂的检索需求。本文提出了一种基于场景图的遥感场景检索方法，利用图神经网络将遥感场景对应的场景图数据映射为图级别的特征向量，利用图特征向量匹配结果逆推遥感场景检索结果。针对场景图的学习，本文制作了一套包含2380对遥感场景图的数据集，包含24类实体，8类拓扑空间关系，9类方位关系，具备空间关系结构化的表征，空间拓扑信息和方位信息齐全等优势。试验表明：基于场景图的遥感场景检索结果，在实体类别、拓扑关系、方位关系的检索准确性高，特别是与国际上具有代表性的几类遥感场景检索方法相比，本文方法在拓扑关系和方位关系的检索精度指标上有较大提升。

关键词: 场景图, 遥感场景检索, 知识图谱, 图神经网络

Abstract:

Currently, most remote sensing scene retrieval is based on deep feature similarity matching of remote sensing images, which makes it difficult to directly represent the relationship information between scene entities and lacks a way to directly express spatial structure and semantics. Therefore, it cannot meet the complex retrieval requirements of users for remote sensing scene. This paper proposes a remote sensing scene retrieval method based on scene graph, which uses a graph neural network to map the scene graph data corresponding to the remote sensing scene to graph level feature vectors. The matching results of the graph feature vectors are used to reverse the remote sensing scene retrieval results. To train the graph neural network, this paper has created a dataset of 2380 pairs of remote sensing scene graphs, including 24 types of entities, 8 types of topological spatial relationships, and 9 types of directional relationships that have a structured representation of spatial relationships in remote sensing scenes. The spatial topological and orientational information is complete. The experiment shows that the remote sensing scene retrieval results based on scene graphs have high retrieval accuracy in entity categories, topological relationships, and orientation relationships. Especially compared with several representative international remote sensing scene retrieval methods, the scene retrieval accuracy indicators in topological and orientation relationships obtained by this method have a great improvement.

Key words: scene graph, remote sensing scene retrieval, knowledge graph, graph neural networks

中图分类号:

P237

唐佳怡, 童晓冲, 邱春平, 雷亚现, 雷毅, 宋好帅. 基于场景图的遥感场景检索方法[J]. 测绘学报, 2025, 54(1): 123-135.

Jiayi TANG, Xiaochong TONG, Chunping QIU, Yaxian LEI, Yi LEI, Haoshuai SONG. Remote sensing scene retrieval method based on scene graph[J]. Acta Geodaetica et Cartographica Sinica, 2025, 54(1): 123-135.

图/表 16

图1

表1

图2

图3

图4

表2

图5

表3

图6

表4

图7

表5

表6

表7

表8

表9

参考文献 31

[1]	张艺超. 面向遥感图像的深度哈希检索方法研究[D]. 西安: 西安光学精密机械研究所, 2023.
	ZHANG Yichao. Research on deep hash retrieval methods for remote sensing images[D]. Xi'an: Xi'an Institute of Optics and Precision Mechanics, 2023.
[2]	LI Y, MA J, ZHANG Y. Image retrieval from remote sensing big data: a survey[J]. Information Fusion, 2021, 67: 94-115.
[3]	周维勋. 基于深度学习特征的遥感影像检索研究[D]. 武汉: 武汉大学, 2019.
	ZHOU Weixun. Research on remote sensing image retrieval based on deep learning features[D]. Wuhan: Wuhan University, 2019.
[4]	周玉琢. 基于多注意力机制和语义对齐的跨模态遥感图文检索研究[D]. 武汉: 华中科技大学, 2022.
	ZHOU Yuzhuo. Research on cross modal remote sensing image and text retrieval based on multi attention mechanism and semantic alignment[D]. Wuhan: Huazhong University of Science and Technology, 2022.
[5]	CAO M, LI S, Li J, et al. Image-text retrieval: a survey on recent research and development[J/OL]. [2022-12-16]. https://arxiv.org/abs/2203.14713.
[6]	FAN D, DONG Y, ZHANG Y. Satellite image matching method based on deep convolutional neural network[J]. Journal of Geodesy and Geoinformation Science, 2019, 2(2): 90.
[7]	李彦胜, 张永军. 耦合知识图谱和深度学习的新一代遥感影像解译范式[J]. 武汉大学学报(信息科学版), 2022, 47(8): 1176-1190.
	LI Yansheng, ZHANG Yongjun. A new paradigm of remote sensing image interpretation by coupling knowledge graph and deep learning[J]. Geomatics and Information Science of Wuhan University, 2022, 47(8): 1176-1190.
[8]	JOHNSON J, KRISHNA R, STARK M, et al. Image retrieval using scene graphs[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society, 2015: 3668-3678.
[9]	YOON S, KANG W Y, JEON S, et al. Image-to-image retrieval by learning similarity between scene graphs[C]//Proceedings of 2021 AAAI Conference on Artificial Intelligence, 2021, 35(12): 10718-10726.
[10]	王旭东. 基于图理论的场景图检索方法研究与实现[D]. 西安: 西安电子科技大学, 2021.
	WANG Xudong. Research and implementation of scene image retrieval method based on graph theory[D]. Xi'an: Xidian University, 2021.
[11]	LIN Z, ZHU F, KONG Y, et al. SRSG and S2SG: a model and a dataset for scene graph generation of remote sensing images from segmentation results[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-11.
[12]	LI P, ZHANG D, WULAMU A, et al. Semantic relation model and dataset for remote sensing scene understanding[J]. ISPRS International Journal of Geo-Information, 2021, 10(7): 488.
[13]	LU X, WANG B, ZHENG X, et al. Exploring models and data for remote sensing image caption generation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 56(4): 2183-2195.
[14]	ZHU G, ZHANG L, JIANG Y, et al. Scene graph generation: a comprehensive survey[J/OL]. [2022-12-20]. https://arxiv.org/abs/2201.00443.
[15]	ZHOU J, XIE C, GONG S, et al. Data augmentation on graphs: a technical survey[J/OL]. [2022-11-22]. https://arxiv.org/abs/2212.09970.
[16]	CHENG G, WANG J, LI K, et al. Anchor-free oriented proposal generator for object detection[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-11.
[17]	XIA G S, HU J, HU F, et al. AID: a benchmark data set for performance evaluation of aerial scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(7): 3965-3981.
[18]	YANG Y, NEWSAM S. Bag-of-visual-words and spatial extensions for land-use classification[C]//Proceedings of the 18th SIGSPATIAL international conference on advances in geographic information systems. New York: Association for Computing Machinery, 2010: 270-279.
[19]	EGENHOFER M J, FRANZOSA R D. Point-set topological spatial relations[J]. International Journal of Geographical Information System, 1991, 5(2): 161-174.
[20]	王大力, 童晓冲, 孟丽, 等. 文本中空间信息的结构化建模与语义定位[J]. 测绘学报, 2023, 52(8): 1398-1410. DOI:. doi: 10.11947/j.AGCS.2023.20220066
	WANG Dali, TONG Xiaochong, MENG Li, et al. Structural modeling of spatial information in texts and semantic localization[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(8): 1398-1410. DOI:. doi: 10.11947/j.AGCS.2023.20220066
[21]	WANG D, TONG X, DAI C, et al. Voxel modeling and association of ubiquitous spatiotemporal information in natural language texts[J]. International Journal of Digital Earth, 2023, 16(1): 868-890.
[22]	CHEN K, LIU C, CHEN H, et al. RSPrompter: learning to prompt for remote sensing instance segmentation based on visual foundation model[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024.
[23]	RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of 2021 International Conference on Machine Learning. Cambridge: JMLR, 2021: 8748-8763.
[24]	CHEN T, KORNBLITH S, NOROUZI M, et al. A simple framework for contrastive learning of visual representations[C]//Proceedings of 2020 International Conference on Machine Learning. Cambridge: JMLR, 2020: 1597-1607.
[25]	LI Y, GU C, DULLIEN T, et al. Graph matching networks for learning the similarity of graph structured objects[C]//Proceedings of 2019 International Conference on Machine Learning. Cambridge: JMLR, 2019: 3835-3845.
[26]	LI Y, TARLOW D, BROCKSCHMIDT M, et al. Gated graph sequence neural networks[[J/OL]. [2022-12-23]. https://arxiv.org/abs/1511.05493.
[27]	LI Y, VAN GEMERT J. Deep unsupervised image hashing by maximizing bit entropy[C]//Proceedings of 2021 AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2021, 35(3): 2002-2010.
[28]	SU S, ZHANG C, HAN K, et al. Greedy Hash: towards fast optimization for accurate Hash coding in CNN[J]. Advances in Neural Information Processing Systems, 2018, 31.
[29]	HE K, FAN H, WU Y, et al. Momentum contrast for unsupervised visual representation learning[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society, 2020: 9729-9738.
[30]	DJOUFACK B L. CLIP-RS: a cross-modal remote sensing image retrieval based on CLIP, a Northern Virginia case study[D]. Blacksburg: Virginia Tech., 2022.
[31]	WANG Y, WANG L, LI Y, et al. A theoretical analysis of NDCG type ranking measures[C]//Proceedings of 2013 Conference on Learning Theory. Cambridge: JMLR, 2013: 25-54.

方法	NDCG
方法	K=5	K=10	K=20
Bihalf	0.902 3	0.901 9	0.902 6
Greedyhash	0.889 1	0.889 2	0.889 6
Simclr	0.906 1	0.906 5	0.907 4
Moco	0.882 6	0.883 0	0.883 2
CLIP-RS	0.941 8	0.942 7	0.943 1
本文方法	0.915 6	0.913 1	0.908 6

方法	NDCG
方法	K=5	K=10	K=20
Bihalf	0.679 8	0.646 5	0.615 2
Greedyhash	0.700 5	0.663 4	0.637 7
Simclr	0.774 9	0.736 1	0.686 5
Moco	0.706 5	0.673 5	0.623 8
CLIP-RS	0.888 6	0.869 1	0.828 4
本文方法	0.983 8	0.980 3	0.969 4

方法	NDCG
方法	K=5	K=10	K=20
Bihalf	0.636 8	0.641 1	0.653 5
Greedyhash	0.624 6	0.632 9	0.645 7
Simclr	0.720 1	0.722 3	0.727 7
Moco	0.654 8	0.654 1	0.661 0
CLIP-RS	0.624 7	0.635 5	0.653 3
本文方法	0.801 2	0.805 5	0.804 5

方法	NDCG
方法	K=5	K=10	K=20
本文方法（不进行特征级边数据增强）	0.910 1	0.903 7	0.902 6
本文方法（不进行拓扑级边数据增强）	0.913 5	0.909 7	0.905 7
本文方法（不进行特征级节点数据增强）	0.881 9	0.881 8	0.884 4
本文方法	0.915 6	0.913 1	0.908 6

方法	NDCG
方法	K=5	K=10	K=20
本文方法（不进行特征级边数据增强）	0.964 1	0.957 9	0.946 9
本文方法（不进行拓扑级边数据增强）	0.961 6	0.952 4	0.942 8
本文方法（不进行特征级节点数据增强）	0.957 1	0.948 5	0.939 8
原始设计	0.983 8	0.980 3	0.969 4

基于场景图的遥感场景检索方法

Remote sensing scene retrieval method based on scene graph

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 16

参考文献 31

相关文章 13

编辑推荐

Metrics

本文评价

[1]	张付兵, 孙群, 徐青, 马京振, 黄文君, 陈若虚. 随机森林和图神经网络支持下的河系自动分级与选取方法[J]. 测绘学报, 2025, 54(9): 1697-1711.
[2]	黄哲琨, 钱海忠, 蔡中祥, 王骁, 王俊威, 孔令辉. 基于图神经网络的多尺度网状河系分类匹配方法[J]. 测绘学报, 2025, 54(2): 371-384.
[3]	吴浩宇, 朱庆, 丁雨淋, 鲍榴, 刘利. 数据模型知识协同驱动的隧道围岩高精度数字孪生建模方法[J]. 测绘学报, 2025, 54(10): 1893-1906.
[4]	石岩, 王达, 邓敏, 杨学习. 时空异常探测：从数据驱动到知识驱动的内涵转变与实现路径[J]. 测绘学报, 2024, 53(8): 1493-1504.
[5]	孔博, 艾廷华, 杨敏, 吴昊, 余华飞, 肖天元. 等高线形态知识与图神经网络联合作用下的黄土地貌类型识别[J]. 测绘学报, 2024, 53(6): 1154-1164.
[6]	陈占龙, 鲁谢春, 徐永洋. 基于图顶点深度聚类的建筑物合并方法[J]. 测绘学报, 2024, 53(4): 736-749.
[7]	李彦胜, 吴敏郎, 张永军. 知识图谱约束深度网络的高分辨率遥感影像场景分类[J]. 测绘学报, 2024, 53(4): 677-688.
[8]	李佳, 李静, 刘海砚, 陆川伟, 陈晓慧, 刘俊楠, 石文. 地理知识图谱增强与多时空条件约束的轨迹预测[J]. 测绘学报, 2024, 53(10): 2021-2033.
[9]	魏智威, 肖屹, 童莹, 许文嘉, 王洋. 基于知识图谱的直线型建筑物模式识别方法[J]. 测绘学报, 2023, 52(8): 1355-1363.
[10]	慎利, 徐柱, 李志林, 刘万增, 崔秉良. 从地理信息服务到地理知识服务:基本问题与发展路径[J]. 测绘学报, 2021, 50(9): 1194-1202.
[11]	刘万增, 陈军, 翟曦, 李然, 王新鹏, 赵勇, 朱秀丽, 徐柱, 赵婷婷, 彭云璐, 慎利. 时空知识中心的研究进展与应用[J]. 测绘学报, 2021, 50(9): 1183-1193.
[12]	朱庆, 曾浩炜, 丁雨淋, 谢潇, 刘飞, 张利国, 李海峰, 胡翰, 张骏骁, 陈力, 陈琳, 张鹏程, 何华贵. 重大滑坡隐患分析方法综述[J]. 测绘学报, 2019, 48(12): 1551-1561.
[13]	蒋秉川, 万刚, 许剑, 李锋, 温荟琦. 多源异构数据的大规模地理知识图谱构建[J]. 测绘学报, 2018, 47(8): 1051-1061.

遥感图像	场景图

方法	NDCG
方法	K=5	K=10	K=20
本文方法（不进行特征级边数据增强）	0.775 7	0.764 2	0.757 7
本文方法（不进行拓扑级边数据增强）	0.770 3	0.770 7	0.768 4
本文方法（不进行特征级节点数据增强）	0.778 1	0.768 4	0.766 3
原始设计	0.801 2	0.805 5	0.804 5