Remote sensing scene retrieval method based on scene graph

doi:10.11947/j.AGCS.2025.20230439

Abstract

Abstract:

Currently, most remote sensing scene retrieval is based on deep feature similarity matching of remote sensing images, which makes it difficult to directly represent the relationship information between scene entities and lacks a way to directly express spatial structure and semantics. Therefore, it cannot meet the complex retrieval requirements of users for remote sensing scene. This paper proposes a remote sensing scene retrieval method based on scene graph, which uses a graph neural network to map the scene graph data corresponding to the remote sensing scene to graph level feature vectors. The matching results of the graph feature vectors are used to reverse the remote sensing scene retrieval results. To train the graph neural network, this paper has created a dataset of 2380 pairs of remote sensing scene graphs, including 24 types of entities, 8 types of topological spatial relationships, and 9 types of directional relationships that have a structured representation of spatial relationships in remote sensing scenes. The spatial topological and orientational information is complete. The experiment shows that the remote sensing scene retrieval results based on scene graphs have high retrieval accuracy in entity categories, topological relationships, and orientation relationships. Especially compared with several representative international remote sensing scene retrieval methods, the scene retrieval accuracy indicators in topological and orientation relationships obtained by this method have a great improvement.

Key words: scene graph, remote sensing scene retrieval, knowledge graph, graph neural networks

CLC Number:

P237

Jiayi TANG, Xiaochong TONG, Chunping QIU, Yaxian LEI, Yi LEI, Haoshuai SONG. Remote sensing scene retrieval method based on scene graph[J]. Acta Geodaetica et Cartographica Sinica, 2025, 54(1): 123-135.

Figures/Tables 16

Fig. 1

Tab. 1

Fig. 2

Fig. 3

Fig. 4

Tab. 2

Fig. 5

Tab. 3

Fig. 6

Tab. 4

Fig. 7

Tab. 5

Tab. 6

Tab. 7

Tab. 8

Tab. 9

References 31

[1]	张艺超. 面向遥感图像的深度哈希检索方法研究[D]. 西安: 西安光学精密机械研究所, 2023.
	ZHANG Yichao. Research on deep hash retrieval methods for remote sensing images[D]. Xi'an: Xi'an Institute of Optics and Precision Mechanics, 2023.
[2]	LI Y, MA J, ZHANG Y. Image retrieval from remote sensing big data: a survey[J]. Information Fusion, 2021, 67: 94-115.
[3]	周维勋. 基于深度学习特征的遥感影像检索研究[D]. 武汉: 武汉大学, 2019.
	ZHOU Weixun. Research on remote sensing image retrieval based on deep learning features[D]. Wuhan: Wuhan University, 2019.
[4]	周玉琢. 基于多注意力机制和语义对齐的跨模态遥感图文检索研究[D]. 武汉: 华中科技大学, 2022.
	ZHOU Yuzhuo. Research on cross modal remote sensing image and text retrieval based on multi attention mechanism and semantic alignment[D]. Wuhan: Huazhong University of Science and Technology, 2022.
[5]	CAO M, LI S, Li J, et al. Image-text retrieval: a survey on recent research and development[J/OL]. [2022-12-16]. https://arxiv.org/abs/2203.14713.
[6]	FAN D, DONG Y, ZHANG Y. Satellite image matching method based on deep convolutional neural network[J]. Journal of Geodesy and Geoinformation Science, 2019, 2(2): 90.
[7]	李彦胜, 张永军. 耦合知识图谱和深度学习的新一代遥感影像解译范式[J]. 武汉大学学报(信息科学版), 2022, 47(8): 1176-1190.
	LI Yansheng, ZHANG Yongjun. A new paradigm of remote sensing image interpretation by coupling knowledge graph and deep learning[J]. Geomatics and Information Science of Wuhan University, 2022, 47(8): 1176-1190.
[8]	JOHNSON J, KRISHNA R, STARK M, et al. Image retrieval using scene graphs[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society, 2015: 3668-3678.
[9]	YOON S, KANG W Y, JEON S, et al. Image-to-image retrieval by learning similarity between scene graphs[C]//Proceedings of 2021 AAAI Conference on Artificial Intelligence, 2021, 35(12): 10718-10726.
[10]	王旭东. 基于图理论的场景图检索方法研究与实现[D]. 西安: 西安电子科技大学, 2021.
	WANG Xudong. Research and implementation of scene image retrieval method based on graph theory[D]. Xi'an: Xidian University, 2021.
[11]	LIN Z, ZHU F, KONG Y, et al. SRSG and S2SG: a model and a dataset for scene graph generation of remote sensing images from segmentation results[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-11.
[12]	LI P, ZHANG D, WULAMU A, et al. Semantic relation model and dataset for remote sensing scene understanding[J]. ISPRS International Journal of Geo-Information, 2021, 10(7): 488.
[13]	LU X, WANG B, ZHENG X, et al. Exploring models and data for remote sensing image caption generation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 56(4): 2183-2195.
[14]	ZHU G, ZHANG L, JIANG Y, et al. Scene graph generation: a comprehensive survey[J/OL]. [2022-12-20]. https://arxiv.org/abs/2201.00443.
[15]	ZHOU J, XIE C, GONG S, et al. Data augmentation on graphs: a technical survey[J/OL]. [2022-11-22]. https://arxiv.org/abs/2212.09970.
[16]	CHENG G, WANG J, LI K, et al. Anchor-free oriented proposal generator for object detection[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-11.
[17]	XIA G S, HU J, HU F, et al. AID: a benchmark data set for performance evaluation of aerial scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(7): 3965-3981.
[18]	YANG Y, NEWSAM S. Bag-of-visual-words and spatial extensions for land-use classification[C]//Proceedings of the 18th SIGSPATIAL international conference on advances in geographic information systems. New York: Association for Computing Machinery, 2010: 270-279.
[19]	EGENHOFER M J, FRANZOSA R D. Point-set topological spatial relations[J]. International Journal of Geographical Information System, 1991, 5(2): 161-174.
[20]	王大力, 童晓冲, 孟丽, 等. 文本中空间信息的结构化建模与语义定位[J]. 测绘学报, 2023, 52(8): 1398-1410. DOI:. doi: 10.11947/j.AGCS.2023.20220066
	WANG Dali, TONG Xiaochong, MENG Li, et al. Structural modeling of spatial information in texts and semantic localization[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(8): 1398-1410. DOI:. doi: 10.11947/j.AGCS.2023.20220066
[21]	WANG D, TONG X, DAI C, et al. Voxel modeling and association of ubiquitous spatiotemporal information in natural language texts[J]. International Journal of Digital Earth, 2023, 16(1): 868-890.
[22]	CHEN K, LIU C, CHEN H, et al. RSPrompter: learning to prompt for remote sensing instance segmentation based on visual foundation model[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024.
[23]	RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of 2021 International Conference on Machine Learning. Cambridge: JMLR, 2021: 8748-8763.
[24]	CHEN T, KORNBLITH S, NOROUZI M, et al. A simple framework for contrastive learning of visual representations[C]//Proceedings of 2020 International Conference on Machine Learning. Cambridge: JMLR, 2020: 1597-1607.
[25]	LI Y, GU C, DULLIEN T, et al. Graph matching networks for learning the similarity of graph structured objects[C]//Proceedings of 2019 International Conference on Machine Learning. Cambridge: JMLR, 2019: 3835-3845.
[26]	LI Y, TARLOW D, BROCKSCHMIDT M, et al. Gated graph sequence neural networks[[J/OL]. [2022-12-23]. https://arxiv.org/abs/1511.05493.
[27]	LI Y, VAN GEMERT J. Deep unsupervised image hashing by maximizing bit entropy[C]//Proceedings of 2021 AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2021, 35(3): 2002-2010.
[28]	SU S, ZHANG C, HAN K, et al. Greedy Hash: towards fast optimization for accurate Hash coding in CNN[J]. Advances in Neural Information Processing Systems, 2018, 31.
[29]	HE K, FAN H, WU Y, et al. Momentum contrast for unsupervised visual representation learning[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society, 2020: 9729-9738.
[30]	DJOUFACK B L. CLIP-RS: a cross-modal remote sensing image retrieval based on CLIP, a Northern Virginia case study[D]. Blacksburg: Virginia Tech., 2022.
[31]	WANG Y, WANG L, LI Y, et al. A theoretical analysis of NDCG type ranking measures[C]//Proceedings of 2013 Conference on Learning Theory. Cambridge: JMLR, 2013: 25-54.

方法	NDCG
方法	K=5	K=10	K=20
Bihalf	0.902 3	0.901 9	0.902 6
Greedyhash	0.889 1	0.889 2	0.889 6
Simclr	0.906 1	0.906 5	0.907 4
Moco	0.882 6	0.883 0	0.883 2
CLIP-RS	0.941 8	0.942 7	0.943 1
本文方法	0.915 6	0.913 1	0.908 6

方法	NDCG
方法	K=5	K=10	K=20
Bihalf	0.679 8	0.646 5	0.615 2
Greedyhash	0.700 5	0.663 4	0.637 7
Simclr	0.774 9	0.736 1	0.686 5
Moco	0.706 5	0.673 5	0.623 8
CLIP-RS	0.888 6	0.869 1	0.828 4
本文方法	0.983 8	0.980 3	0.969 4

方法	NDCG
方法	K=5	K=10	K=20
Bihalf	0.636 8	0.641 1	0.653 5
Greedyhash	0.624 6	0.632 9	0.645 7
Simclr	0.720 1	0.722 3	0.727 7
Moco	0.654 8	0.654 1	0.661 0
CLIP-RS	0.624 7	0.635 5	0.653 3
本文方法	0.801 2	0.805 5	0.804 5

方法	NDCG
方法	K=5	K=10	K=20
本文方法（不进行特征级边数据增强）	0.910 1	0.903 7	0.902 6
本文方法（不进行拓扑级边数据增强）	0.913 5	0.909 7	0.905 7
本文方法（不进行特征级节点数据增强）	0.881 9	0.881 8	0.884 4
本文方法	0.915 6	0.913 1	0.908 6

方法	NDCG
方法	K=5	K=10	K=20
本文方法（不进行特征级边数据增强）	0.964 1	0.957 9	0.946 9
本文方法（不进行拓扑级边数据增强）	0.961 6	0.952 4	0.942 8
本文方法（不进行特征级节点数据增强）	0.957 1	0.948 5	0.939 8
原始设计	0.983 8	0.980 3	0.969 4