Dynamic construction of high-resolution remote sensing image sample datasets and intelligent interpretation applications

doi:10.11947/j.AGCS.2024.20230469

Abstract

Abstract:

In the era of artificial intelligence, the interpretation of remote sensing images is moving towards automation and intelligence. High-quality sample datasets are crucial for this development. A massive amount of high-quality spatio-temporal geospatial information and derived products has been accumulated in China, which serve as an important source for deep learning-driven intelligent interpretation of remote sensing images. Leveraging existing data resources can promote the depth and breadth of artificial intelligence and remote sensing interpretation applications. Aiming at the limitations of existing sample datasets, such as regional restrictions, limited timeliness and sample types, this paper proposes a dynamic construction technique for a high-resolution remote sensing image intelligent interpretation sample datasets based on existing data resources. Firstly, a business-driven framework for sample datasets demand generation, dynamic construction, and intelligent application is proposed based on the characteristics analysis of publicly available sample datasets for feature extraction, land cover classification, and change detection. Secondly, this research investigates methods and implementation processes for sample generation based on historical interpretation results, as well as sample refinement through SAM (segment anything model) large model prompt learning-guided cleaning. Furthermore, a sample dataset with regional, temporal, scale, multi-sensor, and multi-type features is designed, along with spatial-temporal-land cover relationships. The study explores the dynamic reconstruction process of sample dataset quantification-retrieval-combination, achieving dynamic management of spatiotemporal samples and multidimensional retrieval. Finally, intelligent interpretation applications such as land cover classification, feature extraction, and change detection are conducted to validate the feasibility of the proposed methods. The aim is to promote the effective utilization of sample datasets based on existing foundational data, as well as the interconnection of sample construction-management-application and data-model-business, providing reference ideas for the construction and application of high-resolution remote sensing image intelligent interpretation sample datasets.

Key words: high-resolution remote sensing images, sample datasets, sample refinement, dynamic construction, intelligent interpretation, deep learning, land cover classification, change detection

CLC Number:

P237

Haiyan GU, Yi YANG, Haitao LI, Lijian SUN, Shaopeng DING, Shiqi LIU. Dynamic construction of high-resolution remote sensing image sample datasets and intelligent interpretation applications[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(6): 1165-1179.

Figures/Tables 23

Fig.1

Tab.1

Fig.2

Fig.3

Fig.4

Fig.5

Tab.2

Fig.6

Fig.7

Fig.8

Tab.3

Tab.4

Tab.5

Tab.6

Fig.9

Fig.10

Fig.11

Fig.12

Tab.7

Fig.13

Fig.14

Fig.15

Fig.16

References 41

[1]	李德仁, 童庆禧, 李荣兴, 等. 高分辨率对地观测的若干前沿科学问题[J]. 中国科学：地球科学, 2012, 42(6):805-813.
	LI Deren, TONG Qingxi, LI Rongxing, et al. Current issues in high-resolution Earth observation technology[J]. Scientia Sinica (Terrae), 2012, 42(6):805-813.
[2]	李德仁, 王密, 沈欣, 等. 从对地观测卫星到对地观测脑[J]. 武汉大学学报(信息科学版), 2017, 42(2):143-149.
	LI Deren, WANG Mi, SHEN Xin, et al. From earth observation satellite to earth observation brain[J]. Geomatics and Information Science of Wuhan University, 2017, 42(2):143-149.
[3]	张继贤, 顾海燕, 杨懿, 等. 高分辨率遥感影像智能解译研究进展与趋势[J]. 遥感学报, 2021, 25(11):2198-2210.
	ZHANG Jixian, GU Haiyan, YANG Yi, et al. Research progress and trend of high-resolution remote sensing imagery intelligent interpretation[J]. National Remote Sensing Bulletin, 2021, 25(11):2198-2210.
[4]	张继贤, 李海涛, 顾海燕, 等. 人机协同的自然资源要素智能提取方法[J]. 测绘学报, 2021, 50(8):1023-1032.DOI:10.11947/j.AGCS.2021.20210102.
	ZHANG Jixian, LI Haitao, GU Haiyan, et al. Study on man-machine collaborative intelligent extraction for natural resource features[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(8):1023-1032.DOI:10.11947/j.AGCS.2021.20210102.
[5]	冯权泷, 陈泊安, 李国庆, 等. 遥感影像样本数据集研究综述[J]. 遥感学报, 2022, 26(4):589-605.
	FENG Quanlong, CHEN Boan, LI Guoqing, et al. A review for sample datasets of remote sensing imagery[J]. National Remote Sensing Bulletin, 2022, 26(4):589-605.
[6]	龚健雅, 许越, 胡翔云, 等. 遥感影像智能解译样本库现状与研究[J]. 测绘学报, 2021, 50(8):1013-1022. DOI:10.11947/j.AGCS.2021.20210085.
	GONG Jianya, XU Yue, HU Xiangyun, et al. Status analysis and research of sample database for intelligent interpretation of remote sensing image[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(8):1013-1022.DOI:10.11947/j.AGCS.2021.20210085.
[7]	MNIH V. Machine learning for aerial image labeling[D]. Toronto: University of Toronto, 2013.
[8]	LU Xiaoqiang, WANG Binqiang, ZHENG Xiangtao, et al. Exploring models and data for remote sensing image caption generation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(4):2183-2195.
[9]	MAGGIORI E, TARABALKA Y, CHARPIAT G, et al. Can semantic labeling methods generalize to any city? The Inria aerial image labeling benchmark[C]//Proceedings of 2017 IEEE International Geoscience and Remote Sensing Symposium. Fort Worth: IEEE, 2017.
[10]	JI Shunping, WEI Shiqing, LU Meng. Fully convolutional networks for multisource building extraction from an open aerial and satellite imagery dataset[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(1):574-586.
[11]	YANG Yongke, XIAO Pengfeng, FENG Xuezhi, et al. Accuracy assessment of seven global land cover datasets over China[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2017, 125:156-173.
[12]	FRIEDLM A, SULLA-MENASHE D, TAN Bin, et al. MODIS collection 5 global land cover: algorithm refinements and characterization of new datasets[J]. Remote Sensing of Environment, 2010, 114(1):168-182.
[13]	JOHNSON B, XIE Zhixiao. Classifying a high resolution image of an urban area using super-object information[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2013, 83:40-49.
[14]	VOLPI M, FERRARI V. Semantic segmentation of urban scenes by learning local class interactions[C]//Proceedings of 2015 IEEE Conferenceon Computer Vision and Pattern Recognition Workshops. Boston: IEEE, 2015.
[15]	TONG Xinyi, XIA Guisong, LU Qikai, et al. Land-cover classification with high-resolution remote sensing images using transferable deep models[J]. Remote Sensing of Environment, 2020, 237:111322.
[16]	TIAN Shiqi, ZHENG Zhuo, MA Ailong, et al. Hi-UCD: a large-scale dataset for urban semantic change detection in remote sensing imagery[EB/OL]. [2023-10-08]. https://arxiv.org/pdf/2011.03247.
[17]	CHEN Hao, SHI Zhenwei. A spatial-temporal attention-based method and a new dataset for remote sensing image change detection[J]. Remote Sensing, 2020, 12(10):1662.
[18]	PETRIŞOR A. Assessment of the green infrastructure of bucharest using corine and urban atlas data[J]. Urbanism, Arhitectura, Constructii, 2015, 6(2):19-24.
[19]	LONG Yang, XIA Guisong, LI Shengyang, et al. On creating benchmark dataset for aerial image interpretation: reviews, guidances, and million-AID[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14:4205-4230.
[20]	ZUO Yi, LI Lingling, LIU Xu, et al. Robust instance-based semi-supervised learning change detection for remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62:1-15.
[21]	WANG Lukang, ZHANG Min, SHI Wenzhong. CS-WSCDNet: class activation mapping and segment anything model-based framework for weakly supervised change detection[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61:1-12.
[22]	RUSSELL B C, TORRALBA A, MURPHY K P, et al. LabelMe: a database and web-based tool for image annotation[J]. International Journal of Computer Vision, 2008, 77(1):157-173.
[23]	DUTTA A, ZISSERMAN A. The VIA annotation software for images, audio and video[C]//Proceedings of the 27th ACM International Conference on Multimedia. Nice: ACM Press, 2019: 2276-2279.
[24]	CASTREJON L, KUNDU K, URTASUN R, et al. Annotating object instances with a polygon-RNN[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017.
[25]	ACUNA D, LING Huan, KAR A, et al. Efficient interactive annotation of segmentation datasets with polygon-RNN++[C]//Proceedings of 2018 IEEE/CVF Conferenceon Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 859-868.
[26]	KIRILLOV A, MINTUN E, RAVI N, et al. Segment anything[EB/OL]. [2023-10-08]. https://arxiv.org/abs/2304.02643.
[27]	ZOU Xueyan, YANG Jianwei, ZHANG Hao, et al. Segment everything everywhere all at once[EB/OL]. [2023-10-08]. https://arxiv.org/pdf/2304.06718.
[28]	ZHANG Chaoning, HAN Dongshen, QIAO Yu, et al. Faster segment anything: towards lightweight SAM for mobile applications[EB/OL]. [2023-10-08]. https://arxiv.org/pdf/2306.14289.
[29]	XIONG Yunyang, VARADARAJAN B, WU Lemeng, et al. EfficientSAM: leveraged masked image pretraining for efficient segment anything[EB/OL]. [2023-10-08]. https://arxiv.org/pdf/2312.00863.
[30]	SONG Yanfei, PU Bangzheng, WANG Peng, et al. SAM-lightening: alightweight segment anything model with dilated flash attention to achieve 30 times acceleration[EB/OL]. [2023-10-08]. https://arxiv.org/pdf/2403.09195.
[31]	蒋正锋, 何韬, 施艳玲, 等. 融合卷积注意力机制与深度残差网络的遥感图像分类[J]. 激光杂志, 2022, 43(4):76-81.
	JIANG Zhengfeng, HE Tao, SHI Yanling, et al. Remote sensing image classification based on convolutional block attention module and deep residual network[J]. Laser Journal, 2022, 43(4):76-81.
[32]	李道纪, 郭海涛, 卢俊, 等. 遥感影像地物分类多注意力融和U型网络法[J]. 测绘学报, 2020, 49(8):1051-1064.DOI:10.11947/j.AGCS.2020.20190407.
	LI Daoji, GUO Haitao, LU Jun, et al. A remote sensing image classification procedure based on multilevel attention fusion U-Net[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(8):1051-1064. DOI:10.11947/j.AGCS.2020.20190407.
[33]	ZHENG Xianwei, HUAN Linxi, XIA Guisong, et al. Parsing very high resolution urban scene images by learning deep ConvNets with edge-aware loss[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 170:15-28.
[34]	张永生, 张振超, 童晓冲, 等. 地理空间智能研究进展和面临的若干挑战[J]. 测绘学报, 2021, 50(9):1137-1146.DOI:10.11947/j.AGCS.2021.20200420.
	ZHANG Yongsheng, ZHANG Zhenchao, TONG Xiaochong, et al. Progress and challenges of geospatial artificial intelligence[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(9):1137-1146.DOI:10.11947/j.AGCS.2021.20200420.
[35]	史文中, 张敏. 人工智能用于遥感目标可靠性识别：总体框架设计、现状分析及展望[J]. 测绘学报, 2021, 50(8):1049-1058.DOI:10.11947/j.AGCS.2021.20210095.
	SHI Wenzhong, ZHANG Min. Artificial intelligence for reliable object recognition from remotely sensed data: overall framework design, review and prospect[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(8):1049-1058.DOI:10.11947/j.AGCS.2021.20210095.
[36]	龚健雅, 张觅, 胡翔云, 等. 智能遥感深度学习框架与模型设计[J]. 测绘学报, 2022, 51(4):475-487.DOI:10.11947/j.AGCS.2022.20220027.
	GONG Jianya, ZHANG Mi, HU Xiangyun, et al. The design of deep learning framework and model for intelligent remote sensing[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(4):475-487.DOI:10.11947/j.AGCS.2022.20220027.
[37]	TAO Cheng, YANG Zhang, JAMES H. Network spacetime AI: concepts, methods and applications[J]. Journal of Geodesy and Geoinformation Science, 2022, 5(3):78-92.
[38]	GONG Jianya, JI Shunping. Photogrammetry and deep learning[J]. Journal of Geodesy and Geoinformation Science, 2018, 1(1):1-15.
[39]	陈军, 刘万增, 武昊, 等. 智能化测绘的基本问题与发展方向[J]. 测绘学报, 2021, 50(8):995-1005.DOI:10.11947/j.AGCS.2021.20210235.
	CHEN Jun, LIU Wanzeng, WU Hao, et al. Smart surveying and mapping: fundamental issues and research agenda[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(8):995-1005.DOI:10.11947/j.AGCS.2021.20210235.
[40]	陈军, 艾廷华, 闫利, 等. 智能化测绘的混合计算范式与方法研究[J/OL]. 测绘学报: 1-19 [2024-05-18]. http://kns.cnki.net/kcms/detail/11.2089.P.20240415.1049.002.html.
	CHEN Jun, AI Tinghua, YAN Li, et al. Hybrid computational paradigm and methods for intelligentized surveying and mapping[J/OL]. Acta Geodaetica et Cartographica Sinica: 1-19 [2024-05-18]. http://kns.cnki.net/kcms/detail/11.2089.P.20240415.1049.002.html.
[41]	CHEN Jun, LI Zhilin, LI songnian, et al. From digitalized to intelligentized surveying and mapping: fundamental issues and research agenda[J]. Journal of Geodesy and Geoinformation Science, 2022, 5(2):148-160. DOI:10.11947/j.JGGS.2022.0213.

地理国情信息分类码				深度学习的自定义编码
代码	一级	二级	三级	代码	类别
100	种植土地			1	水田
110		水田		1
120		旱地		2	旱地
130		果园		3	园地
131			乔灌果园	3
132			藤本果园	3
133			草本果园	3
140		茶园		3
150		桑园		3
160		橡胶园		3
170		苗圃		3
180		花圃		3
190		其他经济苗木		3
191			其他乔灌经济苗木	3
192			其他藤本经济苗木	3
193			其他草本经济苗木	3

项目	内容
分区	东北、华北、华东、华中、华南、西南、西北
时相	覆盖全年多时相，主要考虑夏季和冬季两个差异度最大的时相，其中冬季时相的时间跨度设定为12月—次年3月，夏季时相的时间跨度设定为6—9月
地形地貌	平地、丘陵地、山地、高山地
影像类型	分辨率优于2 m的高分辨率遥感影像（如北京二号、高分一号、高分二号、资源三号等）
数据集类型	要素提取、地表覆盖分类、变化检测

序号	字段名	字段简写	字段含义	填写规则	填写方式	类型	示例
1	FID	FID	序号ID	自增ID	自动	Int	27
2	TreeID	TreeID	Tree编码	样本的Tree编码，采用3位序列编码，同层最大编码不超过1000个节点	自动	String	003029
3	SamSet Name	YBMC	样本集名称	对应并总结概括样本集中样本内容	人工	String	内蒙古耕地
4	SampSet Nano	YBJX	样本集名称简写	样本集名称的简写形式	人工	String	蒙耕地
5	SampSetDate	JCJSJ	创建时间	采集时间	自动	Date	2022-08-30
6	Creator	JCJR	创建人	操作人员姓名	人工	String	张三
7	SampBak	JBak	备注	样本集的辅助性说明	人工	String	该样本集主要针对草原生态监测有关的科研样本
8	SampSet Link	YBJ	样本集	关联样本信息表	自动	String	YB3715_2022

序号	字段名	字段简写	字段含义	填写规则	填写方式	类型	示例
1	FID	FID	数据记录ID	自增ID	自动	Int	99
2	SampleID	YBID	样本ID	样本唯一标识码：数据源_X_Y	自动	Int	GF1293_75
3	Classification scheme	YBCS	分类体系码	—	自动	Int	2
4	Classification Code	CC	分类码	—	自动	Int
5	Longitude	X	经度	度（小数）	自动	Double	118.653 498 7
6	Latitude	Y	纬度	度（小数）	自动	Double	41.349 348 47
7	Elevation	Z	高程	整数（米）	自动	Double	155
8	DataSource	SJY	数据源	数据源信息表中序号	自动	Int	14
9	SampleWidth	YBKD	样本宽度	样本宽度（像素）	人工	Int	1024
10	Sample Height	YBGD	样本高度	样本高度（像素）	人工	Int	1024
11	SampleDate	CJSJ	采集时间	采集时间	自动	Date	2021-08-30
12	Operator	CJR	采集人	人名	人工	String	张某
13	Bak	Bak	备注	辅助性说明	人工	String	钢架大棚房
14	SampleImage	YBIM	样本影像	大对象	自动	Blob
15	SampleLabel	YBLB	样本标签	大对象	自动	Blob

序号	国产光学卫星名称	卫星名称简写	全色分辨率/m	多光谱分辨率/m	幅宽/km
1	北京二号	BJ2	0.8	3.2	24
2	高分一号	GF1	2	8	35
3	高分二号	GF2	0.8	3.2	45
4	高分六号	GF6	2	8	90
5	资源三号	ZY3	2.1	5.8	50
6	高景一号	GJ1	0.5	2	12