Geo-cognitive models and methods for intelligent interpretation of remotely sensed big data

doi:10.11947/j.AGCS.2022.20220279

Abstract

Abstract: With the explosive growth of remotely sensed data and computing power, and the breakthrough of intelligent analysis algorithms, there is an urgent need to improve the capabilities to match in remotely sensed big data processing and analysis. Aiming at the crucial problems of coupling association and cross fusion of remotely sensed big data intelligent processing and geographical cognition in complex scenes, this paper analyzes the characteristics and relationships between remotely sensed big data and geographical science, puts forward the idea of building deep network of multimodal knowledge fusion and intelligent interpretation of remotely sensed data for geographical cartography, and establishes the framework of remotely sensed big data intelligent processing and application system. A general high-resolution remote sensing intelligent processing system for technology development and an intelligent application platform for industry applications are proposed, respectively, in order to boost the technological innovations and engineering applications of intelligent interpretation using remotely sensed big data.

Key words: remotely sensed big data, geography, intelligent image processing, land resources surveys

CLC Number:

P237

ZHANG Bing, YANG Xiaomei, GAO Lianru, MENG Yu, SUN Xian, XIAO Chenchao, NI Li. Geo-cognitive models and methods for intelligent interpretation of remotely sensed big data[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(7): 1398-1415.

References

[1] 陈军,武昊,张继贤,等.自然资源调查监测技术体系构建的方向与任务[J].地理学报, 2022, 77(5):1041-1055. CHEN Jun, WU Hao, ZHANG Jixian, et al. Building natural resources surveying and monitoring technological system:direction and research agenda[J]. Acta Geographica Sinica, 2022, 77(5):1041-1055.
[2] 刘南威,郭有立,张争胜.综合自然地理学[M]. 3版.北京:科学出版社, 2009:30-31. LIU Nanwei, GUO Youli, ZHANG Zhengsheng. Integrated physical geography[M].3rd ed. Beijing:Science Press, 2009:30-31.
[3] 张兵.当代遥感科技发展的现状与未来展望[J].中国科学院院刊, 2017, 32(7):774-784. ZHANG Bing. Current status and future prospects of remote sensing[J]. Bulletin of the Chinese Academy of Sciences, 2017, 32(7):774-784.
[4] ZHANG Bing, CHEN Zhengchao, PENG Dailiang, et al. Remotely sensed big data:evolution in model development for information extraction[point of view] [J]. Proceedings of the IEEE, 2019, 107(12):2294-2301.
[5] 张兵.遥感大数据时代与智能信息提取[J].武汉大学学报(信息科学版), 2018, 43(12):1861-1871. ZHANG Bing. Remotely sensed big data era and intelligent information extraction[J]. Geomatics and Information Science of Wuhan University, 2018, 43(12):1861-1871.
[6] 朱建章,石强,陈凤娥,等.遥感大数据研究现状与发展趋势[J].中国图象图形学报, 2016, 21(11):1425-1439. ZHU Jianzhang, SHI Qiang, CHEN Feng'e, et al. Research status and development trends of remote sensing big data[J]. Journal of Image and Graphics, 2016, 21(11):1425-1439.
[7] 郑度,欧阳,周成虎.对自然地理区划方法的认识与思考[J].地理学报, 2008, 63(6):563-573. ZHENG Du, OU Yang, ZHOU Chenghu. Understanding of and thinking over geographical regionalization methodology[J]. Acta Geographica Sinica, 2008, 63(6):563-573.
[8] BLASCHKE T, HAY G J, KELLY M, et al. Geographic object-based image analysis-towards a new paradigm[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2014, 87:180-191.
[9] 王志华,杨晓梅,周成虎.面向遥感大数据的地学知识图谱构想[J].地球信息科学学报, 2021, 23(1):16-28. WANG Zhihua, YANG Xiaomei, ZHOU Chenghu. Geographic knowledge graph for remote sensing big data[J]. Journal of Geo-Information Science, 2021, 23(1):16-28.
[10] 闾国年,俞肇元,袁林旺,等.地图学的未来是场景学吗?[J].地球信息科学学报, 2018, 20(1):1-6. LV Guonian, YU Zhaoyuan, YUAN Linwang, et al. Is the future of cartography the scenario science?[J]. Journal of Geo-Information Science, 2018, 20(1):1-6.
[11] TOBLER W R. A computer movie simulating urban growth in the Detroit region[J]. Economic Geography, 1970, 46(S1):234-240.
[12] GOODCHILD M F. The validity and usefulness of laws in geographic information science and geography[J]. Annals of the Association of American Geographers, 2004, 94(2):300-303.
[13] ZHU Axing, LU Guonian, LIU Jing, et al. Spatial prediction based on Third Law of Geography[J]. Annals of GIS, 2018, 24(4):225-240.
[14] 唐宁,安玮,徐昊骙,等.从数据到表征:人类认知对人工智能的启发[J].应用心理学, 2018, 24(1):3-14. TANG Ning, AN Wei, XU Haokui, et al. From "data" to "representation":how human cognition can inspire artificial intelligence[J]. Chinese Journal of Applied Psychology, 2018, 24(1):3-14.
[15] YOKOYA N, GROHNFELDT C, CHANUSSOT J. Hyperspectral and multispectral data fusion:a comparative review of the recent literature[J]. IEEE Geoscience and Remote Sensing Magazine, 2017, 5(2):29-56.
[16] HONG Danfeng, HEWei, YOKOYA N, et al. Interpretable hyperspectral artificial intelligence:when nonconvex modeling meets hyperspectral remote sensing[J]. IEEE Geoscience and Remote Sensing Magazine, 2021, 9(2):52-87.
[17] XIE Qi, ZHOU Minghao, ZHAO Qian, et al. MHF-Net:an interpretable deep network for multispectral and hyperspectral image fusion[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(3):1457-1473. DOI:10.1109/TPAMI.2020.3015691.
[18] ZHENG Ke, GAO Lianru, LIAO Wenzhi, et al. Coupled convolutional neural network with adaptive response function learning for unsupervised hyperspectral super resolution[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(3):2487-2502. DOI:10.1109/TGRS.2020.3006534.
[19] YAO Jing, HONG Danfeng, CHANUSSOT J, et al. Cross-attention in coupled unmixing nets for unsupervised hyperspectral super-resolution[C]//Proceedings of the 16th European Conference on Computer Vision. Glasgow, UK:Springer, 2020:208-224.
[20] LI Jiaxin, ZHENG Ke, YAO Jing, et al. Deep unsupervised blind hyperspectral and multispectral data fusion[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19:60073055. DOI:10.1109/LGRS.2022.3151779.
[21] 张良培,沈焕锋.遥感数据融合的进展与前瞻[J].遥感学报, 2016, 20(5):1050-1061. ZHANG Liangpei, SHEN Huanfeng. Progress and future of remote sensing data fusion[J]. Journal of Remote Sensing, 2016, 20(5):1050-1061.
[22] LI Yunfei,LI Jun, HE Lin, et al. A new sensor bias-driven spatio-temporal fusion model based on convolutional neural networks[J]. Science China Information Sciences, 2020, 63(4):140302. DOI:10.1007/s11432-019-2805-y.
[23] TAN Zhenyu, GAO Meiling, LI Xinghua, et al. A flexible reference-insensitive spatiotemporal fusion model for remote sensing images using conditional generative adversarial network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60:5601413.
[24] GIANINETTO M, RUSMINI M, MARCHESI A, et al. Integration of COSMO-SkyMed and GeoEye-1 data with object-based image analysis[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2015, 8(5):2282-2293.
[25] HANG Renlong, LI Zhu, GHAMISI P, et al. Classification of hyperspectral and LiDAR data using coupled CNNs[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(7):4939-4950.
[26] YIN Jiadi, DONG Jinwei, HAMM N A S, et al. Integrating remote sensing and geospatial big data for urban land use mapping:a review[J]. International Journal of Applied Earth Observation and Geoinformation, 2021, 103:102514.
[27] FAN Runyu, FENG Ruyi, HAN Wei, et al. Urban functional zone mapping with a bibranch neural network via fusing remote sensing and social sensing data[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14:11737-11749.
[28] SRIVASTAVA S, VARGAS-MUÑOZ J E, TUIA D. Understanding urban landuse from the above and ground perspectives:a deep learning, multimodal solution[J]. Remote Sensing of Environment, 2019, 228:129-143.
[29] HONG Danfeng, GAO Lianru, YOKOYA N, et al. More diverse means better:multimodal deep learning meets remote-sensing imagery classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(5):4340-4354.
[30] NGIAM J, KHOSLA A, KIM M, et al. Multimodal deep learning[C]//Proceedings of the 28th International Conference on International Conference on Machine Learning. Madison, WI, USA:Omnipress, 2011:686-696.
[31] HONG Danfeng, GAO Lianru, HANG Renlong, et al. Deep encoder-decoder networks for classification of hyperspectral and LiDAR data[J]. IEEE Geoscience and Remote Sensing Letters, 2020, 19:5500205.
[32] HAN Zhu, HONG Danfeng, GAO Lianru, et al. Multi-modal hyperspectral unmixing:insights from attention networks[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:5524913.
[33] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal, Canada:MIT Press, 2014, 2:2672-2680.
[34] HONG Danfeng, YAO Jing, MENG Deyu, et al. Multimodal GANs:toward crossmodal hyperspectral-multispectral image segmentation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(6):5103-5113.
[35] ZHAO Yunpu, SHEN Shikun, HU Jiarui, et al. Cloud removal using multimodal GAN with adversarial consistency loss[J]. IEEE Geoscience and Remote Sensing Letters, 2021, 19:8015605.
[36] LIU Xun, HONG Danfeng, CHANUSSOT J, et al. Modality translation in remote sensing time series[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60:5401614.
[37] WU Xin, HONG Danfeng, CHANUSSOT J. Convolutional neural networks for multimodal remote sensing data classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60:5517010.
[38] 李彦胜,张永军.耦合知识图谱和深度学习的新一代遥感影像解译范式[J/OL].武汉大学学报(信息科学版), 2021:1-23.[2002-01-17]. https://doi.org/10.13203/j.whugis20210652. LI Yansheng, ZHANG Yongjun. A new paradigm of remote sensing image interpretation by coupling knowledge graph and deep learning[J/OL]. Geomatics and Information Science of Wuhan University, 2021:1-23.[2002-01-17].https://doi.org/10.13203/j.whugis20210652.
[39] HONG Danfeng, YOKOYA N, XIA Guisong, et al. X-ModalNet:a semi-supervised deep cross-modal network for classification of remote sensing data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 167:12-23.
[40] PATHAK D, AGRAWAL P, EFROS A A, et al. Curiosity-driven exploration by self-supervised prediction[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu, HI, USA:IEEE, 2017:488-489.
[41] HONG Danfeng, GAO Lianru, YAO Jing, et al. Endmember-guided unmixing network (EGU-Net):a general deep learning framework for self-supervised hyperspectral unmixing[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021:1-14. DOI:10.1109/TNNLS.2021.3082289.
[42] 王家耀.地图学原理与方法[M]. 2版.北京:科学出版社, 2014. WANG Jiayao. Principles and methods of cartography[M]. 2nd ed. Beijing:Science Press, 2014.
[43] 赵忠明,周天颖,严泰来.空间信息技术原理及其应用.上册[M].北京:科学出版社, 2013. ZHAO Zhongming, ZHOU Tianying, YAN Tailai. Principles and applications of spatial information technology. Volume 1[M].Beijing:Science Press, 2013.
[44] 龚明劼,张鹰,张芸.卫星遥感制图最佳影像空间分辨率与地图比例尺关系探讨[J].测绘科学, 2009, 34(4):232-233, 60. GONG Mingjie, ZHANG Ying, ZHANG Yun. The discussion of the relation between the scale and the best spatial resolution in making cartographies with RS[J]. Science of Surveying and Mapping, 2009, 34(4):232-233, 60.
[45] 李莉,强跃,何泽平.卫星遥感制图最佳空间分辨率与地图比例尺的适配关系研究[J].安徽农业科学, 2011, 39(30):18996-18997, 19033. LI Li, QIANG Yue, HE Zeping. Study on the relationship between the best spatial resolution and the map scale in the satellite remote sensing cartographies[J]. Journal of Anhui Agricultural Sciences, 2011, 39(30):18996-18997, 19033.
[46] 骆继花,王鸿燕,谢志英.地图比例尺与遥感影像分辨率的关系探讨[J].测绘与空间地理信息, 2015, 38(12):61-64, 71. LUO Jihua, WANG Hongyan, XIE Zhiying. The discussion of the relation between map scale and remote sensing image resolution[J]. Geomatics&Spatial Information Technology, 2015, 38(12):61-64, 71.
[47] 阎守邕,王涛,刘亚岚,等.遥感影像人机交互判读系统及其技术特点[J].遥感学报, 2002, 6(3):198-204. YAN Shouyong, WANG Tao, LIU Yalan, et al. An interactive image interpretation system and its characteristics[J]. Journal of Remote Sensing, 2002, 6(3):198-204.
[48] 刘清,吴文魁,张斌才.遥感影像自动解译与变化检测方法研究与应用[J].测绘与空间地理信息, 2020, 43(12):122-125, 129. LIU Qing, WU Wenkui, ZHANG Bincai. Research and application of remote sensing image automatic interpretation and change detection[J]. Geomatics&Spatial Information Technology, 2020, 43(12):122-125, 129.
[49] 张继贤,顾海燕,杨懿,等.高分辨率遥感影像智能解译研究进展与趋势[J].遥感学报, 2021, 25(11):2198-2210. ZHANG Jixian, GU Haiyan, YANG Yi, et al. Research progress and trend of high-resolution remote sensing imagery intelligent interpretation[J]. Journal of Remote Sensing, 2021, 25(11):2198-2210.
[50] SAINI R, GHOSH S K. Crop classification on single date sentinel-2 imagery using random forest and suppor vector machine[C]//Proceedings of 2018 International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. Dehradun, India:ISPRS, 2018:683-688.
[51] GONG Peng, LIU Han, ZHANG Meinan, et al. Stable classification with limited sample:transferring a 30m resolution sample set collected in 2015 to mapping 10m resolution global land cover in 2017[J]. Science Bulletin, 2019, 64(6):370-373.
[52] YANG Ning, LIU Diyou, FENG Quanlong, et al. Large-scale crop mapping based on machine learning and parallel computation with grids[J]. Remote Sensing, 2019, 11(12):1500.
[53] WU Guangming, GUO Yimin, SONG Xiaoya, et al. A stacked fully convolutional networks with feature alignment framework for multi-label land-cover segmentation[J]. Remote Sensing, 2019, 11(9):1051.
[54] ZHANG Wei, TANG Ping, ZHAO Lijun. Fast and accurate land-cover classification on medium-resolution remote-sensing images using segmentation models[J]. International Journal of Remote Sensing, 2021, 42(9):3277-3301.
[55] 韩衍欣,蒙继华.面向地块的农作物遥感分类研究进展[J].国土资源遥感, 2019, 31(2):1-9. HAN Yanxin, MENG Jihua. A review of per-field crop classification using remote sensing[J]. Remote Sensing for Land&Resources, 2019, 31(2):1-9.
[56] BAKKER J. Every field in Minnesota:building a geographically scalable satellite imagery analytics system for mapping crop fields[D]. Minnesota:University of Minnesota, 2020.
[57] ZHANG Xiao, LIU Liangyun, WU Changshan, et al. Development of a global 30 m impervious surface map using multisource and multitemporal remote sensing datasets with the Google Earth Engine platform[J]. Earth System Science Data, 2020, 12(3):1625-1648.
[58] YANG Jie, HUANG Xin. The 30 m annual land cover dataset and its dynamics in China from 1990 to 2019[J]. Earth System Science Data, 2021, 13(8):3907-3925.
[59] HELBER P, BISCHKE B, DENGEL A, et al. EuroSAT:a novel dataset and deep learning benchmark for land use and land cover classification[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12(7):2217-2226.
[60] XIA Guisong, HU Jingwen, HU Fan, et al. AID:a benchmark data set for performance evaluation of aerial scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(7):3965-3981.
[61] CHENG Gong, HAN Junwei, LU Xiaoqiang. Remote sensing image scene classification:benchmark and state of the art[J]. Proceedings of the IEEE, 2017, 105(10):1865-1883.
[62] CHENG Gong, HAN Junwei, ZHOU Peicheng, et al. Multi-class geospatial object detection and geographic image classification based on collection of part detectors[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2014, 98:119-132.
[63] LI Ke, WAN Gang, CHENG Gong, et al. Object detection in optical remote sensing images:a survey and a new benchmark[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 159:296-307.
[64] SUN Xian, WANG Peijin, YAN Zhiyuan, et al. FAIR1M:a benchmark dataset for fine-grained object recognition in high-resolution remote sensing imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2022, 184:116-130.
[65] JI Shunping, WEI Shiqing, LU Meng. Fully convolutional networks for multisource building extraction from an open aerial and satellite imagery data set[J]. IEEE Transactions on Geoscience and Remote Sensing. 2019, 57:574-586.
[66] ZHANG Mi, HU Xiangyun, ZHAO Like, et al. Learning dual multi-scale manifold ranking for semantic segmentation of high-resolution images[J]. Remote Sensing, 2017, 9(5):500.
[67] BENEDEK C, SZIRANYI T. Change detection in optical aerial images by a multilayer conditional mixed Markov model[J]. IEEE Transactions on Geoscience and Remote Sensing, 2009, 47(10):3416-3430.
[68] LEBEDEV M A,VIZILTER Y V, VYGOLOV O V, et al. Change detection in remote sensing images using conditional adversarial networks[C]//Proceedings of the International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. Riva del Garda, Italy:ISPRS, 2018:565-571.
[69] CHEN Hao, SHI Zhenwei. A spatial-temporal attention-based method and a new dataset for remote sensing image change detection[J]. Remote Sensing, 2020, 12(10):1662.
[70] 李四海,刘振民,何广顺,等.天津海域遥感图像解译应用系统[J].海洋测绘, 2005, 25(4):13-15. LI Sihai,LIU Zhenmin, HE Guangshun, et al. An application system of remote sensing imagery interpretation for Tianjin coast area[J]. Hydrographic Surveying and Charting, 2005, 25(4):13-15.
[71] 路志越,毛炜青.上海市海量遥感影像管理与共享服务云平台建设与应用研究[J].上海国土资源, 2016, 37(2):96-99. LU Zhiyue, MAO Weiqing. Research on the construction and application of a management and sharing platform for large quantities of remote sensing image data in Shanghai[J]. Shanghai Land&Resources, 2016, 37(2):96-99.
[72] 吕军,石岚.基于高分辨率遥感影像数据处理与信息提取系统的设计及实现[J].智能计算机与应用, 2020, 10(4):299-302, 305. LV Jun, SHI Lan. Design and implementation of data processing and information extraction system based on high resolution remote sensing image[J]. Intelligent Computer and Applications, 2020, 10(4):299-302, 305.
[73] GORELICK N, HANCHER M, DIXON M, et al. Google Earth Engine:planetary-scale geospatial analysis for everyone[J]. Remote Sensing of Environment, 2017, 202:18-27.
[74] DONG Jinwei, XIAO Xiangming, KOU Weili, et al. Tracking the dynamics of paddy rice planting area in 1986-2010 through time series Landsat images and phenology-based algorithms[J]. Remote Sensing of Environment, 2015, 160:99-113.
[75] DONG Jinwei, XIAO Xiangming, MENARGUEZ M A, et al. Mapping paddy rice planting area in northeastern Asia with Landsat 8 images, phenology-based algorithm and Google Earth Engine[J]. Remote Sensing of Environment, 2016, 185:142-154.
[76] PHALKE A R, ÖZDOĞAN M, THENKABAIL P S, et al. Mapping croplands of Europe, middle east, Russia, and central Asia using landsat, random forest, and Google Earth Engine[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 167:104-122.
[77] TIAN Fuyou, WU Bingfang, ZENG Hongwei, et al. Efficient identification of corn cultivation area with multitemporal synthetic aperture radar and optical images in the Google Earth Engine cloud platform[J]. Remote Sensing, 2019, 11(6):629.
[78] 图雅.基于物种分布模型的中国针茅属植物潜在分布区及其影响因子分析[D].北京:北京林业大学, 2020. TU Ya. Based on species distribution model analysis suitable distribution area of stipa and the correlation with climate factors in China[D]. Beijing:Beijing Forestry University, 2020.
[79] 阿迪力·艾合麦提.基于生态位模型的新疆马鹿三个亚种生境评价[D].乌鲁木齐:新疆大学, 2020. AHMAT A. Habitat assessment of three subspecies of red deer in Xinjiang based on niche model[D].Urumqi:Xinjiang University, 2020.
[80] 高燕.越南大陆海岸线遥感智能解译方法研究[D].郑州:信息工程大学, 2014. GAO Yan. Research on Vietnam mainland coastline intelligence interpretation with remote sensing images[D].Zhengzhou:Information Engineering University, 2014.
[81] 刘炫麟.基于无人机航拍图像的植物种类智能识别研究[D].长沙:中南林业科技大学, 2021. LIU Xuanlin. Research on intelligent identification of plant species based on UAV aerial images[D].Changsha:Central South University of Forestry and Technology, 2021.
[82] 陈柳.基于无人机影像的一种农田景观小尺度地物的分类方法研究[D].沈阳:沈阳农业大学, 2020. CHEN Liu. A classification method of farmland landscape small-scale features based on UAV image[D]. Shenyang:Shenyang Agricultural University, 2020.
[83] 方灿明.场景解译框架下的高速铁路沿线建筑物隐患自动识别[D].成都:西南交通大学, 2017. FANG Canming. Automatic recognition of potential hazardous buildings along the high-speed railway in the framework of scene interpretation[D]. Chengdu:Southwest Jiaotong University, 2017.
[84] 李二珠.遥感图像场景深度学习与应用研究——以城市结构类型识别为例[D].南京:南京大学, 2017. LI Erzhu. Research on remotely sensed image scene deep learning and application-A case study of urban structure types recognition[D]. Nanjing:Nanjing University, 2017.