Building extraction via convolutional neural networks from an open remote sensing building dataset

doi:10.11947/j.AGCS.2019.20180206

Abstract

Abstract: Automatic extraction of buildings from remote sensing images is significant to city planning, popular estimation, map making and updating.We report several important developments in building extraction. Automatic building recognition from remote sensing data has been a scientific challenge of more than 30 years. Traditional methods based on empirical feature design can hardly realize automation. Advanced deep learning based methods show prospects but have two limitations now. Firstly, large and accurate building datasets are lacking while such dataset is the necessary fuel for deep learning. Secondly, the current researches only concern building's pixel wise semantic segmentation and the further extractions on instance-level and vector-level are urgently required. This paper proposes several solutions. First, we create a large, high-resolution, accurate and open-source building dataset, which consists of aerial and satellite images with both raster and vector labels. Second,we propose a novel structure based on fully neural network which achieved the best accuracy of semantic segmentation compared to most recent studies. Third, we propose a building instance segmentation method which expands the current studies of pixel-level segmentation to building-level segmentation. Experiments proved our dataset's superiority in accuracy and multi-usage and our methods' advancement. It is expected that our researches might push forward the challenging building extraction study.

Key words: building extraction, semantic segmentation, instance segmentation, convolutional neural network, deep learning

CLC Number:

P237

JI Shunping, WEI Shiqing. Building extraction via convolutional neural networks from an open remote sensing building dataset[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(4): 448-459.

References

[1] LIOW Y T, PAVLIDIS T. Use of shadows for extracting buildings in aerial images[J]. Computer Vision, Graphics and Image Processing, 1990, 49(2):242-277.
[2] SIRMACEK B, UNSALAN C. Building detection from aerial images using invariant color features and shadow information[C]//Proceedings of the 23rd International Symposium on Computer and Information Sciences. Istanbul, Turkey:IEEE, 2008:1-5.
[3] ZHONG Shenghua, HUANG Jianjun, XIE Weixin. A new method of building detection from a single aerial photograph[C]//Proceedings of the 9th International Conference on Signal Processing. Beijing, China:IEEE, 2008:1219-1222.
[4] ZHANG Yun. Optimisation of building detection in satellite images by combining multispectral classification and texture filtering[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 1999, 54(1):50-60.
[5] 艾泽天. LiDAR辅助遥感影像建筑物分类识别和提取研究[D]. 焦作:河南理工大学, 2011. AI Zetian. Research on classification recognition and extraction of buildings by RS image assisted by LiDAR data[D]. Jiaozuo:Henan Polytechnic University, 2011.
[6] LI Yong, WU Huayi. Adaptive building edge detection by combining LiDAR data and aerial images[J]. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Science, 2008(37):197-202.
[7] FERRAIOLI G. Multichannel InSAR building edge detection[J]. IEEE Transactions on Geoscience and Remote Sensing, 2010, 48(3):1224-1231.
[8] 吕凤华, 舒宁, 龚龑, 等. 利用多特征进行航空影像建筑物提取[J]. 武汉大学学报(信息科学版), 2017, 42(5):656-660. LÜ Fenghua, SHU Ning, GONG Yan, et al. Regular building extraction from high resolution image based on multilevel-features[J]. Geomatics and Information Science of Wuhan University, 2017, 42(5):656-660.
[9] DUNAEVA A V, KORNILOV F A. Specific shape building detection from aerial imagery in infrared range[J]. Vychislitelnaya Matematika Inform., 2017, 6(3):84-100.
[10] 吴炜, 骆剑承, 沈占锋, 等. 光谱和形状特征相结合的高分辨率遥感图像的建筑物提取方法[J]. 武汉大学学报(信息科学版), 2012, 37(7):800-805. WU Wei, LUO Jiancheng, SHEN Zhanfeng, et al. Building extraction from high resolution remote sensing imagery based on spatial-spectral method[J]. Geomatics and Information Science of Wuhan University, 2012, 37(7):800-805.
[11] AWRANGJEB M, ZHANG C, FRASER C S. Improved building detection using texture information[J]. International Archives of the Photogrammetry, 2011, ⅩⅩⅩⅧ-3/W22:143-148.
[12] 何春阳, 曹鑫, 史培军, 等. 基于Landsat7 ETM⁺全色数据纹理和结构信息复合的城市建筑信息提取[J]. 武汉大学学报(信息科学版), 2004, 29(9):800-804. HE Chunyang, CAO Xin, SHI Peijun, et al. Incorporation of texture and structure information for urban building detection by using Landsat7 ETM⁺ panchromatic image[J]. Geomatics and Information Science of Wuhan University, 2004, 29(9):800-804.
[13] CHEN Dongyue, SHANG Shibo, WU Chengdong. Shadow-based building detection and segmentation in high-resolution remote sensing image[J]. Journal of Multimedia, 2014, 9(1):181-188.
[14] 黄明. 联合LiDAR数据和遥感影像的建筑物三维模型提取方法[D]. 太原:太原理工大学, 2014. HUANG Ming. The three-dimentional model of extraction method of buildings combined LiDAR data and remote sensing images[D]. Taiyuan:Taiyuan University of Technology, 2014.
[15] 穆超, 余洁, 许磊, 等. 基于高分辨率遥感影像的DSM建筑物点的提取研究[J]. 武汉大学学报(信息科学版), 2009, 34(4):414-417. MU Chao, YU Jie, XU Lei, et al. Research on extracting building points from the DSM data combining the high-resolution remote sensing image[J]. Geomatics and Information Science of Wuhan University, 2009, 34(4):414-417.
[16] 高春霞, 谢明鸿, 宋纳. 基于DSM与阴影的彩色遥感影像建筑物提取[J]. 郑州轻工业学院学报(自然科学版), 2015, 30(5-6):108-112. GAO Chunxia, XIE Minghong, SONG Na. Building extraction from color remote sensing imagery based on DSM and shadows[J]. Journal of Zhengzhou University of Light Industry(Natural Science Edition), 2015, 30(5-6):108-112.
[17] ZHONG Chen, XU Qizhi, YANG Feng, et al. Building change detection for high-resolution remotely sensed images based on a semantic dependency[C]//Proceedings of 2015 IEEE International Geoscience and Remote Sensing Symposium. Milan, Italy:IEEE, 2015:3345-3348.
[18] 虞欣, 郑肇葆. 基于语义特征和贝叶斯网络的航空影像建筑物及灌木分类[J]. 测绘学报, 2010, 39(3):245-250. YU Xin, ZHENG Zhaobao. Classification of aerial images' building and shrub based on semantic features and Bayesian networks[J]. Acta Geodaetica et Cartographica Sinica, 2010, 39(3):245-250.
[19] GUO Jiayi, PAN Zongxu, LEI Bin, et al. Automatic color correction for multisource remote sensing images with wasserstein CNN[J]. Remote Sensing, 2017, 9(5):483.
[20] YAO Yuan, JIANG Zhiguo, ZHANG Haopeng, et al. Chimney and condensing tower detection based on faster R-CNN in high resolution remote sensing images[C]//Proceedings of 2017 IEEE International Geoscience and Remote Sensing Symposium. Fort Worth, TX:IEEE, 2017:3329-3332.
[21] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe, Nevada:Curran Associates Inc., 2012:1097-1105.
[22] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv preprintarXiv:1409.1556, 2014.
[23] SZEGEDY C, LIU Wei, JIA Yangqing, et al. Going deeper with convolutions[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA:IEEE, 2015:1-9.
[24] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV:IEEE, 2016:770-778.
[25] DENG Jia, DONG Wei, SOCHER R, et al. ImageNet:a large-scale hierarchical image database[C]//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, FL:IEEE, 2009.
[26] LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO:common objects in context[C]//Proceedings of the 13th European Conference on Computer Vision. Zurich, Switzerland:Springer, 2014:740-755.
[27] LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA:IEEE, 2015:3431-3440.
[28] BADRINARAYANAN V, KENDALL A, CIPOLLA R. SegNet:a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12):2481-2495.
[29] NOH H, HONG S, HAN B. Learning deconvolution network for semantic segmentation[C]//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile:IEEE, 2015:1520-1528.
[30] RONNEBERGER O, FISCHER P, BROX T. U-Net:convolutional networks for biomedical image segmentation[C]//Proceedings of 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich, Germany:Springer, 2015:234-241.
[31] MAGGIORI E, TARABALKA Y, CHARPIAT G, et al. Convolutional neural networks for large-scale remote-sensing image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(2):645-657.
[32] YUAN Jiangye. Learning building extraction in aerial scenes with convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(11):2793-2798.
[33] GIRSHICK R, DONAHUE J, DARRELL T, et al. Region-based convolutional networks for accurate object detection and segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(1):142-158.
[34] GIRSHICK R. Fast R-CNN[C]//Proceedings of 2015 IEEE International Conference on Computer Vision (ICCV).Santiago, Chile:IEEE, 2015.
[35] REN Shaoqing, HE Kaiming, GIRSHICK R, et al. Faster R-CNN:towards real-time object detection with region proposal networks[C]//International Conference on Neural Information Processing Systems. Cambridge, USA:MIT Press, 2015:91-99.
[36] REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once:unified, real-time object detection[J]. arXiv preprint arXiv:1506.02640, 2015:779-788.
[37] HE Kaiming, GKIOXARI G, DOLLÁR P, et al. Mask R-CNN[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018. DOI:10.1109/TPAMI.2018.2844175.
[38] MNIH V. Machine learning for aerial image labeling[D]. Toronto:University of Toronto, 2013.
[39] MAGGIORI E, TARABALKA Y, CHARPIAT G, et al. Can semantic labeling methods generalize to any city? the inria aerial image labeling benchmark[C]//Proceedings of 2017 IEEE International Geoscience and Remote Sensing Symposium. Fort Worth, TX:IEEE, 2017:3226-3229.
[40] WU Guangming, SHAO Xiaowei, GUO Zhiling, et al. Automatic building segmentation of aerial imagery using multi-constraint fully convolutional networks[J]. Remote Sensing, 2018, 10(3):407.