Robust merging of subblock reconstructions for parallel structure from motion in photogrammetry

doi:10.11947/j.AGCS.2024.20220321

Abstract

Abstract: In this paper, we proposed an improved parallel structure from motion pipeline in photogrammetry by robustifying the merging of subblock reconstructions in a better fashion. Specifically, the whole block represented by a view graph is divided into a number of overlapped subblocks via graph partition and expansion, and an improved incremental SfM is employed to generate the SfM reconstruction of each subblock. To merge these subblock SfM reconstructions in a more robust manner, a subblock graph indicating the overlapping relationship of subblock reconstructions is first built. By considering the geometry consistencies of subblock triplets, gross errors are detected. Then, we leverage the algebraic properties of subblock triplets, which aims to make them more geometrically consistent, to refine the relative transformations between subblock reconstructions. Finally, more accurate relative transformations between subblock reconstructions can be obtained to boost the subsequential merging. Experimental results using UAV images show that the proposed method can guarantee robustness in the subblock reconstruction merging stage. The precision of our SfM results is better than several state-of-the-art parallel SfM methods and the popular COLMAP. Furthermore, it has significant potential for use in photogrammetry and 3D Real Scene reconstruction.

Key words: photogrammetry, 3D real scene, UAV images, parallel structure from motion

CLC Number:

P227

XIAO Teng, WANG Xin, MEI Xi, YE Zhiwei, YAN Qingsong, DENG Fei. Robust merging of subblock reconstructions for parallel structure from motion in photogrammetry[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(2): 332-343.

References

[1] SNAVELY N, SEITZ S M, SZELISKI R. Photo tourism:exploring photo collections in 3D[J]. ACM Transactions on Graphics, 25(3):835-846.
[2] AGARWAL S, FURUKAWA Y, SNAVELY N, et al. Building Rome in a day[J]. Communications of the ACM, 2011, 54(10):105-112.
[3] WU Changchang. Towards linear-time incremental structure from motion[C]//Proceedings of 2013 International Conference on 3D Vision. Seattle:IEEE, 2013:127-134.
[4] SCHONBERGER J L, FRAHM J M. Structure-from-motion revisited[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas:IEEE, 2016:4104-4113.
[5] WANG X, XIAO T, KASTEN Y.A hybrid global structure from motion method for synchronously estimating global rotations and global translations[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 174:35-55.
[6] XIAO Teng, YAN Qingsong, MA Weile, et al. Progressive structure from motion by iteratively prioritizing and refining match pairs[J]. Remote Sensing, 2021, 13(12):2340.
[7] JIANG S, JIANG C, JIANG W. Efficient structure from motion for large-scale UAV images:a review and a comparison of SfM tools[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 167:230-251.
[8] 许彪, 董友强, 张力, 等. 分区优化混合SfM方法[J]. 测绘学报, 2022, 51(1):115-126.DOI:10.11947/j.AGCS.2021.20210105. XU Biao, DONG Youqiang, ZHANG Li, et al. A hybrid SfM method based on partition optimization[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(1):115-126.DOI:10.11947/j.AGCS.2021.20210105.
[9] 李清泉, 邵成立, 万剑华, 等. 优视摄影测量与泛在实景三维数据采集:以实景三维青岛为例[J]. 武汉大学学报(信息科学版), 2022, 47(10):1587-1597. LI Qingquan, SHAO Chengli, WAN Jianhua, et al. Optimized views photogrammetry and ubiquitous real 3D data acquisition with the application case in Qingdao[J]. Geomatics and Information Science of Wuhan University, 2022, 47(10):1587-1597.
[10] LYU Jingguo, YANG Xingbin, ZHANG Danlu, et al. High-resolution remote sensing image semi-global matching method considering geometric constraints of connection points and image texture information [J]. Journal of Geodesy and Geoinformation Science, 2021,4(4):97-112.
[11] BHOWMICK B, PATRA S, CHATTERJEE A, et al.Divide and conquer:a hierarchical approach to large-scale structure-from-motion[J]. Computer Vision and Image Understanding, 2017, 157:190-205.
[12] CHEN Shu, LIANG Luming, OUYANG Jianquan. Accurate structure from motion using consistent cluster merging[J].Multimedia Tools and Applications, 2022, 81(17):24913-24935.
[13] CHEN Yu, SHEN Shuhan, CHEN Yisong, et al. Graph-based parallel large scale structure from motion[J]. Pattern Recognition, 2020, 107:107537.
[14] CUI Hainan, SHEN Shuhan, GAO Xiang, et al. CSFM:Community-based structure from motion[C]//Proceedings of 2017 IEEE International Conference on Image Processing.Beijing:IEEE, 2017:4517-4521.
[15] FANG Meiling, POLLOK T, QU Chengchao. Merge-SfM:merging partial reconstructions[C]//Proceedings of the 30th British Machine Vision Conference. Cardiff:British Machine Vision Association,2019:29.
[16] XIE Xiuchuan, YANG Tao, LI Dongdong, et al. Hierarchical clustering-aligning framework based fast large-scale 3D reconstruction using aerial imagery[J]. Remote Sensing, 2019, 11(3):315.
[17] ZHU S. Accurate, scalable and parallel structure from motion[D].Hong Kong:Hong Kong University of Science and Technology, 2017.
[18] ZHU Siyu, ZHANG Runze, ZHOU Lei, et al. Very large-scale global SfM by distributed motion averaging[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Salt Lake City:IEEE, 2018:4568-4577.
[19] FARENZENA M, FUSIELLO A, GHERARDI R. Structure-and-motion pipeline on a hierarchical cluster tree[C]//Proceedings of 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops. Kyoto:IEEE, 2009:1489-1496.
[20] GHERARDI R, FARENZENA M, FUSIELLO A. Improving the efficiency of hierarchical structure-and-motion[C]//Proceedings of 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.San Francisco:IEEE, 2010:1594-1600.
[21] TOLDO R, GHERARDI R, FARENZENA M, et al. Hierarchical structure-and-motion recovery from uncalibrated images[J]. Computer Vision and Image Understanding, 2015, 140:127-143.
[22] CUI Zhaopeng, JIANG Nianjuan, TANG Chengzhou, et al. Linear global translation estimation with feature tracks[C]//Proceedings of 2015 British Machine Vision Conference. Swansea:British Machine Vision Association, 2015.
[23] SHI Jianbo, MALIK J. Normalized cuts and image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(8):888-905.
[24] KASTEN Y, GEIFMAN A, GALUN M, et al. Algebraic characterization of essential matrices and their averaging in multiview settings[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul:IEEE, 2019:5895-5903.
[25] MOULON P, MONASSE P, PERROT R, et al. OpenMVG:open multiple view geometry[C].Proceedings of 2016 International Workshop on Reproducible Research in Pattern Recognition. Cham:Springer, 2016:60-74.
[26] BOYD S, PARIKH N, CHU E, et al.Distributed optimization and statistical learning via the alternating direction method of multipliers[J]. Foundations and Trends ® in Machine Learning, 2010, 3(1):1-122.
[27] TRIGGS B, MCLAUCHLAN P F, HARTLEY R I, et al.Bundle adjustment-a modern synthesis[C]//Proceedings of 1999 International workshop on vision algorithms. Berlin:Springer,1999:298-372.
[28] YAO Yao, LUO Zixin, LI Shiwei, et al. BlendedMVS:a large-scale dataset for generalized multi-view stereo networks[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle:IEEE, 2020:1790-1799.