Close-range photogrammetric differential rendering using geometry buffer for the material reconstruction of non-lambertian surface model

doi:10.11947/j.AGCS.2025.20240257

Abstract

Abstract:

The optical texture material of non-lambertian artifacts such as porcelain is a critical foundation for the realistic visualization and rendering of 3D models. Due to the color differences such as highlights caused by non-lambertian surfaces from different viewing angles, existing photogrammetric texture reconstruction methods crudely alleviate these color differences through color equalization and feathering; Neural radiance field methods based on implicit spatial representation rely solely on multi-layer perceptron to blur spatial fitting for different reflection directions, leading to insufficient realism. Therefore, this study proposes a differentiable rendering method for non-lambertian material reconstruction using close-range photogrammetry geometry images. By employing a handheld stereo scanner and controlled lighting conditions, the method accurately acquires oriented parameters, model geometry, and initial color texture of images. Material properties such as albedo, roughness, and normal perturbation are treated as planar tensors to be optimized, mapped to screen space via rasterization to obtain rich geometric images with coordinates, normal, texture coordinates, and material attributes; Drawing on deferred shading techniques, the geometric image simulates pixel shaders in screen space, achieving differentiable rendering and inverse feedback optimization of unknown material information using the Cook-Torrance microfacet model for highlights. Experimental results on typical non-lambertian targets demonstrate that the proposed method achieves a structural similarity index (SSIM) better than 0.85 compared to real images, improving rendering accuracy by 8.7% and 7.2% over 3D Gaussian points and multi-resolution hash encoding based on implicit space representation, respectively, significantly enhancing the realism of the model.

Key words: close-range photogrammetry, non-lambertian material, geometry image, deferred shading, differentiable r endering

CLC Number:

P234.1

Han HU, Xiaolin GUO, Lang XIONG, Xuming GE, Haowei ZENG, Qing ZHU. Close-range photogrammetric differential rendering using geometry buffer for the material reconstruction of non-lambertian surface model[J]. Acta Geodaetica et Cartographica Sinica, 2025, 54(4): 736-748.

Figures/Tables 15

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Tab. 1

Fig. 9

Fig. 10

Fig. 11

Tab. 2

Fig. 12

Fig. 13

References 40

[1]	WAECHTER M, MOEHRLE N, GOESELE M. Let there be color! large-scale texturing of 3D reconstructions[C]//Proceedings of 2014 European Conference on Computer Vision. Cham: Springer International Publishing, 2014: 836-850.
[2]	ZHOU Qianyi, KOLTUN V. Color map optimization for 3D reconstruction with consumer depth cameras[J]. ACM Transactions on Graphics, 2014, 33(4): 1-10.
[3]	肖腾, 王鑫, 梅熙, 等. 摄影测量局部场景稳健合并的并行式运动恢复结构方法[J]. 测绘学报, 2024, 53(2): 332-343. DOI:. doi: 10.11947/j.AGCS.2024.20220321
	XIAO Teng, WANG Xin, MEI Xi, et al. Robust merging of subblock reconstructions for parallel structure from motion in photogrammetry[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(2): 332-343. DOI:. doi: 10.11947/j.AGCS.2024.20220321
[4]	姜三, 马一尘, 李清泉, 等. 无序无人机影像的并行化SfM三维重建方法[J]. 测绘学报, 2024, 53(5): 946-958. DOI:. doi: 10.11947/j.AGCS.2024.20230335
	JIANG San, MA Yichen, LI Qingquan, et al. Parallel SfM-based 3D reconstruction for unordered UAV images[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(5): 946-958. DOI:. doi: 10.11947/j.AGCS.2024.20230335
[5]	付琳, 洪海波, 王晰, 等. 基于逆向反射模型的非朗伯光度立体视觉[J]. 光学学报, 2020, 40(5): 151-161.
	FU Lin, HONG Haibo, WANG Xi, et al. Non-Lambertian photometric stereo vision based on inverse reflectance model[J]. Acta Optica Sinica, 2020, 40(5): 151-161.
[6]	MILDENHALL B, SRINIVASAN P P, TANCIK M, et al. NeRF: representing scenes as neural radiance fields for view synthesis[J]. Communications of the ACM, 2022, 65(1): 99-106.
[7]	MÜLLER T, EVANS A, SCHIED C, et al. Instant neural graphics primitives with a multiresolution hash encoding[J]. ACM Transactions on Graphics, 2022, 41(4): 1-15.
[8]	FRIDOVICH-KEIL S, YU A, TANCIK M, et al. Plenoxels: radiance fields without neural networks[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 5501-5510.
[9]	XU Qiangeng, XU Zexiang, PHILIP J, et al. Point-NeRF: point-based neural radiance fields[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 5438-5448.
[10]	KERBL B, KOPANAS G, LEIMKUEHLER T, et al. 3D Gaussian splatting for real-time radiance field rendering[J]. ACM Transactions on Graphics, 2023, 42(4): 1-14.
[11]	AZINOVIC D, MARTIN-BRUALLA R, GOLDMAN D B, et al. Neural RGB-D surface reconstruction[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 6290-6301.
[12]	WANG Peng, LIU Lingjie, LIU Yuan, et al. NeuS: learning neural implicit surfaces by volume rendering for multi-view reconstruction[EB/OL]. [2024-04-20]. https://arxiv.org/abs/2106.10689v3.
[13]	YANG Bangbang, BAO Chong, ZENG Junyi, et al. NeuMesh: learning disentangled neural mesh-based implicit field for Geometry and Texture editing[C]//Proceedings of 2022 European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 597-614.
[14]	WIZADWONGSA S, PHONGTHAWEE P, YENPHRAPHAI J, et al. NeX: real-time view synthesis with neural basis expansion[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 8534-8543.
[15]	MILLER B, GEORGIEV I, JAROSZ W. A null-scattering path integral formulation of light transport[J]. ACM Transactions on Graphics, 2019, 38(4): 1-13.
[16]	HART J C. Sphere tracing: a geometric method for the antialiased ray tracing of implicit surfaces[J]. The Visual Computer, 1996, 12(10): 527-545.
[17]	TEWARI A, THIES J, MILDENHALL B, et al. Advances in neural rendering[J]. Computer Graphics Forum, 2022, 41(2): 703-735.
[18]	TUCKER R, SNAVELY N. Single-view view synthesis with multiplane images[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 551-560.
[19]	MÜLLER T, ROUSSELLE F, NOVÁK J, et al. Real-time neural radiance caching for path tracing[J]. ACM Transactions on Graphics, 2021, 40(4): 1-16.
[20]	CHEN Anpei, XU Zexiang, GEIGER A, et al. TensoRF: tensorial radiance fields[C]//Proceedings of 2022 European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 333-350.
[21]	许威威, 周漾, 吴鸿智, 等. 可微绘制技术研究进展[J]. 中国图象图形学报, 2021, 26(6): 1521-1535.
	XU Weiwei, ZHOU Yang, WU Hongzhi, et al. Differential rendering: a survey[J]. Journal of Image and Graphics, 2021, 26(6): 1521-1535.
[22]	HU Han, DING Yulin, ZHU Qing, et al. Stable least-squares matching for oblique images using bound constrained optimization and a robust loss function[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016, 118: 53-67.
[23]	ENGEL J, SCHÖPS T, CREMERS D. LSD-SLAM: large-scale direct monocular SLAM[C]//Proceedings of 2014 European Conference on Computer Vision. Cham: Springer International Publishing, 2014: 834-849.
[24]	VU H H, LABATUT P, PONS J P, et al. High accuracy and visibility-consistent dense multiview stereo[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(5): 889-901.
[25]	FURUKAWA Y, PONCE J. Dense 3D motion capture from synchronized video streams[M]//RONFARD R, TAUBIN G, ed. Image and geometry processing for 3D cinematography. Berlin: Springer, 2010: 193-211.
[26]	GOURAUD H. Continuous shading of curved surfaces[M]//WOLFE R, ed. Seminal graphics: pioneering efforts that shaped the field. New York: Association for Computing Machinery, 1998: 87-93.
[27]	ZHU Qing, WANG Zhendong, HU Han, et al. Leveraging photogrammetric mesh models for aerial-ground feature point matching toward integrated 3D reconstruction[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 166: 26-40.
[28]	KARIS B. Real shading in unreal Engine 4[J]. SIGGRAPH, 2013, 4(3): 1-21.
[29]	LAINE S, HELLSTEN J, KARRAS T, et al. Modular primitives for high-performance differentiable rendering[J]. ACM Transactions on Graphics, 2020, 39(6): 1-14.
[30]	HARGREAVES S, HARRIS M. Deferred shading[C]//Proceedings of 2004 Game Developers Conference. San Jose: [s. n.]. 2004, 2: 31.
[31]	中观自动化科技有限公司. AtlaScan多模式多功能量测3D扫描仪[EB/OL]. [2024-04-20]. https://www.zg-3d.com/AtlaScan.html.
	ZG Technology Co., Ltd.. AtlaScan multimodal multi-functional measurement 3D scanner[EB/OL]. [2024-04-20]. https://www.zg-3d.com/AtlaScan.html.
[32]	CHENG H K, SCHWING A G. XMem: long-term video object segmentation with an Atkinson-Shiffrin memory model[C]//Proceedings of 2022 European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 640-658.
[33]	AGISOFT . Agisoft metashape[EB/OL]. [2024-04-20]. https://www.agisoft.com/.
[34]	KAJIYA J T. The rendering equation[C]//Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM Press, 1986: 143-150.
[35]	WALTER B, MARSCHNER S R, LI H, et al. Microfacet models for refraction through rough surfaces[C]//Proceedings of the 18th Eurographics Conference on Rendering Techniques. Goslar: Eurographics Association, 2007: 195-206.
[36]	WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
[37]	ZHANG R, ISOLA P, EFROS A A, et al. The unreasonable effectiveness of deep features as a perceptual metric[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 586-595.
[38]	BOSS M, JAMPANI V, BRAUN R, et al. Neural-pil: neural pre-integrated lighting for reflectance decomposition[J]. Advances in Neural Information Processing Systems, 2021, 34: 10691-10704.
[39]	VERBIN D, HEDMAN P, MILDENHALL B, et al. Ref-NeRF: structured view-dependent appearance for neural radiance fields[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 5481-5490.
[40]	朱庆, 张利国, 丁雨淋, 等. 从实景三维建模到数字孪生建模[J]. 测绘学报, 2022, 51(6): 1040-1049. DOI:. doi: 10.11947/j.AGCS.2022.20210640
	ZHU Qing, ZHANG Liguo, DING Yulin, et al. From real 3D modeling to digital twin modeling[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(6): 1040-1049. DOI:. doi: 10.11947/j.AGCS.2022.20210640

名称	模型尺寸（高×宽）/cm	网格模型数量（顶点/面片）	空三精度/像素
花瓶	10×6	6.06×10⁵/12.11×10⁵	1.44
碗	7×13	11.55×10⁵/23.10×10⁵	1.11
锥形花盆	12.5×9	23.69×10⁵/47.29×10⁵	0.52
罐	12×10	22.54×10⁵/44.91×10⁵	0.58
桶形花盆	14×15	45.79×10⁵/91.57×10⁵	0.77

方法	PSNR↑	SSIM↑	LPIPS↓
摄影测量方法	14.61	0.81	0.13
3DGS	17.03	0.80	0.11
InstantNGP	18.26	0.79	0.15
Neural-PIL	17.67	0.61	0.22
Ref-NeRF	21.93	0.81	0.28
本文方法	24.19	0.89	0.09
摄影测量方法	15.44	0.89	0.08
3DGS	24.49	0.88	0.10
InstantNGP	22.78	0.92	0.14
Neural-PIL	18.69	0.56	0.33
Ref-NeRF	21.75	0.82	0.31
本文方法	27.56	0.91	0.09
摄影测量方法	20.24	0.85	0.14
3DGS	29.01	0.86	0.14
InstantNGP	21.99	0.87	0.24
Neural-PIL	17.64	0.55	0.26
Ref-NeRF	23.48	0.84	0.28
本文方法	33.23	0.95	0.04
摄影测量方法	17.62	0.86	0.13
3DGS	29.78	0.88	0.11
InstantNGP	21.92	0.88	0.28
Neural-PIL	19.06	0.60	0.20
Ref-NeRF	26.37	0.88	0.26
本文方法	28.93	0.93	0.08
摄影测量方法	19.43	0.85	0.12
3DGS	25.56	0.82	0.13
InstantNGP	23.04	0.84	0.31
Neural-PIL	18.85	0.47	0.36
Ref-NeRF	24.77	0.82	0.33
本文方法	30.82	0.93	0.11