3D scene graph representation and application for intelligent indoor spaces

doi:10.11947/j.AGCS.2024.20230482

Abstract

Abstract:

Existing methods for indoor 3D scene representation focus on object-oriented descriptions, with element representations limited to object-level semantic understanding. These methods lack the ability to express complex relational information within indoor scenes. Addressing the demands of intelligent indoor space tasks, there is a critical need for a structured model that can comprehensively and accurately describe the geometry, semantics, and relationships of indoor elements, while also supporting semantic retrieval and analytical reasoning. Based on the fundamental theory of 3D scene graphs, this paper innovatively proposes a 3D scene graph representation model tailored for intelligent indoor spaces. It systematically introduces the hierarchical organization, geometric representation, semantic description, and relational description methods of indoor 3D scene graphs. A conceptual model is established that uniformly describes the geometry, semantics, and relationships of indoor elements. Additionally, this graph model is compatible with existing 3D scene representation methods, ensuring good data compatibility. Finally, a comprehensive multi-level relational 3D scene graph model is constructed based on the publicly available IFC model. This model's application capabilities, potential, and limitations are systematically explored and analyzed through applications such as complex scene retrieval and topological analysis, in conjunction with large language models. The results demonstrate that the indoor 3D scene graph model possesses complex computation and analysis capabilities, can be directly integrated with large language models, and enables complex scene analysis applications through simple natural language prompts.

Key words: indoor modeling, graph model, IFC, CityGML, large language model

CLC Number:

P208

Shengjun TANG, Siqi DU, Weixi WANG, Renzhong GUO. 3D scene graph representation and application for intelligent indoor spaces[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(7): 1355-1370.

Figures/Tables 21

Fig.1

Tab.8

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Fig.9

Fig.10

Fig.11

Fig.12

Fig.13

References 49

[1]	应申, 朱利平, 李霖, 等. 基于室内空间特征的室内地图表达[J]. 导航定位学报, 2015, 3(4):74-78, 91.
	YING Shen, ZHU Liping, LI Lin, et al. Representation of indoor maps with the analysis of indoor space[J]. Journal of Navigation and Positioning, 2015, 3(4):74-78, 91.
[2]	高伟, 吴毅红, 申抒含, 等. 视觉主导的多传感器融合机器地图构建与定位技术[M]//卫星导航定位与北斗系统应用. 北京: 测绘出版社, 2019: 138-143.
	GAO Wei, WU Yihong, SHEN Shuhan, et al. Vision led multi-sensor fusion machine map construction and positioning technology [M]//Satellite navigation positioning and Beidou system application. Beijing: Surveying and Mapping Press, 2019: 138-143
[3]	徐凯, 胡瑞珍, 杨鑫. 几何引导的主动式三维感知与交互[J]. 图学学报, 2022, 43(6):1049-1056.
	XU Kai, HU Ruizhen, YANG Xin. Geometry-guided active 3D perception and interaction[J]. Journal of Graphics, 2022, 43(6):1049-1056.
[4]	LAAKSO M, KIVINIEMI A. The IFC standard: a review of history, development, and standardization[J]. Electronic Journal of Information Technology in Construction, 2012, 17:134-161.
[5]	GRÖGER G, PLÜMER L. CityGML: interoperable semantic 3D city models[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2012, 71:12-33.
[6]	LI K J, CONTI G, KONSTANTINIDIS E, et al. OGC IndoorGML: a standard approach for indoor maps[M]//Geographical and fingerprinting data to create systems for indoor positioning and indoor/outdoor navigation. Amsterdam: Elsevier, 2019: 187-207.
[7]	朱庆, 张利国, 丁雨淋, 等. 从实景三维建模到数字孪生建模[J]. 测绘学报, 2022, 51(6):1040-1049. DOI: 10.11947/j.AGCS.2022.20210640.
	ZHU Qing, ZHANG Liguo, DING Yulin, et al. From real 3D modeling to digital twin modeling[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(6):1040-1049. DOI: 10.11947/j.AGCS.2022.20210640.
[8]	游天, 周成虎, 陈曦. 室内地图表示方法研究与实践[J]. 测绘科学技术学报, 2014, 31(6):635-640.
	YOU Tian, ZHOU Chenghu, CHEN Xi. The research and practice of indoor map representation[J]. Journal of Geomatics Science and Technology, 2014, 31(6):635-640.
[9]	危双丰, 刘振彬, 赵江洪, 等. SLAM室内三维重建技术综述[J]. 测绘科学, 2018, 43(7):15-26.
	WEI Shuangfeng, LIU Zhenbin, ZHAO Jianghong, et al. A review of indoor 3D reconstruction with SLAM[J]. Science of Surveying and Mapping, 2018, 43(7):15-26.
[10]	崔扬. 移动LiDAR点云室内三维结构化重建方法和关键技术研究[J]. 测绘学报, 2021, 50(7):990. DOI: 10.11947/j.AGCS.2021.20200592.
	CUI Yang. Research on methodology and the key technology of indoor 3D structured reconstruction from mobile LiDAR point clouds[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(7):990. DOI: 10.11947/j.AGCS.2021.20200592.
[11]	付燕平, 严庆安, 廖杰, 等. 基于彩色图像引导的RGB-D相机追踪与三维重建[J]. 武汉大学学报(工学版), 2022, 55(1):92-100.
	FU Yanping, YAN Qing'an, LIAO Jie, et al. RGB-D camera tracking and 3D reconstruction via color image guiding[J]. Engineering Journal of Wuhan University, 2022, 55(1):92-100.
[12]	WALD J, DHAMO H, NAVAB N, et al. Learning 3D semantic scene graphs from 3D indoor reconstructions[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 3961-3970.
[13]	王玮琦, 游雄, 杨剑, 等. 一种改进匹配点对选取策略的ElasticFusion室内三维重建算法[J]. 武汉大学学报(信息科学版), 2020, 45(9):1469-1477.
	WANG Weiqi, YOU Xiong, YANG Jian, et al. Elastic fusion for indoor 3D reconstruction with an improved matching points selection strategy[J]. Geomatics and Information Science of Wuhan University, 2020, 45(9):1469-1477.
[14]	HOU Ji, DAI A, NIEBNER M. 3D-SIS: 3D semantic instance segmentation of RGB-D scans[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 4421-4430.
[15]	SU Hang, MAJI S, KALOGERAKIS E, et al. Multi-view convolutional neural networks for 3D shape recognition[C]//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago: IEEE, 2015: 945-953.
[16]	SONG Shuran, LICHTENBERG S P, XIAO Jianxiong. SUN RGB-D: a RGB-D scene understanding benchmark suite[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 567-576.
[17]	NAJIBI M, LAI Guangda, KUNDU A, et al. DOPS: learning to detect 3D objects and predict their 3D shapes[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 11913-11922.
[18]	熊汉江, 郑先伟, 丁友丽, 等. 基于2D-3D语义传递的室内三维点云模型语义分割[J]. 武汉大学学报(信息科学版), 2018, 43(12):2303-2309.
	XIONG Hanjiang, ZHENG Xianwei, DING Youli, et al. Semantic segmentation of indoor 3D point cloud model based on 2D-3D semantic transfer[J]. Geomatics and Information Science of Wuhan University, 2018, 43(12):2303-2309.
[19]	YI Li, ZHAO Wang, WANG He, et al. GSPN: generative shape proposal network for 3D instance segmentation in point cloud[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 3947-3956.
[20]	LIU Yunze, YI Li, ZHANG Shanghang, et al. P4Contrast: contrastive learning with pairs of point-pixel pairs for RGB-D scene understanding[EB/OL]. [2023-12-01]. http://arxiv.org/pdf/1907.11692.
[21]	CHARLES R Q, HAO Su, MO Kaichun, et al. PointNet: deep learning on point sets for 3D classification and segmentation[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu: IEEE, 2017: 652-660.
[22]	邓林涛, 方志军. 基于特征负反馈卷积的点云分析方法[J]. 激光与光电子学进展, 2022, 59(12):1210006.
	DENG Lintao, FANG Zhijun. Point cloud analysis method based on feature negative feedback convolution[J]. Laser & Optoelectronics Progress, 2022, 59(12):1210006.
[23]	CHEN X, WU H, LICHTI D, et al. Extraction of indoor objects based on the exponential function density clustering model[J]. Information Sciences, 2022, 607:1111-1135.
[24]	NING Xiaojuan, WANG Yinghui, HAO Wen, et al. Structure-based object classification and recognition for 3D scenes in point clouds[C]//Proceedings of 2014 International Conference on Virtual Reality and Visualization. Shenyang: IEEE, 2014: 166-173.
[25]	KIRILLOV A, HE Kaiming, GIRSHICK R, et al. Panoptic segmentation[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 9404-9413.
[26]	REN Dayong, WU Zhengyi, LI Jiawei, et al. Point attention network for point cloud semantic segmentation[J]. Science China Information Sciences, 2022, 65(9):192104.
[27]	CHEN Kai, OUYANG W, LOY C C, et al. Hybrid task cascade for instance segmentation[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recgnition. Long Beach: IEEE, 2019: 4974-4983.
[28]	RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation[C]//Proceedings of the 18th International Conference of Medical Image Computing and Computer. Cham: Springer, 2015: 234-241.
[29]	FANG Hao, LAFARGE F, PAN Cihui, et al. Floorplan generation from 3D point clouds: a space partitioning approach[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 175:44-55.
[30]	LIU Chen, WU Jiajun, KOHLI P, et al. Raster-to-vector: revisiting floorplan transformation[C]//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 2195-2203.
[31]	张蔚, 王腾, 卢政达, 等. 基于CNN的住宅平面图元素识别与布局语义分析[J]. 中国体视学与图像分析, 2020, 25(2):174-182.
	ZHANG Wei, WANG Teng, LU Zhengda, et al. Floor plan recognition and semantic layout analysis based on a convolutional neural network[J]. Chinese Journal of Stereology and Image Analysis, 2020, 25(2):174-182.
[32]	FANG Hao, PAN Cihui, HUANG Hui. Structure-aware indoor scene reconstruction via two levels of abstraction[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 178:155-170.
[33]	HAN Jiali, RONG Mengqi, JIANG Hanqing, et al. Vectorized indoor surface reconstruction from 3D point cloud with multistep 2D optimization[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 177:57-74.
[34]	TRAN H, KHOSHELHAM K. Procedural reconstruction of 3D indoor models from lidar data using reversible jump Markov chain Monte Carlo[J]. Remote Sensing, 2020, 12(5):838.
[35]	HAN Jiali, LIU Yuzhou, RONG Mengqi, et al. FloorUSG: indoor floorplan reconstruction by unifying 2D semantics and 3D geometry[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2023, 196:490-501.
[36]	JAYARAJ P, RAMIYA A M. D citygml building modelling from LiDAR point cloud data[J]. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2018, 425:175-180.
[37]	王敬淳, 汤圣君, 王伟玺, 等. 室内三维点云空间自动划分与规则化方法[J]. 测绘通报, 2022(8):93-97, 138.
	WANG Jingchun, TANG Shengjun, WANG Weixi, et al. Automatic division and regularization of indoor 3D point cloud space[J]. Bulletin of Surveying and Mapping, 2022(8):93-97, 138.
[38]	TANG Shengjun, LI Xiaoming, ZHENG Xianwei, et al. BIM generation from 3D point clouds by combining 3D deep learning and improved morphological approach[J]. Automation in Construction, 2022, 141:104422.
[39]	丁雨淋, 何小波, 朱庆, 等. 实时威胁态势感知的室内火灾疏散路径动态优化方法[J]. 测绘学报, 2016, 45(12):1464-1475. DOI: 10.11947/j.AGCS.2016.20160053.
	DING Yulin, HE Xiaobo, ZHU Qing, et al. A dynamic optimization method of indoor fire evacuation route based on realtime situation awareness[J]. Acta Geodaetica et Cartographica Sinica, 2016, 45(12):1464-1475. DOI: 10.11947/j.AGCS.2016.20160053.
[40]	邓中亮, 王小恒. 一种用于大型建筑火灾中的应急疏散算法[J]. 软件, 2011, 32(2):112-114.
	DENG Zhongliang, WANG Xiaoheng. An emergency evacuation algorithm in large building fire[J]. Computer Engineering & Software, 2011, 32(2):112-114.
[41]	朱庆, 胡明远, 许伟平, 等. 面向火灾动态疏散的三维建筑信息模型[J]. 武汉大学学报(信息科学版), 2014, 39(7):762-766, 872.
	ZHU Qing, HU Mingyuan, XU Weiping, et al. 3D building information model for facilitating dynamic analysis of indoor fire emergency[J]. Geomatics and Information Science of Wuhan University, 2014, 39(7):762-766, 872.
[42]	韩李涛, 郭欢, 张海思. 一种多出口室内应急疏散路径规划算法[J]. 测绘科学, 2018, 43(12):105-110.
	HAN Litao, GUO Huan, ZHANG Haisi. An algorithm for route planning applied in multi-exit indoor emergency evacuation[J]. Science of Surveying and Mapping, 2018, 43(12):105-110.
[43]	李清泉, 周宝定, 马威, 等. GIS辅助的室内定位技术研究进展[J]. 测绘学报, 2019, 48(12):1498-1506. DOI: 10.11947/j.AGCS.2019.20190455.
	LI Qingquan, ZHOU Baoding, MA Wei, et al. Research process of GIS-aided indoor localization[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(12):1498-1506. DOI: 10.11947/j.AGCS.2019.20190455.
[44]	刘涛, 张星, 李清泉, 等. 顾及地标可视性的室内导航路径优化算法[J]. 武汉大学学报(信息科学版), 2017, 42(1):43-48.
	LIU Tao, ZHANG Xing, LI Qingquan, et al. An indoor pedestrian route planning algorithm based on landmark visibility[J]. Geomatics and Information Science of Wuhan University, 2017, 42(1):43-48.
[45]	陈锐志, 陈亮. 基于智能手机的室内定位技术的发展现状和挑战[J]. 测绘学报, 2017, 46(10):1316-1326. DOI: 10.11947/j.AGCS.2017.20170383.
	CHEN Ruizhi, CHEN Liang. Indoor positioning with smartphones: the state-of-the-art and the challenges[J]. Acta Geodaetica et Cartographica Sinica, 2017, 46(10):1316-1326. DOI: 10.11947/j.AGCS.2017.20170383.
[46]	WALD J, NAVAB N, TOMBARI F. Learning 3D semantic scene graphs with instance embeddings[J]. International Journal of Computer Vision, 2022, 130(3):630-651.
[47]	ZHU G, ZHANG L, JIANG Y, et al. Scene graph generation: a comprehensive survey[EB/OL]. [2024-01-25]. http://arxiv.org/abs/2201.00443.
[48]	ZLATANOVA S. On 3D topological relationships[C]//Proceedings of 2000 International Workshop on Database & Expert Systems Applications. London: IEEE, 2000.
[49]	IFC Wiki. KIT IFC Examples[EB/OL]. [2024-02-15]. https://www.ifcwiki.org/index.php?title=KIT_IFC_Examples.

级别	描述
复杂建筑要素	用于描述一个建筑群，一座建筑可以跨越几个相连或不相连的建筑，组成一个建筑群，它是由一个以上的独立建筑组成的。
单体建筑要素	用于描述一个单体建筑，一个单体建筑至少包含一个楼层要素，建筑中不同楼层可能具有特定用途或所有权。
楼层要素	用于描述单个楼层空间，每个楼层由各种室内空间要素和室内实体要素组成。
功能空间要素	用于描述具有室内空间功能属性的结构信息，可以包含房间、阳台、走廊等功能空间。
连接空间要素	用于描述连接不同楼层的要素信息，可以包含楼梯、电梯等。
构造实体要素	用于描述建筑的固有组成部分的要素信息，包含墙、楼板、梁、柱、门、窗等要素。
附属实体要素	用于描述附属于建筑物内特定空间的要素信息，包含家具、设施、传感器等要素。

室内要素节点类型	三维点云	不规则三角网	构造实体几何	扫描体	包围盒
复杂建筑要素	√	√	√	√	√
单体建筑要素	√	√	√	√	√
楼层要素	√	√	√	√	√
功能空间要素	√		√	√	√
连接空间要素	√		√	√	√
构造实体要素	√		√	√	√
附属实体要素	√	√			√

序号	要素基本信息类型	定义
1	描述	对该要素的基本情况进行文字描述
2	类别	描述该要素的类别
3	权属	描述该要素的权属关系

序号	复杂建筑要素语义类型	定义
1	基本信息	一个建筑群的基本信息，包括名称和所有权
2	地址	一个建筑群的地址
3	单体建筑数量	组成一个建筑群的单个建筑物的数量

序号	单体建筑要素语义类型	定义
1	基本信息	一个建筑群的基本信息，包括名称和所有权
2	地址	一个建筑群的地址
3	地点	组成一个建筑群的单个建筑物的数量
4	使用方法	组成复杂的各个建筑物的信息
5	楼层数	建筑物内的层数
6	完成日期	建筑物建成的日期
7	访问权限	进入建筑物的限制