SCA-SAM: semantic segmentation for remote sensing images based on scale context attention and efficient SAM transfer

doi:10.11947/j.AGCS.2026.20260063

Abstract

Abstract:

To address the challenges in semantic segmentation for high-resolution remote sensing images, including significant scale variations, dense small objects and thin boundaries that are easily disturbed by complex textures, as well as long-tailed class distributions that hinder rare-class learning, this paper proposed an end-to-end framework, scale context attention segment anything model (SCA-SAM), built upon segment anything model (SAM). The proposed method inserted scale context attention (SCA) modules at multiple stages of the encoder to progressively aggregate multi-scale information, thereby enhancing representations for small objects and complex boundaries. In addition, low-rank adaptation (LoRA) was adopted for parameter-efficient fine-tuning on attention-related projections, where only a small set of task-specific parameters was updated to achieve efficient transfer to remote sensing texture statistics and spatial organization. A composite loss tailored to class imbalance and hard regions was further incorporated to improve discrimination of rare classes and boundary areas. Experiments on the UAVid, ISPRS Vaihingen, and ISPRS Potsdam datasets demonstrated that SCA-SAM achieved consistent improvements in both overall and per-class metrics, while delivering better accuracy and stronger generalization stability with a low parameter overhead.

Key words: remote sensing image semantic segmentation, segment anything model, low-rank adaptation, scale context enhancement

CLC Number:

P237

Xiangyu Zhao, Chunju Zhang, Yifan Pei, Chenxi Li, Jun Zhang, Wei Xu, Chun Lan, Linfeng Lü, Hongbo Liang. SCA-SAM: semantic segmentation for remote sensing images based on scale context attention and efficient SAM transfer[J]. Acta Geodaetica et Cartographica Sinica, 2026, 55(8): 1452-1464.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

URL: http://xb.chinasmp.com/EN/10.11947/j.AGCS.2026.20260063

http://xb.chinasmp.com/EN/Y2026/V55/I8/1452

Figures/Tables 19

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Tab. 1

Tab. 2

Tab. 3

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Tab. 4

Fig. 12

Fig. 13

Tab. 5

Fig. 14

References 32

[1]	陈超, 梁锦涛, 杨刚, 等. 面向土地覆盖精准分类的遥感特征参数优选方法[J]. 测绘学报, 2024, 53(7): 1401-1416. DOI: . doi: 10.11947/j.AGCS.2024.20230327
	Chen Chao, Liang Jintao, Yang Gang, et al. Remote sensing parameters optimization for accurate land cover classifi-cation[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(7): 1401-1416. DOI: . doi: 10.11947/j.AGCS.2024.20230327
[2]	蒋伟杰, 张春菊, 徐兵, 等. AED-Net：滑坡灾害遥感影像语义分割模型[J]. 地球信息科学学报, 2023, 25(10): 2012-2025.
	Jiang Weijie, Zhang Chunju, Xu Bing, et al. AED-Net: semantic segmentation model for landslide recognition from remote sensing images[J]. Journal of Geo-Information Science, 2023, 25(10): 2012-2025.
[3]	Wu Kang, Zhang Yingying, Ru Lixiang, et al. A semantic-enhanced multi-modal remote sensing foundation model for Earth observation[J]. Nature Machine Intelligence, 2025, 7(8): 1235-1249.
[4]	杨明旺, 赵丽科, 叶林峰, 等. 基于卷积神经网络的遥感影像建筑物提取方法综述[J]. 地球信息科学学报, 2024, 26(6): 1500-1516.
	Yang Mingwang, Zhao Like, Ye Linfeng, et al. A review of convolutional neural networks related methods for building extraction from remote sensing images[J]. Journal of Geo-Information Science, 2024, 26(6): 1500-1516.
[5]	Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 3431-3440.
[6]	伍广明, 陈奇, Shibasaki R, 等. 基于U型卷积神经网络的航空影像建筑物检测[J]. 测绘学报, 2018, 47(6): 864-872. DOI: . doi: 10.11947/j.AGCS.2018.20170651
	Wu Guangming, Chen Qi, Shibasaki R, et al. High precision building detection from aerial imagery using a U-Net like convolutional architecture[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6): 864-872. DOI: . doi: 10.11947/j.AGCS.2018.20170651
[7]	许泽宇, 沈占锋, 李杨, 等. 增强型DeepLab算法和自适应损失函数的高分辨率遥感影像分类[J]. 遥感学报, 2022, 26(2): 406-415.
	Xu Zeyu, Shen Zhanfeng, Li Yang, et al. Classification of high-resolution remote sensing images based on enhanced DeepLab algorithm and adaptive loss function[J]. National Remote Sensing Bulletin, 2022, 26(2): 406-415.
[8]	柴华彬, 严超, 邹友峰, 等. 利用PSP Net实现湖北省遥感影像土地覆盖分类[J]. 武汉大学学报(信息科学版), 2021, 46(8): 1224-1232.
	Chai Huabin, Yan Chao, Zou Youfeng, et al. Land cover classification of remote sensing image of Hubei province by using PSP Net[J]. Geomatics and Information Science of Wuhan University, 2021, 46(8): 1224-1232.
[9]	Dosovitskiy A. An image is worth 16x16 words: transformers for image recognition at scale[C]//Proceedings of 2021 International Conference on Learning Representations. San Diego: Open Review.net, 2021.
[10]	Chen Wei, Bruzzone L, Dang Bo, et al. REST: holistic learning for end-to-end semantic segmentation of whole-scene remote sensing imagery[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026, 48(1): 693-710.
[11]	Li Yansheng, Wu Yuning, Cheng Gong, et al. MEET: a million-scale dataset for fine-grained geospatial scene classification with zoom-free remote sensing imagery[J]. IEEE/CAA Journal of Automatica Sinica, 2025, 12(5): 1004-1023.
[12]	Kirillov A, Mintun E, Ravi N, et al. Segment anything[C]//Proceedings of 2023 IEEE/CVF International Conference on Computer Vision (ICCV). Paris: IEEE, 2023: 4015-4026.
[13]	Wang Di, Zhang Jing, Du Bo, et al. SAMRS: scaling-up remote sensing segmentation dataset with segment anything model[C]//Proceedings of 2023 Advances in Neural Information Processing Systems 36. New Orleans: NeurIPS, 2023: 8815-8827.
[14]	Yan Zhiyuan, Li Junxi, Li Xuexue, et al. RingMo-SAM: a foundation model for segment anything in multimodal remote-sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 5625716.
[15]	Liu Zhuoran, Li Zizhen, Liang Ying, et al. RSPS-SAM: a remote sensing image panoptic segmentation method based on SAM[J]. Remote Sensing, 2024, 16(21): 4002.
[16]	Zheng Linghao, Pu Xinyang, Zhang Su, et al. Tuning a SAM-based model with multicognitive visual adapter to remote sensing instance segmentation[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2025, 18: 2737-2748.
[17]	刘思涌, 赵毅力. 微调SAM的遥感图像高效语义分割模型DP-SAM[J]. 中国图象图形学报, 2025, 30(8): 2884-2896.
	Liu Siyong, Zhao Yili. DP-SAM: efficient semantic segmentation of remote sensing images by fine-tuning SAM[J]. Journal of Image and Graphics, 2025, 30(8): 2884-2896.
[18]	Lu Xiaoyan, Weng Qihao. Multi-LoRA fine-tuned segment anything model for urban man-made object extraction[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5637519.
[19]	Shan Zhe, Liu Yang, Zhou Lei, et al. ROS-SAM: high-quality interactive segmentation for remote sensing moving object[C]//Proceedings of 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2025: 3625-3635.
[20]	Hu E J, Shen Yelong, Wallis P, et al. LoRA: low-rank adaptation of large language models[C]//Proceedings of 2021 International Conference on Learning Representations. San Diego: Open Review.net, 2021.
[21]	Lü Ye, Vosselman George, Xia Guisong, et al. UAVid: a semantic segmentation dataset for UAV imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 165: 108-119.
[22]	Gerke M, Rottensteiner F, Wegner J D, et al. ISPRS semantic labeling contest[C]//Proceedings of 2014 Photogrammetric Computer Vision. Zurich: [s.n.], 2014.
[23]	Li Sirui, Peng Linkai, Zhang Zheyuan, et al. TAGS: 3D tumor-adaptive guidance for SAM[PP/OL]. V2. arXiv (2025-08-27) [2026-01-10]. https://doi.org/10.48550/arXiv.2505.17096.
[24]	罗健伟, 张银胜. DMFPNet：增强多尺度目标感知的双路径高分辨率遥感图像分割算法[J]. 地球信息科学学报, 2025, 27(5): 1195-1213.
	Luo Jianwei, Zhang Yinsheng. DMFPNet: dual-path high-resolution remote sensing image segmentation algorithm for enhanced multiscale target detection[J]. Journal of Geo-Information Science, 2025, 27(5): 1195-1213.
[25]	刘帅, 李笑迎, 于梦, 等. 高分辨率遥感图像双解耦语义分割网络模型[J]. 测绘学报, 2023, 52(4): 638-647. DOI: . doi: 10.11947/j.AGCS.2023.20210455
	Liu Shuai, Li Xiaoying, Yu Meng, et al. Dual decoupling semantic segmentation model for high-resolution remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(4): 638-647. DOI: . doi: 10.11947/j.AGCS.2023.20210455
[26]	Wang Libo, Li Rui, Zhang Ce, et al. UNetFormer: a UNet-like transformer for efficient semantic segmentation of remote sensing urban scene imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2022, 190: 196-214.
[27]	Xie Enze, Wang Wenhai, Yu Zhiding, et al. SegFormer: simple and efficient design for semantic segmentation with transformers[C]//Proceedings of the 35th Conference on Neural Information Processing Systems. Red Hook: Curran Associates, 2021: 12077-12090.
[28]	Wang Libo, Li Rui, Duan Chenxi, et al. A novel transformer based semantic segmentation scheme for fine-resolution remote sensing images[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 6506105.
[29]	Wu Honglin, Huang Peng, Zhang Min, et al. CMTFNet: CNN and multiscale transformer fusion network for remote-sensing image semantic segmentation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 2004612.
[30]	Zhang Zhuoyang, Cai Han, Han Song. EfficientViT-SAM: accelerated segment anything model without performance loss[C]//Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Seattle: IEEE, 2024: 7859-7863.
[31]	Chai Shurong, Jain R K, Teng Shiyu, et al. Ladder fine-tuning approach for SAM integrating complementary network[J]. Procedia Computer Science, 2024, 246: 4951-4958.
[32]	Zhang Jie, Li Yunxin, Yang Xubing, et al. RSAM-SEG: a SAM-based model with prior knowledge integration for remote sensing image semantic segmentation[J]. Remote Sensing, 2025, 17(4): 590.

模型	IoU								mIoU	F₁值	OA
模型	建筑	道路	树木	灌丛	移动车辆	静止车辆	人	杂物	mIoU	F₁值	OA
UNetFormer	90.06	78.13	76.01	66.69	68.61	66.64	43.22	62.01	68.92	80.89	86.67
SegFormer	85.20	69.55	64.76	57.02	59.64	52.80	34.66	49.36	59.12	73.33	80.31
DCSwin	90.12	78.10	76.49	66.28	75.32	69.08	45.79	62.86	70.50	82.07	86.98
CMTFNet	91.70	80.74	78.24	69.12	75.72	69.42	43.59	65.48	71.75	82.82	88.27
SAM（Frozen）	88.94	74.06	71.94	56.60	71.12	62.10	40.04	56.03	65.11	78.00	83.84
EfficientViTSAM	90.41	76.47	77.51	68.14	69.01	65.64	37.69	62.73	68.45	80.35	87.10
SAM-LST	88.68	75.17	75.93	66.16	70.18	66.13	45.83	59.71	68.47	80.69	85.85
RSAM-Seg	92.12	78.98	78.11	68.88	72.19	67.90	42.31	65.48	70.75	82.10	88.15
SCA-SAM	92.56	79.80	79.14	71.96	76.62	72.42	49.24	65.86	73.45	84.14	88.83

模型	IoU					mIoU	F₁值	OA
模型	不透水表面	建筑	灌丛	树木	车辆	mIoU	F₁值	OA
UNetFormer	88.43	92.97	74.29	82.06	81.38	83.83	91.07	91.86
SegFormer	88.95	92.90	74.85	82.28	81.01	84.00	91.17	92.04
DCSwin	87.21	90.39	73.92	82.45	80.72	82.94	90.57	91.23
CMTFNet	88.59	93.43	75.05	83.48	81.96	84.50	91.48	92.27
SAM（Frozen）	78.11	82.83	62.58	76.47	61.63	72.32	83.65	85.72
EfficientViTSAM	86.31	91.73	73.55	81.15	73.81	81.31	89.53	91.05
SAM-LST	86.44	91.04	73.96	82.71	76.37	82.10	90.04	91.20
RSAM-Seg	88.37	92.27	74.62	83.01	76.22	82.90	90.50	91.82
SCA-SAM	89.27	93.33	75.54	83.80	81.07	84.60	91.53	92.45

模型	IoU					mIoU	F₁值	OA
模型	不透水表面	建筑	灌丛	树木	车辆	mIoU	F₁值	OA
UNetFormer	84.30	90.73	76.38	79.02	90.77	84.24	91.33	89.54
SegFormer	84.54	90.57	74.47	76.51	88.07	82.83	90.48	89.02
DCSwin	85.17	91.35	76.25	77.41	91.05	84.25	91.32	89.72
CMTFNet	86.84	93.39	77.47	79.03	90.57	85.46	92.04	90.74
SAM（Frozen）	71.17	76.52	61.51	62.46	79.90	70.31	82.34	80.06
EfficientViTSAM	84.78	91.70	75.89	79.02	87.24	83.73	91.04	89.78
SAM-LST	82.59	87.65	74.72	76.90	90.49	82.47	90.27	88.26
RSAM-Seg	84.30	90.36	74.72	77.43	89.00	83.16	90.68	89.03
SCA-SAM	86.92	93.46	77.92	79.11	92.02	85.89	92.28	90.81

试验组	训练策略	可训参数量				吞吐量（张/s）	总参数量	mIoU/（%）
试验组	训练策略	ViT	SCA模块	解码器	总可训	吞吐量（张/s）	总参数量	UAVid	ISPRS Vaihingen	ISPRS Potsdam
SCA-SAM	LoRA	0.15×10⁶	1.48×10⁶	4.34×10⁶	5.97×10⁶	11.89	114.23×10⁶	73.45	84.60	85.89
w/o SCA	LoRA	0.15×10⁶	0×10⁶	4.34×10⁶	4.49×10⁶	13.66	94.16×10⁶	71.99	82.61	84.12
Full Train	全训练	28.39×10⁶	1.48×10⁶	4.34×10⁶	34.22×10⁶	8.99	114.23×10⁶	75.05	84.63	86.34
Full Freeze	全冻结	0.00×10⁶	1.48×10⁶	4.34×10⁶	5.82×10⁶	12.01	114.23×10⁶	29.15	82.01	84.27

损失组合	CB-FocalCE	Focal-Tversky	TopK-CE	IoU								mIoU	F₁值	OA
损失组合	CB-FocalCE	Focal-Tversky	TopK-CE	建筑	道路	树木	灌丛	移动车辆	静止车辆	人	杂物	mIoU	F₁值	OA
1	√	√	√	92.56	79.80	79.14	71.96	76.62	72.42	49.24	65.86	73.45	84.14	88.83
2		√	√	92.14	80.98	78.57	69.58	75.47	70.25	43.77	66.82	72.20	83.13	88.61
3	√		√	92.75	81.92	78.47	69.60	74.55	69.40	42.22	68.11	72.13	83.01	88.89
4	√	√		92.48	80.70	77.78	69.49	73.24	68.77	43.92	66.96	71.67	82.78	88.47