Efficient-communication on-orbit distributed hyperspectral image processing

doi:10.11947/j.AGCS.2024.20230594

Abstract

Abstract:

In recent years, with the increase in the number of on-orbit remote sensing satellites and advancements in hyperspectral imaging technology, there has been a sharp rise in the volume of available hyperspectral data, marking an era of big data applications and data-driven scientific discoveries. However, this substantial and wide-ranging volume of hyperspectral data poses significant challenges for deep learning algorithms to learn and infer on a single node, hindering real-time and efficient intelligent interpretation of information. Therefore, there is a need for comprehensive multi-satellite resource distributed cooperative analysis to address the block effects caused by block processing. However, collaborative processing inherently involves information interaction and transmission, necessitating gradient compression to reduce the transmitted information further, thereby alleviating communication bottlenecks in distributed learning. This paper comprehensively discusses various mainstream efficient communication gradient compression algorithms, specifically focusing on their pros and cons in communication-constrained on-orbit environments, and provides insights into the developmental trends of gradient compression. Through extensive experimental comparisons, we comprehensively evaluate the performance of various gradient compression methods in hyperspectral image processing. These experiments demonstrate the applicability and performance differences of different methods, providing robust references for selecting the most suitable gradient compression methods in practical applications in the future.

Key words: distributed learning, gradient compression, efficient-communication, hyperspectral image, on-orbit processing

CLC Number:

P237

Weiying XIE, Zixuan WANG, Yunsong LI. Efficient-communication on-orbit distributed hyperspectral image processing[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(4): 589-598.

Figures/Tables 8

Fig. 1

Fig. 2

Tab. 1

Tab. 2

Fig. 3

Tab. 3

Tab. 4

Fig. 4

References 45

[1]	VERAVERBEKE S, DENNISON P, GITAS I, et al. Hyperspectral remote sensing of fire: state-of-the-art and future perspectives[J]. Remote Sensing of Environment, 2018, 216:105-121.
[2]	赵文波. “中国高分” 科技重大专项在对地观测发展历程中的阶段研究[J]. 遥感学报, 2019, 23(6):1036-1045.
	ZHAO Wenbo. Phase research and practice of upgrading earth observation from test application to system effectiveness in China[J]. Journal of Remote Sensing, 2019, 23(6):1036-1045.
[3]	HAUT J M, PAOLETTI M E, MORENO-ÁLVAREZ S, et al. Distributed deep learning for remote sensing data interpretation[J]. Proceedings of the IEEE, 2021, 109(8):1320-1349.
[4]	BI Qi, QIN Kun, ZHANG Han, et al. APDC-net: attention pooling-based convolutional network for aerial scene classification[J]. IEEE Geoscience and Remote Sensing Letters, 2020, 17(9):1603-1607.
[5]	ZHU R, YAN L, MO N, et al. Attention-based deep feature fusion for the scene classification of high-resolution remote sensing images[J]. Remote Sensing, 2019, 11(17):1996.
[6]	ZHANG Wei, TANG Ping, ZHAO Lijun. Remote sensing image scene classification using CNN-CapsNet[J]. Remote Sensing, 2019, 11(5):494.
[7]	XU Kejie, HUANG Hong, LI Yuan, et al. Multilayer feature fusion network for scene classification in remote sensing[J]. IEEE Geoscience and Remote Sensing Letters, 2020, 17(11):1894-1898.
[8]	WANG Zeqin, WEN Ming, XU Yuedong, et al. Communication compression techniques in distributed deep learning: a survey[J]. Journal of Systems Architecture, 2023, 142:102927.
[9]	SEIDE F, FU Hao, DROPPO J, et al. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs[C]//Proceedings of the 5th annual conference of the international speech communication association. Singapore: ACM Press, 2014.
[10]	BERNSTEIN J, WANG Y X, AZIZZADENESHELI K, et al. SignSGD: compressed optimisation for non-convex problems[C]//Proceedings of 2018 International Conference on Machine Learning. New York: PMLR, 2018: 560-569.
[11]	WEN Wei, XU Cong, YAN Feng, et al. TernGrad: ternary gradients to reduce communication in distributed deep learning[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach: ACM Press, 2017: 1508-1518.
[12]	ALISTARH D, GRUBIC D, LI J Z, et al. QSGD: communication-efficient SGD via gradient quantization and encoding[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach: ACM Press, 2017: 1707-1718.
[13]	ZHENG S, HUANG Z, KWOK J. Communication-efficient distributed blockwise momentum SGD with error-feedback[C]//Proceedings of 2019 Advances in Neural Information Processing Systems. Vancouver: Curran Associates, Inc, 2019.
[14]	KARIMIREDDY S P, REBJOCK Q, STICH S, et al. Error feedback fixes signsgd and other gradient compression schemes[C]//Proceedings of 2019 International Conference on Machine Learning. California: PMLR, 2019: 3252-3261.
[15]	FAGHRI F, TABRIZIAN I, MARKOV I, et al. Adaptive gradient quantization for data-parallel SGD[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Vancouver: ACM Press, 2020: 3174-3185.
[16]	GUO J, LIU W, WANG W, et al. Accelerating distributed deep learning by adaptive gradient quantization[C]//Proceedings of 2020 IEEE International Conference on Acoustics, Speech and Signal Processing. Barcelona: IEEE, 2020: 1603-1607.
[17]	SONG L, ZHAO K, PAN P, et al. Communication efficient SGD via gradient sampling with Bayes prior[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021.
[18]	LIN Yujun, HAN Song, MAO Huizi, et al. Deep gradient compression: reducing the communication bandwidth for distributed training[C]//Proceedings of 2018 International Conference on Learning Representations. Vancouver: OpenReview.net, 2018: 2458-2467.
[19]	STICH S U, CORDONNIER J B, JAGGI M. Sparsified SGD with memory[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. Montréal: ACM Press, 2018: 4452-4463.
[20]	ALHAM F A, KENNETH H. Sparse communication for distributed gradient descent [C]//Proceedings of 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen: Association for Computational Linguistics, 2017: 440-445.
[21]	CORMEN T H, LEISERSON C E, RIVEST R L, et al. Introduction to algorithms[M]. Cambridge: MIT Press, 2022.
[22]	DUTTA A, BERGOU E H, ABDELMONIEM A M, et al. On the discrepancy between the theoretical analysis and practical implementations of compressed communication for distributed deep learning[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(4):3817-3824.
[23]	WANG Huazheng, KIM S, MCCORD-SNOOK E, et al. Variance reduction in gradient exploration for online learning to rank[C]//Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. Paris: ACM Press, 2019: 835-844.
[24]	BOUNEFFOUF D. Exponentiated gradient exploration foractive learning[J]. Computers, 2016, 5(1):1.
[25]	ABRAHAMYAN L, CHEN Y, BEKOULIS G, et al. Learned gradient compression for distributed deep learning[J]. IEEE Trans Neural Netw Learn Syst, 2022, 33(12):7330-7344.
[26]	TAO Zeyi, XIA Qi, LI Qun, et al. CE-SGD: communication-efficient distributed machine learning[C]//Proceedings of 2021 IEEE Global Communications Conference. Madrid: IEEE, 2021: 1-7.
[27]	IVKIN N, ROTHCHILD D, ULLAH E, et al. Communication-efficient distributed SGD with sketching[J]. Advances in Neural Information Processing Systems, 2019, 32.
[28]	BASU D, DATA D, KARAKUS C, et al. Qsparse-local-SGD: distributed SGD with quantization, sparsification and local computations[J]. Advances in Neural Information Processing Systems, 2019, 32.
[29]	LU Qu, LIU Wantao, HAN Jizhong, et al. Multi-stage gradient compression: overcoming the communication bottleneck in distributed deep learning[C]//Proceedings of 2018 International Conference on Neural Information Processing. Cham: Springer, 2018: 107-119.
[30]	SATTLER F, WIEDEMANN S, MULLER K R, et al. Sparse binary compression: towards distributed deep learning with minimal communication[C]//Proceedings of 2019 International Joint Conference on Neural Networks. Budapest: IEEE, 2019: 1-8.
[31]	HU Kaifan, WU Chengkun, ZHU En. HGC: hybrid gradient compression in distributed deep learning[M]//Advances in Artificial Intelligence and Security. Cham: Springer International Publishing, 2021: 15-27.
[32]	SHAH S M, LAU V K N. Model compression for communication efficient federated learning [J]. IEEE Transactions on Neural Networks and Learning Systems, 2023, 34:5937-5951.
[33]	YAN Guangfeng, LI Tan, HUANG Shaolun, et al. AC-SGD: adaptively compressed SGD for communication-efficient distributed learning[J]. IEEE Journal on Selected Areas in Communications, 2022, 40(9):2678-2693.
[34]	MORDIDO G, VAN KEIRSBILCK M, KELLER A. Monte carlo gradient quantization[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Seattle: IEEE. 2020: 718-719.
[35]	WANG Hongyi, SIEVERT S, CHARLES Z, et al. ATOMO: communication-efficient learning via atomic sparsification[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. Montréal: ACM Press, 2018: 9872-9883.
[36]	YU Mingchao, LIN Zhifeng, NARRA K, et al. GradiVeQ: vector quantization for bandwidth-efficient gradient aggregation in distributed CNN training[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. Montréal: ACM Press, 2018: 5129-5139.
[37]	VOGELS T, KARIMIREDDY S P, JAGGI M. PowerSGD: practical low-rank gradient compression for distributed optimization[J]. Advances in Neural Information Processing Systems, 2019, 32.
[38]	YURTSEVER A, UDELL M, TROPP J, et al. Sketchy decisions: convex low-rank matrix optimization with optimal storage[C]//Proceedings of 2017 Artificial Intelligence and Statistics. Lauderdale: PMLR. 2017.
[39]	GUNASEKAR S, LEE J, SOUDRY D, et al. Characterizing implicit bias in terms of optimization geometry[C]//Proceedings of 2018 International Conference on Machine Learning. New York: PMLR, 2018: 1832-1841.
[40]	LI Daixun, XIE Weiying, LI Yunsong, et al. FedFusion: manifold-driven federated learning for multi-satellite and multi-modality fusion[J]. IEEE Transactions onGeoscience and Remote Sensing, 2024, 62:1-13.
[41]	HONG Danfeng, HAN Zhu, YAO Jing, et al. SpectralFormer: rethinking hyperspectral image classification with transformers[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:3130716.
[42]	DING Kexing, LU Ting, FU Wei, et al. Global-local transformer network for HSI and LiDAR data joint classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:3216319.
[43]	DENBY B, LUCIA B. Orbital edge computing: nanosatellite constellations as a new class of computer system[C]//Proceedings of the 25th International Conference on Architectural Support for Programming Languages and Operating Systems. Lausanne: ACM Press, 2020: 939-954.
[44]	ROY S K, DERIA A, HONG Danfeng, et al.Multimodal fusion transformer for remote sensing image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61:3286826.
[45]	SHEN Dunbin, MA Xiaorui, KONG Wenfeng, et al. Hyperspectral target detection based on interpretable representation network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61:3302950.

方法	OA	AA	Kappa系数	压缩比
SGD	0.812 2	0.884 3	0.786 8	1×
QSGD	0.798 6	0.878 6	0.770 9	8×
signSGD	0.769 7	0.851 2	0.739 4	32×
DGC	0.777 2	0.845 2	0.746 7	32×
DGC	0.708 3	0.820 6	0.670 9	1000×
AC-SGD	0.786 3	0.859 4	0.757 6	32×
AC-SGD	0.751 7	0.843 6	0.718 7	1000×
MCGQ	0.812 3	0.880 1	0.787 2	32×
MCGQ	0.702 2	0.807 7	0.663 8	1000×

方法	OA	AA	Kappa系数	压缩比
SGD	0.933 8	0.935 8	0.928 1	1×
QSGD	0.945 2	0.945 7	0.940 5	8×
signSGD	0.928 5	0.934 0	0.922 4	32×
DGC	0.929 2	0.935 1	0.928 3	32×
DGC	0.927 4	0.936 2	0.921 2	1000×
AC-SGD	0.930 3	0.938 1	0.924 3	32×
AC-SGD	0.907 5	0.915 0	0.899 6	1000×
MCGQ	0.935 1	0.933 9	0.929 5	32×
MCGQ	0.919 7	0.927 7	0.911 6	1000×

方法	OA	AA	Kappa系数	压缩比
SGD	0.904 9	0.603 6	0.862 6	1×
QSGD	0.901 2	0.601 1	0.859 6	8×
signSGD	0.884 6	0.579 6	0.840 1	32×
DGC	0.887 5	0.587 8	0.849 8	32×
DGC	0.876 9	0.572 4	0.834 6	1000×
AC-SGD	0.890 2	0.588 9	0.847 6	32×
AC-SGD	0.868 7	0.554 7	0.817 7	1000×
MCGQ	0.897 5	0.584 6	0.847 8	32×
MCGQ	0.876 7	0.567 8	0.824 1	1000×

方法	AUC	压缩比
SGD	0.998 3	1×
QSGD	0.996 1	8×
signSGD	0.972 7	32×
DGC	0.996 8	32×
DGC	0.991 5	1000×
AC-SGD	0.997 5	32×
AC-SGD	0.9968	1000×
MCGQ	0.991 6	32×
MCGQ	0.990 4	1000×