Industrial equipment identification based on multimodal adaptive fusion

SHI Yicong; ZHAO Xiaoli; CHEN Mingxuan; ZHANG Yuyue

doi:10.12299/jsues.24-0133

Volume 39 Issue 3

Sep. 2025

Turn off MathJax

Article Contents

Article Navigation > Journal of Shanghai University of Engineering Science > 2025 > 39(3): 320-325

SHI Yicong, ZHAO Xiaoli, CHEN Mingxuan, ZHANG Yuyue. Industrial equipment identification based on multimodal adaptive fusion[J]. Journal of Shanghai University of Engineering Science, 2025, 39(3): 320-325. doi: 10.12299/jsues.24-0133

Citation:

SHI Yicong, ZHAO Xiaoli, CHEN Mingxuan, ZHANG Yuyue. Industrial equipment identification based on multimodal adaptive fusion[J]. Journal of Shanghai University of Engineering Science, 2025, 39(3): 320-325. doi: 10.12299/jsues.24-0133

Citation:

PDF( 1185 KB)

Industrial equipment identification based on multimodal adaptive fusion

doi: 10.12299/jsues.24-0133

School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai 201620, China

Received Date: 2024-04-28
Available Online: 2025-12-22
Publish Date: 2025-09-30

Abstract

Abstract

Multimodal machine learning, which combines image and text data, is able to improve the accuracy of industrial equipment identification. To address the limitation that existing industrial equipment identification algorithms rely solely on image data and fail to leverage the widely available textual data in industrial settings, a multimodal industrial equipment identification model based on image and text was proposed. This model consists of a novel image-text dataset and a multimodal identification algorithm. The dataset images were collected from real scenarios, and the text was automatically annotated by a multimodal large model. The identification algorithm can adaptively adjust the weight of the modalities, integrate the information of the two modalities, and assist the model in decision-making. Experimental results show that the proposed algorithm can achieve an improvement of 17.62% in identification accuracy compared to single-modal image data, and outperforms other multimodal recognition algorithms by 5.3%, demonstrating its effectiveness.
- multimodal machine learning,
- multimodal fusion,
- industrial equipment identification

FullText(HTML)

References(15)

References

[1]	陶俊鹏, 张玮东, 钟倩文, 等. 基于振动信号图像特征的降噪残差网络轴承故障诊断[J] . 噪声与振动控制, 2024, 44(3): 109 − 116,169. doi: 10.3969/j.issn.1006-1355.2024.03.017
[2]	周子杰, 展金, 李胜铭, 等. 复杂环境下的烧结机篦条故障实时检测方法研究[J/OL] . 机械科学与技术, 2024. DOI: 10.13433/j.cnki.1003-8728.20240076.
[3]	吴大钰, 王岩松, 李燕, 等. 基于声信号的汽车发动机故障诊断方法综述[J] . 渤海大学学报(自然科学版), 2008, 29(3): 264 − 267.
[4]	张阳, 刘瑾. 基于字符增强的工业设备故障命名实体识别[J] . 电子科技, 2024, 37(10): 48 − 54.
[5]	刘传洋, 吴一全. 基于红外图像的电力设备识别及发热故障诊断方法研究进展[J] . 中国电机工程学报, 2025, 45(6): 2171 − 2195.
[6]	徐哲壮, 黄平, 陈丹, 等. 融合机器视觉与邻近度估计的相似工业设备识别策略研究[J] . 仪器仪表学报, 2023, 44(1): 283 − 290.
[7]	王雨滢, 赵庆生, 梁定康. 基于深度学习网络的电气设备图像分类[J] . 科学技术与工程, 2020, 20(23): 9491 − 9496. doi: 10.3969/j.issn.1671-1815.2020.23.035
[8]	YAO N, CHENG K. Electric power equipment image recognition based on deep forest learning model with few samples[J] . Journal of Physics: Conference Series, 2021, 1732: 012025. doi: 10.1088/1742-6596/1732/1/012025
[9]	WANG Y T, LIU H R, WANG D L, et al. Image processing in fault identification for power equipment based on improved super green algorithm[J] . Computers & Electrical Engineering, 2020, 87: 106753.
[10]	LI J N, LI D X, SAVARESE S, et al. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models[C] //Proceedings of the 40th International Conference on Machine Learning. Honolulu: PMLR, 2023: 19730-19742.
[11]	LIU H T, LI C Y, WU Q Y, et al. Visual instruction tuning[C] //Proceedings of the 37th Conference on Neural Information Processing Systems. New Orleans: Curran Associates Inc. , 2023: 1516.
[12]	肖进胜, 饶天宇, 贾茜, 等. 基于图切割的拉普拉斯金字塔图像融合算法[J] . 光电子·激光, 2014, 25(7): 1416 − 1424.
[13]	周永福, 李文龙, 胡冉冉. 多尺度特征融合的双通道SSD行人头部检测算法[J] . 激光与光电子学进展, 2021, 58(24): 375 − 386.
[14]	YANG Z X, ZHU L C, WU Y, et al. Gated channel transformation for visual recognition[C] //Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 11791 − 11800.
[15]	KUMAR G K, NANDAKUMAR K. Hate-CLIPper: multimodal hateful meme classification based on cross-modal interaction of CLIP features[C] //Proceedings of the 2nd Workshop on NLP for Positive Impact. Abu Dhabi: Association for Computational Linguistics, 2022: 171 − 183.