Fatigue detection framework based on intelligent fusion of multimodal features
-
摘要: 为充分利用驾驶员疲劳姿势变化,提出一种基于多模态特征智能融合的疲劳检测方法,通过特征提取模型获取身体和面部数据的中间层特征。为突出身体关键区域的变化,采用人体姿态(BlazePose)估计提取身体关键点并生成高斯热图,进一步将身体中间层特征图与关键点热图融合。针对上半身疲劳姿势变化的动态幅度特性,沿时间维度分块计算自注意力,以降低模型的计算成本。在特征层面将面部与姿态特征图沿通道拼接,借助多头自注意力捕捉跨模态全局依赖关系,实现多模态特征的智能融合。在包含面部与上半身双视角、多种光照条件的数据集上进行验证,结果表明:与仅依赖面部或者上身姿势的单特征检测方法相比,提出方法的准确率分别提升3.4%和11.9%,达到96.24%,整体网络模型的推理速度提高30%以上。Abstract: To fully exploit variations in driver fatigue-related posture, an intelligent multimodal feature fusion method was proposed for fatigue detection. Intermediate representations of body and facial data were extracted by feature extraction model. To emphasize key upper-body regions, BlazePose was applied to detect 17 body keypoints and generate corresponding Gaussian heatmaps, which were further fused with the body feature maps. Self-attention was computed within fixed temporal blocks to effectively capture dynamic amplitude variations of fatigue postures while reducing computational overhead. Finally, facial and body feature maps were concatenated along the channel dimension and processed using multi-head self-attention to model cross-modal global dependencies, thereby achieving deep multimodal fusion. Experiments were conducted on a dual-view (face and upper body), multi-illumination driving dataset, the results demonstrate that compared with single-modality approaches relying only on facial data or upper-body posture, the proposed method improves accuracy by 3.4% and 11.9%, respectively, achieving 96.24%, and the inference speed of the overall network model is increased by more than 30%.
-
Key words:
- fatigue posture /
- body keypoints /
- self-attention /
- inference speed
-
表 1 模型推理时间分析
Table 1. Analysis of model inference time
网络 权重/MB 时间/ms 全注意力计算 (仅身体) 88.3 352 (11.73 ms/帧) 全注意力计算 (仅面部) 90.8 368 (14.26 ms/帧) 分块注意力计算 (身体 + 面部) 65.5 229 (7.63 ms/帧) -
[1] Saurav S, Mathur S, Sang I, et al. Yawn detection for driver’s drowsiness prediction using bi-directional LSTM with CNN features[C] //11th International Conference on Intelligent Human Computer Interaction. Allahabad: Springer, 2020: 189 − 200. [2] 徐敬一, 田瑾, 刘翔, 等. 基于注意力机制改进的疲劳驾驶检测方法[J] . 传感器与微系统, 2024, 43(4): 115 − 118. doi: 10.13873/J.1000-9787(2024)04-0115-04 [3] Ansari S, Du H P, Naghdy F, et al. Automatic driver cognitive fatigue detection based on upper body posture variations[J] . Expert Systems with Applications, 2022, 203: 117568. doi: 10.1016/j.eswa.2022.117568 [4] Takahashi K, Sunagawa M, Shikii S I, et al. Skeleton features-based drowsiness detection for a semi-autonomous driving system in real road conditions[C] //2022 IEEE 11th Global Conference on Consumer Electronics (GCCE). Osaka: IEEE, 2022: 649 − 650. [5] Arnab A, Dehghani M, Heigold G, et al. ViViT: a video vision transformer[PP/OL] . (2021-11-01)[2024-08-25] . https://doi.org/10.48550/arXiv.2103.15691. [6] Chen W Y, Wang A R. Enhanced facial expression recognition based on facial action unit intensity and region[C] //2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC). Honolulu: IEEE, 2023: 1939 − 1944. [7] Bazarevsky V, Grishchenko I, Raveendran K, et al. BlazePose: on-device real-time body pose tracking[EB/OL] . (2020-06-17)[2024-12-26] . https://arxiv.org/abs/2006.10204. [8] Child R, Gray S, Radford A, et al. Generating long sequences with sparse transformers[PP/OL] . (2019-04-23)[2024-12-26] . https://arxiv.org/abs/1904.10509. [9] Ortega J D, Kose N, Cañas P, et al. DMD: a large-scale multi-modal driver monitoring dataset for attention and alertness analysis[C] //16th European Conference on Computer Vision. Glasgow: Springer, 2020: 387 − 405. -
下载: