Moxibustion temperature control combining behavior cloning and reinforcement learning
-
摘要: 针对传统自动控制算法在艾灸机器人温度控制中有效治疗时间短、稳态温度波动大,以及行为克隆算法在复杂温度变化场景下适应性差等问题,提出一种基于行为克隆与强化学习的艾灸温度控制策略。利用实际环境中收集的艾灸数据对LSTM-KAN时序预测模型进行训练,预测下一时刻的艾灸温度并构建仿真环境。引入对决网络和噪声网络对传统双深度Q网络(double deep Q-network, DDQN)算法进行改进,利用行为克隆初始化策略网络,再结合强化学习进一步优化策略网络。在仿真与实际环境中对比验证不同算法的性能表现。实验结果表明,该控制策略在有效性与稳定性方面具有显著优势。Abstract: To address the problems of short effective treatment time and large steady-state temperature fluctuation in traditional automatic control algorithms for temperature control of moxibustion robots, as well as the poor adaptability of behavior cloning algorithms in complex temperature change scenarios, a moxibustion temperature control strategy based on behavior cloning and reinforcement learning was proposed. First, moxibustion data collected in real environments were used to train the LSTM-KAN temporal prediction model, so as to predict the moxibustion temperature at the next moment and build a simulation environment. Second, a dueling network and a noisy network were introduced to improve the traditional double deep Q-network (DDQN) algorithm; the policy network was initialized by behavior cloning and then further optimized by reinforcement learning. Finally, the performance of different algorithms was compared and verified in both simulated and real environments. Experimental results show that the proposed control strategy has significant advantages in effectiveness and stability.
-
Key words:
- moxibustion robot /
- behavioral cloning /
- reinforcement learning /
- dueling network /
- noisy network
-
表 1 2种预测模型性能对比
Table 1. Comparison of the performance between two prediction models
模型 训练集RMSE 测试集RMSE 训练集R2 测试集R2 LSTM-FCN 0.25 0.30 0.91 0.91 LSTM-KAN 0.23 0.27 0.93 0.93 表 2 算法主要参数设置
Table 2. Main parameter settings of algorithm
参数 数值 状态维度s_dim 4 动作维度a_dim 8 经验池大小memory_size 40000 迭代总回合M 500 单回合总步长step 180 网络权重更新N 40 批次大小batch_size 64 奖励折扣因子γ 1 表 3 4种算法的训练结果
Table 3. Training results of four algorithms
算法 收敛回合 平均奖励值 DDQN 407 1607 Dueling Net-DDQN 288 1617 Noisy Net-DDQN 399 1616 Dueling-Noisy-DDQN 338 1641 表 4 3种算法的实验结果
Table 4. Experimental results of three algorithms
算法 稳态时
间/s稳态温
度/℃稳态温度
误差/℃超调量/℃ 自动控制 635 43.40 0.21 0.70 行为克隆 720 43.20 0.11 0.40 本研究算法 770 43.21 0.07 0.29 -
[1] 张玉杰, 刘峰辰, 贾梦真, 等. 艾灸治疗疲劳性亚健康的研究进展[J] . 中国中医药现代远程教育, 2021, 19(19): 199 − 202. [2] 姚勇, 毛雷, 邢嘉祺, 等. 智能艾灸手法模拟系统的设计与实现[J] . 北京生物医学工程, 2020, 39(6): 588 − 593, 649. [3] 谢丁一, 陈日新, 王绥卓, 等. 一种热敏灸机器人及其控制方法: 118178202A[P] . 2024-06-14. [4] 袁浩. 共享艾灸理疗系统的设计与实现[D] . 太原: 中北大学, 2019. [5] 张博, 黄山, 张浛芮, 等. 基于强化学习的艾灸机器人温度控制策略研究[J] . 电子测量技术, 2022, 45(24): 60 − 66. doi: 10.19651/j.cnki.emt.2210117 [6] Torabi F, Warnell G, Stone P. Behavioral cloning from observation[C] //Proceedings of the 27th International Joint Conference on Artificial Intelligence. Stockholm: IJCAI, 2018: 4950−4957. [7] Zhu Y F, Stone P, Zhu Y K. Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation[J] . IEEE Robotics and Automation Letters, 2022, 7(2): 4126 − 4133. [8] Fang H J, Fang H S, Wang Y M, et al. AirExo: low-cost exoskeletons for learning whole-arm manipulation in the wild[C/OL] //2024 IEEE International Conference on Robotics and Automation (ICRA). Yokohama: IEEE, 2024: 15031−15038. https://ieeexplore.ieee.org/document/10610799. DOI: 10.1109/ICRA57147.2024.10610799. [9] Chi C, Xu Z J, Pan C E, et al. Universal manipulation interface: in-the-wild robot teaching without in-the-wild robots[PP/OL] . V3. (2024-03-06)[2025-01-12] . https://doi.org/10.48550/arXiv.2402.10329. [10] Shakya A K, Pillai G, Chakrabarty S. Reinforcement learning algorithms: a brief survey[J] . Expert Systems with Applications, 2023, 231: 120495. [11] Van Hasselt H, Guez A, Silver D. Deep reinforcement learning with double q-learning[C] //Proceedings of the 30th AAAI Conference on Artificial Intelligence. Phoenix: AAAI, 2016: 2094 − 2100. [12] Liu Z M, Wang Y X, Vaidya S, et al. KAN: Kolmogorov-Arnold networks[C] //The Thirteenth International Conference on Learning Representations. Singapore: OpenReview. net, 2025: 1 − 47. [13] Plappert M, Andrychowicz M, Ray A, et al. Multi-goal reinforcement learning: challenging robotics environments and request for research[PP/OL] . V2. (2018-02-26)[2025-01-09] . https://doi.org/10.48550/arXiv.1802.09464. [14] 黑鑫鑫, 张彩凡, 王丹, 等. 近5年艾灸作用影响因素的研究进展[J] . 针灸临床杂志, 2016, 32(12): 82 − 84. [15] 周攀, 张建斌, 王玲玲, 等. 不同灸温的艾灸抗炎效应及TRPV1作用机制研究[J] . 中国中医基础医学杂志, 2015, 21(9): 1143 − 1145. doi: 10.19945/j.cnki.issn.1006-3250.2015.09.037 [16] Wang Z Y, Schaul T, Hessel M, et al. Dueling network architectures for deep reinforcement learning[C/OL] //Proceedings of the 33nd International Conference on Machine Learning. New York City: PMLR 48, 2016: 1995−2003. http://proceedings.mlr.press/v48/wangf16.pdf. [17] Peng B Y, Sun Q, Li S E, et al. End-to-end autonomous driving through dueling double deep Q-network[J] . Automotive Innovation, 2021, 4(3): 328 − 337. [18] Plappert M, Houthooft R, Dhariwal P, et al. Parameter space noise for exploration[C] //Proceedings of the 6th International Conference on Learning Representations. Vancouver: OpenReview. net, 2018: 1 − 18. -
下载: