آمار

تعداد نشریات9
تعداد شماره‌ها446
تعداد مقالات5,719
تعداد مشاهده مقاله8,018,458
تعداد دریافت فایل اصل مقاله6,527,290
Adibian, Majid, Ebadzadeh, Mohammad Mahdi. (1404). Double Deep Q Network with Adaptive Prioritized Experience Replay. نشریه مهندسی عمران امیرکبیر, 57(1), 53-62. doi: 10.22060/miscj.2025.23426.5373
Majid Adibian; Mohammad Mahdi Ebadzadeh. "Double Deep Q Network with Adaptive Prioritized Experience Replay". نشریه مهندسی عمران امیرکبیر, 57, 1, 1404, 53-62. doi: 10.22060/miscj.2025.23426.5373
Adibian, Majid, Ebadzadeh, Mohammad Mahdi. (1404). 'Double Deep Q Network with Adaptive Prioritized Experience Replay', نشریه مهندسی عمران امیرکبیر, 57(1), pp. 53-62. doi: 10.22060/miscj.2025.23426.5373
Adibian, Majid, Ebadzadeh, Mohammad Mahdi. Double Deep Q Network with Adaptive Prioritized Experience Replay. نشریه مهندسی عمران امیرکبیر, 1404; 57(1): 53-62. doi: 10.22060/miscj.2025.23426.5373

Double Deep Q Network with Adaptive Prioritized Experience Replay

AUT Journal of Modeling and Simulation
دوره 57، شماره 1، شهریور 2025، صفحه 53-62    اصل مقاله (1.01 M)
نوع مقاله: Research Article
شناسه دیجیتال (DOI): 10.22060/miscj.2025.23426.5373
نویسندگان
Majid Adibian؛ Mohammad Mahdi Ebadzadeh*
Department of Computer Engineering, Amirkabir University of Technology, Tehran, Iran
چکیده
In deep reinforcement learning, experience replay buffers are used to reduce the effects of sequential data and make better use of past experiences. Prioritized Experience Replay (PER) improves upon random sampling by selecting transitions based on their temporal difference (TD) error. However, PER does not consider how important each transition is or how many times it has been used during training. In this paper, we propose a new method for adaptive prioritization that takes into account three additional transition-level factors: reward, usage count (counter), and policy probability—collectively referred to as RCP values. These values are normalized and used alongside the TD error to calculate the probability of selecting each transition from the replay buffer. We evaluate our method on several Atari environments and show that using any of the RCP values individually can improve performance compared to standard PER. To combine all three RCP components, we explore three aggregation functions: minimum, maximum, and mean. Experimental results show that the best aggregation method depends on the environment. However, the mean function generally provides stable improvements across tasks, as it balances all RCP signals and avoids over-relying on any single factor.
کلیدواژه‌ها
Deep Reinforcement Learning؛ Prioritized Experience Replay؛ Deep Q-Network
مراجع
  1. Mnih, V., et al., Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
  2. Lin, L.-J., Self-improving reactive agents based on reinforcement learning, planning, and teaching. Machine learning, 1992. 8: p. 293–321.
  3. Schaul, T., et al., Prioritized Experience Replay. CoRR, 2015. abs/1511.05952.
  4. Atherton, L.A., D. Dupret, and J.R. Mellor, Memory trace replay: the shaping of memory consolidation by neuromodulation. Trends in Neurosciences, 2015. 38(9): p. 560–570.
  5. Ólafsdóttir, H.F., et al., Hippocampal place cells construct reward related sequences through unexplored space. Elife, 2015. 4: p. e06063.
  6. Foster, D.J. and M.A. Wilson, Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature, 2006. 440(7084): p. 680–683.
  7. McNamara, C.G., et al., Dopaminergic neurons promote hippocampal reactivation and spatial memory persistence. Nature neuroscience, 2014. 17(12): p. 1658–1660.
  8. Van Seijen, H. and R. Sutton. Planning by prioritized sweeping with small backups. in International Conference on Machine Learning. 2013. PMLR.
  9. Han, S. and Y. Sung. Diversity actor-critic: Sample-aware entropy regularization for sample-efficient exploration. in International Conference on Machine Learning. 2021. PMLR.
  10. Saglam, B., et al., Actor prioritized experience replay. Journal of Artificial Intelligence Research, 2023. 78: p. 639–672.
  11. Oh, Y., et al. Model-augmented prioritized experience replay. in International Conference on Learning Representations (ICLR). 2021.
  12. Zhang, H., et al., Self-adaptive priority correction for prioritized experience replay. Applied sciences, 2020. 10(19): p. 6925.
  13. Li, A.A., Z. Lu, and C. Miao, Revisiting prioritized experience replay: A value perspective. arXiv preprint arXiv:2102.03261, 2021.
  14. Carrasco-Davis, R.A., et al. Uncertainty Prioritized Experience Replay. in Reinforcement Learning Conference.
  15. Remonda, A., et al., Uncertainty-Based Experience Replay for Task-Agnostic Continual Reinforcement Learning. Transactions on Machine Learning Research, 2025.
  16. Yamani, H., et al., Reward Prediction Error Prioritisation in Experience Replay: The RPE-PER Method. arXiv preprint arXiv:2501.18093, 2025.
  17. Lin, P.-S., et al., Shared-unique Features and Task-aware Prioritized Sampling on Multi-task Reinforcement Learning. arXiv preprint arXiv:2406.00761, 2024.
  18. Soviany, P., et al., Curriculum learning: A survey. International Journal of Computer Vision, 2022. 130(6): p. 1526–1565.
  19. Fedus, W., et al. Revisiting fundamentals of experience replay. in International conference on machine learning. 2020. PMLR.
  20. Ogawa, T., K. Nakagawa, and K. Ikeda. Optimal execution strategy using Deep Q-Network with heuristics policy. in 2024 16th IIAI International Congress on Advanced Applied Informatics (IIAI-AAI). 2024. IEEE.
  21. Zhao, X., et al., A Novel Indicator for Quantifying and Minimizing Information Utility Loss of Robot Teams. IEEE Journal on Selected Areas in Communications, 2025.
آمار
تعداد مشاهده مقاله: 435
تعداد دریافت فایل اصل مقاله: 319


سامانه مدیریت نشریات علمی. طراحی و پیاده سازی از سیناوب