Multi-agent Q-learning with particle filtering for UAV tracking in Open-RAN environment.

Soleymani, Seyed Ahmad; Goudarzi, Shidrokh; Xiao, Pei; Mihaylova, Lyudmila; Shojafar, Mohammad; Wang, Wenwu

Multi-agent Q-learning with particle filtering for UAV tracking in Open-RAN environment.

Lists

Soleymani, Seyed Ahmad, Goudarzi, Shidrokh ORCID: https://orcid.org/0000-0003-0383-3553, Xiao, Pei, Mihaylova, Lyudmila, Shojafar, Mohammad and Wang, Wenwu (2025) Multi-agent Q-learning with particle filtering for UAV tracking in Open-RAN environment. IEEE Transactions on Aerospace and Electronic Systems. pp. 1-21. ISSN 00189251

Preview

PDF (PDF/A)
Multi_agent_Q_learning_with_Particle_Filtering_for_UAV_Tracking_in_Open_Radio_Access_Networks___IEEE_TAES__accessible.pdf - Accepted Version
Available under License Creative Commons Attribution.
Download (1MB) | Preview

Official URL: https://ieeexplore.ieee.org/abstract/document/1096...

Abstract

This paper introduces a method for target tracking that leverages mobile sensor nodes and Unmanned Aerial Vehicles (UAVs) within an Open- Radio Access Network (RAN) framework. Open-RAN is a flexible and standardized architecture that allows open and interoperable components in RANs, promoting efficiency and adaptability. The core methodology involves improving the accuracy and energy consumption tracking in urban areas filled with obstacles and dynamic conditions. Mobile sensor nodes use a particle filtering algorithm to detect and estimate target positions, and this information is relayed to nearby Evolved/Next Generation Node Bs (e/gNBs), which function as the radio access network infrastructure. The e/gNBs manage clusters of UAVs using a specialized xApp integrated into the near-real-time RAN Intelligent Controller (RIC). The UAVs utilize a comprehensive tracking strategy based on received signal strength (RSS), a trilateration algorithm, and an enhanced multi agent Q-learning algorithm (eMAQL). This approach enables UAVs to optimize their flight paths while balancing accuracy, power usage, and communication delays The experimental results show that the system achieves optimal performance with eight discrete actions for eMAQL, with UAVs consuming an average of 90 (watts) of power and maintaining a root mean square error (RMSE) of less than 0.5 (meters) for target position estimation. These results highlight the system's effectiveness in providing precise and energy-efficient tracking in complex urban environments.

Item Type:	Article
Identifier:	10.1109/TAES.2025.3559518
Additional Information:	© 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Keywords:	Accuracy, Delay, Open-RAN, Particle Filtering, Power Consumption, Q-Learning, RSS, Target Tracking, UAV
Subjects:	Construction and engineering > Electrical and electronic engineering
Date Deposited:	08 May 2025
URI:	https://repository.uwl.ac.uk/id/eprint/13516

Downloads

Downloads per month over past year

Actions (login required)

View Item

References

X. Chen, Z. Feng, Z. Wei, F. Gao, and X. Yuan, “Performance of joint sensing-communication cooperative sensing UAV network,” IEEE Transactions on Vehicular Technology, vol. 69, no. 12, pp. 15 545–15 556, 2020.

H. Oh, S. Kim, H.-s. Shin, and A. Tsourdos, “Coordinated standoff tracking of moving target groups using multiple UAVs,” IEEE Transactions on Aerospace and Electronic Systems, vol. 51, no. 2, pp. 1501–1514, 2015.

J. Moon, S. Papaioannou, C. Laoudias, P. Kolios, and S. Kim,“Deep reinforcement learning multi-UAV trajectory control for target tracking,” IEEE Internet of Things Journal, vol. 8, no. 20, pp. 15 441–15 455, 2021.

A. Guerra, D. Dardari, and P. M. Djuric, “Dynamic radar net works of UAVs: A tutorial overview and tracking performance comparison with terrestrial radar networks,” IEEE Vehicular Technology Magazine, vol. 15, no. 2, pp. 113–120, 2020.

D. Huo, L. Dai, R. Chai, R. Xue, and Y. Xia, “Collision-free model predictive trajectory tracking control for UAVs in obstacle environment,” IEEE Transactions on Aerospace and Electronic Systems, vol. 59, no. 3, pp. 2920–2932, 2022.

X. Jiang, N. Li, Y. Guo, D. Yu, and S. Yang, “Localization of multiple rf sources based on bayesian compressive sensing using a limited number of UAVs with airborne RSS sensor,” IEEE Sensors Journal, vol. 21, no. 5, pp. 7067–7079, 2020.

S. A. H. Mohsan, N. Q. H. Othman, Y. Li, M. H. Alsharif, and M. A. Khan, “Unmanned aerial vehicles (UAVs): Practical as pects, applications, open challenges, security issues, and future trends,” Intelligent Service Robotics, vol. 16, no. 1, pp. 109–137, 2023.

K. Meng, Q. Wu, J. Xu, W. Chen, Z. Feng, R. Schober, and A. L. Swindlehurst, “UAV-enabled integrated sensing and communication: Opportunities and challenges,” IEEE Wireless Communications, 2023.

D. Yuan, X. Chang, Z. Li, and Z. He, “Learning adaptive spatial temporal context-aware correlation filters for UAV tracking,” ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), vol. 18, no. 3, pp. 1–18, 2022.

M. Park, S. An, J. Seo, and H. Oh, “Autonomous source search for UAVs using gaussian mixture model-based infotaxis: Algorithm and flight experiments,” IEEE Transactions on Aerospace and Electronic Systems, vol. 57, no. 6, pp. 4238–4254, 2021.

J. Cui, Y. Liu, and A. Nallanathan, “Multi-agent reinforcement learning-based resource allocation for UAV networks,” IEEE Transactions on Wireless Communications, vol. 19, no. 2, pp. 729–743, 2019.

J. Yu, X. Liu, Y. Gao, and X. Shen, “3D channel tracking for UAV-satellite communications in space-air-ground integrated
networks,” IEEE Journal on Selected Areas in Communications,vol. 38, no. 12, pp. 2810–2823, 2020.

Y. Lun, H. Wang, J. Wu, Y. Liu, and Y. Wang, “Target search in dynamic environments with multiple solar-powered UAVs,” IEEE Transactions on Vehicular Technology, vol. 71, no. 9, pp. 9309–9321, 2022.

H. Wu, H. Li, R. Xiao, and J. Liu, “Modeling and simulation of dynamic ant colony’s labor division for task allocation of UAV swarm,” Physica A: Statistical Mechanics and its Applications,vol. 491, pp. 127–141, 2018.

Y.-J. Chen, D.-K. Chang, and C. Zhang, “Autonomous tracking using a swarm of UAVs: A constrained multi-agent reinforcement learning approach,” IEEE Transactions on Vehicular Technology, vol. 69, no. 11, pp. 13 702–13 717, 2020.

S. Li, X. Yang, X. Wang, D. Zeng, H. Ye, and Q. Zhao, “Learning target-aware vision transformers for real-time UAV tracking,” IEEE Transactions on Geoscience and Remote Sensing, 2024.

S. A. Soleymani, M. Shojafar, C. H. Foh, S. Goudarzi, and W. Wang, “Secure target-tracking by UAVs in O-RAN environ ment,” in 2024 IFIP Networking Conference (IFIP Networking). IEEE, 2024, pp. 204–212.

H. Sallouha, M. M. Azari, A. Chiumento, and S. Pollin, “Aerial an chors positioning for reliable RSS-based outdoor localization in urban environments,” IEEE Wireless Communications Letters, vol. 7, no. 3, pp. 376–379, 2017.

X. Li, “RSS-based location estimation with unknown pathloss model,” IEEE Transactions on Wireless Communications, vol. 5, no. 12, pp. 3626–3633, 2006.

S. Papaioannou, S. Kim, C. Laoudias, P. Kolios, S. Kim, T. Theocharides, C. Panayiotou, and M. Polycarpou, “Coordinated CRLB-based control for tracking multiple first responders in 3D environments,” in Proceedings of the International Conference on Unmanned Aircraft Systems (ICUAS). IEEE, 2020,
pp. 1475–1484.

N. Zhou, D. Meng, and S. Lu, “Estimation of the dynamic states of synchronous machines using an extended particle filter,” IEEE Transactions on Power Systems, vol. 28, no. 4, pp. 4152–4161, 2013.

S. S. Dias and M. G. Bruno, “Cooperative target tracking using decentralized particle filtering and RSS sensors,” IEEE Trans actions on Signal Processing, vol. 61, no. 14, pp. 3632–3646, 2013.

J. Fan, Z. Wang, Y. Xie, and Z. Yang, “A theoretical analysis of deep Q-learning,” in Learning for dynamics and control. PMLR, 2020, pp. 486–489.

R. S. Sutton and A. G. Barto, “Reinforcement learning: An introduction,” Robotica, vol. 17, no. 2, pp. 229–235, 1999.

T. Zhang, Y. Xu, J. Loo, D. Yang, and L. Xiao, “Joint computa tion and communication design for UAV-assisted mobile edge computing in iot,” IEEE Transactions on Industrial Informatics, vol. 16, no. 8, pp. 5505–5516, 2019.

X. Gu, G. Zhang, M. Wang, W. Duan, M. Wen, and P.-H. Ho, “UAV-aided energy-efficient edge computing networks: Secu rity offloading optimization,” IEEE Internet of Things Journal, vol. 9, no. 6, pp. 4245–4258, 2021.

H. V. Abeywickrama, B. A. Jayawickrama, Y. He, and E. Dutkiewicz, “Empirical power consumption model for UAVs,” in 2018 IEEE 88th Vehicular Technology Conference (VTC-Fall). IEEE, 2018, pp. 1–5.

H. Ren and M. Q.-H. Meng, “Power adaptive localization algo rithm for wireless sensor networks using particle filter,” IEEE Transactions on Vehicular Technology, vol. 58, no. 5, pp. 2498–2508, 2008.

B. Ristic, S. Arulampalam, and N. Gordon, Beyond the Kalman filter: Particle filters for tracking applications. Artech house, 2003.

S. Goudarzi, S. A. Soleymani, W. Wang, and P. Xiao, “UAV enabled mobile edge computing for resource allocation using cooperative evolutionary computation,” IEEE Transactions on Aerospace and Electronic Systems, 2023.

Y. Lai, Y. L. Che, S. Luo, and K. Wu, “Optimal wireless information and energy transmissions for UAV-enabled cognitive communication systems,” in 2018 IEEE International Conference on
Communication Systems (ICCS). IEEE, 2018, pp. 168–172.

X. Liu, Y. Liu, Y. Chen, and L. Hanzo, “Trajectory design and power control for multi-UAV assisted wireless networks: A machine learning approach,” IEEE Transactions on Vehicular Technology, vol. 68, no. 8, pp. 7957–7969, 2019.

H. Sun, B. Zhang, X. Zhang, Y. Yu, K. Sha, and W. Shi, “FlexEdge: Dynamic task scheduling for a UAV-based on-demand mobile edge server,” IEEE Internet of Things Journal, vol. 9, no. 17, pp. 15 983–16 005, 2022.

G. An, Z. Wu, Z. Shen, K. Shang, and H. Ishibuchi, “Evolutionary multi-objective deep reinforcement learning for autonomous UAV navigation in large-scale complex environments,” in Proceedings of the Genetic and Evolutionary Computation Conference, 2023, pp. 633–641.

J. Zhang, Z. Shi, A. Zhang, Q. Yang, G. Shi, and Y. Wu, “UAV trajectory prediction based on flight state recognition,” IEEE Transactions on Aerospace and Electronic Systems, vol. 60, no. 3, pp. 2629–2641, 2023.

L. P. Kaelbling, M. L. Littman, and A. R. Cassandra, “Planning and acting in partially observable stochastic domains,” Artificial intelligence, vol. 101, no. 1-2, pp. 99–134, 1998.

Y. Xue and W. Chen, “A UAV navigation approach based on deep reinforcement learning in large cluttered 3D environments,” IEEE Transactions on Vehicular Technology, vol. 72, no. 3, pp. 3001–3014, 2022.

Tools

CORE (COnnecting REpositories)

The University of West London

Multi-agent Q-learning with particle filtering for UAV tracking in Open-RAN environment.

Abstract

Downloads

Actions (login required)

Menu