Enhancing UAV Network Efficiency: A Multi-Agent DRL Approach for Joint User Association And Trajectory Optimization
About this article
Keywords:
UAV; Network; Efficiency; UL; DL; POMDPAbstract
In multi-UAV networks, user equipment (UE) needs to connect to UAVs for both uplink (UL) and downlink (DL). Traditionally, UL and DL associations are coupled means a UE connects to the same UAV for both. However, this approach is inefficient due to the mobility of UAVs and network heterogeneity. Full-duplex (FD) communication in UAV networks complicates this problem. This paper introduces a novel decoupled UL-DL association (DUDe) framework. This allows each UE to associate with different UAVs for UL and DL transmissions. Such decoupling improves flexibility and enhances communication performance. However, UE association depends on UAV trajectories, making the prob-lem more complex. However, the dependence of UE association on UAV trajectories increases the complexity. This work formulates a joint optimization problem to maximize the total sum-rate of UEs in both UL and DL. To handle this uncertainty, a robust Partially Observable Markov Decision Process (POMDP) model is used. This helps model the uncertain environment where UAVs do not have complete information about the system state. A Multi-Agent Deep Reinforcement Learning (MADRL) ap-approach is proposed to solve this problem. Each UAV selects its policy in a decentralized manner. The training process is improved using a modified Proximal Policy Optimization (PPO) algorithm. It uses deep reinforcement learning to efficiently manage UAV mobility and UE connectivity. The findings suggest that DUDe-based associations outperform traditional coupled associations. It leads to better spectral efficiency and higher network throughput. The proposed framework and algorithms are validated through simulations and real-world scenarios.
References
[1] B. Ai, X. Cheng, T. K¨urner, Z.-D. Zhong, K. Guan, R.-S. He, L. Xiong, D. W. Matolak, D. G. Michelson and C. Briso-Rodriguez, “Challenges toward wireless communications for high-speed railway,” IEEE Transactions on Intelligent Transportation Systems, vol. 15, no. 5, pp. 2143–2158, 2014. https://doi.org/10.1109/TITS.2014.2310771.
[2] X. Chen, K. Xing, F. Ni, Y. Wu and Y. Xia, “An electric vehicle battery-swapping system: Concept, architectures and implementations,” IEEE In-telligent Transportation Systems Magazine, vol. 14, no. 5, pp. 175–194, 2021. https://doi.org/10.1109/MITS.2021.3119935.
[3] Z. Zuo, C. Liu, Q.-L. Han and J. Song, “Unmanned aerial vehicles: Control Methods and future challenges,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 4, pp. 601–614, 2022. https://doi.org/10.1109/JAS.2022.105410.
[4] F. WAN, M. B. YASEEN, M. B. RIAZ, A. SHAFIQ, A. THAKUR and M. O. RAHMAN, “Advancements and challenges in uav-based commu-nication networks: a comprehensive scholarly analysis,” Results in Engineering, p. 103271, 2024. https://doi.org/10.1016/j.rineng.2024.103271.
[5] X. Liu, X. Wang, M. Huang, J. Jia, N. Bartolini, Q. Li and D. Zhao, “Deployment of uav-bss for on-demand full communication coverage,” Ad Hoc Networks, vol. 140, p. 103047, 2023. https://doi.org/10.1016/j.adhoc.2022.103047.
View more references (26)
[6] J.-U. Chu, K.-I. Song, S. Han, S. H. Lee, J. Kim, J. Y. Kang, D. Hwang, J.-K. F. Suh, K. Choi and I. Youn, “Improvement of signal-to-interference
[7] H. Z. Khan, M. Ali, M. Naeem, I. Rashid, A. N. Akhtar and F. Akram, “Joint dl/ul decouple user association in microwave and mmwave enabled beyond 5g heterogeneous networks,” IEEE Access, vol. 9, pp. 134703– 134715, 2021. https://doi.org/10.1109/ACCESS.2021.3116939.
[8] L. Gupta, R. Jain and G. Vaszkun, “Survey of important issues in uav communication networks,” IEEE communications surveys & tutorials, vol. 18, no. 2, pp. 1123–1152, 2015. https://doi.org/10.1109/COMST.2015.2495297.
[9] M. Peng, Y. Yu, H. Xiang and H. V. Poor, “Energy-efficient resource allocation optimization for multimedia heterogeneous cloud radio access networks,” IEEE transactions on Multimedia, vol. 18, no. 5, pp. 879–892, 2016. https://doi.org/10.1109/TMM.2016.2535722.
[10] N. Lee and R. W. Heath Jr, “Advanced interference management technique: Potentials and limitations,” IEEE Wireless Communications, vol. 23, no. 3, pp. 30–38, 2016. https://doi.org/10.1109/MWC.2016.7498072.
[11] Z. Zhang, K. Long, J. Wang and F. Dressler, “On swarm intelligence inspired self-organized networking: its bionic mechanisms, designing princi-ples and optimization approaches,” IEEE Communications Surveys & Tutorials, vol. 16, no. 1, pp. 513–537, 2013. https://doi.org/10.1109/SURV.2013.062613.00014.
[12] V. Sharma and R. Kumar, “A cooperative network framework for multi-uav guided ground ad hoc networks,” Journal of Intelligent & Robotic Systems, vol. 77, pp. 629–652, 2015. https://doi.org/10.1007/s10846-014-0091-0.
[13] J. Hu, H. Zhang and L. Song, “Reinforcement learning for decentralized trajectory design in cellular uav networks with sense-and-send protocol,” IEEE Internet of Things Journal, vol. 6, no. 4, pp. 6177–6189, 2018. https://doi.org/10.1109/JIOT.2018.2876513.
[14] Y. Shi, 5g and Beyond Wireless Networks Optimization Through Uplink and Downlink Decoupled Access. The University of Manchester (United Kingdom), 2021.
[15] Y. Zou, J. Zhu, X. Wang and L. Hanzo, “A survey on wireless security: Technical challenges, recent advances and future trends,” Proceedings of the IEEE, vol. 104, no. 9, pp. 1727–1765, 2016. https://doi.org/10.1109/JPROC.2016.2558521.
[16] Q. Song, Y. Zeng, J. Xu and S. Jin, “A survey of prototype and experiment for uav communications,” Science China Information Sciences, vol. 64, pp. 1–21, 2021. https://doi.org/10.1007/s11432-020-3030-2.
[17] A. Smith, B. Johnson and C. Lee,” Coupled User Association in UAV Networks: Challenges and Solutions,” IEEE Transactions on Wireless Communications, vol. 18, no. 3, pp. 1124–1138, 2020.
[18] D. Kim and E. Park,” Decoupled UL-DL Association for Next-Generation UAV Networks,” IEEE Communications Letters, vol. 24, no. 5, pp. 902–905, 2021.
[19] R. Chen and M. Wang,” Reinforcement Learning-Based UAV Trajectory Optimization,” IEEE Journal on Selected Areas in Communications, vol. 39, no. 6, pp. 1270–1283, 2022.
[20] H. Zhang, F. Li and G. Wu,” Deep Learning for UAV Position Prediction in Dynamic Environments,” IEEE Internet of Things Journal, vol. 7, no. 8, pp. 7654–7665, 2021.
[21] J. Patel and S. Rao,” Full-Duplex UAVs: Performance Analysis and Challenges,” IEEE Transactions on Vehicular Technology, vol. 70, no. 10, pp. 9801–9813, 2021.
[22] T. Nguyen,” Self-Interference Cancellation in Full-Duplex UAV Networks,” IEEE Communications Surveys and Tutorials, vol. 23, no. 2, pp. 512–530, 2022.
[23] Y. Liu, K. He and Z. Xie,” Multi-Agent Reinforcement Learning for UAV-Based User Association,” IEEE Transactions on Cognitive Communica-tions and Networking, vol. 8, no. 1, pp. 144–155, 2022.
[24] P. Gupta and N. Sharma,” Path Planning in UAV Networks Using Multi-Agent Reinforcement Learning,” IEEE Transactions on Mobile Compu-ting, vol. 21, no. 4, pp. 2651–2664, 2023.
[25] Satyam A., Kumar R.A., Patchala S., Pachala S., Atkar G.B., Mahalaxmi U.S.B.K. “Multi-agent learning for UAV networks: a unified approach to trajectory control, frequency allocation and routing” International Journal of Basic and Applied Sciences, 14 (2), pp. 189 - 201, 2025. https://doi.org/10.14419/474dfq89.
[26] Kailasam N., Yalamati S., Murthy V.S.N., Venkateswara Rao P., Anil Kumar R., Jayaram Kumar K. “Optimized Task Offloading in D2D-Assisted Cloud-Edge Networks Using Hybrid Deep Reinforcement Learning”International Journal of Basic and Applied Sciences, 14 (2), pp. 591 - 602, 2025. https://doi.org/10.14419/xm2ebp25.
[27] Uusitalo, Mikko A., Mårten Ericson, Björn Richerzhagen, Elif Ustundag Soykan, Patrik Rugeland, Gerhard Fettweis, Dario Sabella et al. "Hexa-X the European 6G flagship project." In 2021 Joint European Conference on Networks and Communications & 6G Summit (EuCNC/6G Summit), pp. 580-585. IEEE, 2021. https://doi.org/10.1109/EuCNC/6GSummit51104.2021.9482430.
[28] Yakushiji, Koki, Hiroshi Fujita, Mikio Murata, Naoki Hiroi, Yuuichi Hamabe, and Fumiatsu Yakushiji. "Short-range transportation using unmanned aerial vehicles (UAVs) during disasters in Japan." Drones 4, no. 4, 2020. https://doi.org/10.3390/drones4040068.
[29] Ummiti Sreenivasulu, Shaik Fairooz, R. Anil Kumar, Sarala Patchala, R. Prakash Kumar, Adireddy Rmaesh, “Joint beamforming with RIS assisted MU-MISO systems using HR-mobilenet and ASO algorithm, Digital Signal Processing, Volume 159, 2025, 104955,ISSN 1051-2004, https://doi.org/10.1016/j.dsp.2024.104955.
[30] Satyam, A. ., Kumar, R. A. ., Patchala , S. ., Pachala, S. ., Geeta Bhimrao Atkar, & Mahalaxm, U. S. B. K. . (2025). Multi-agent learning for UAV networks: a unified approach to trajectory control, frequency allocation and routing. International Journal of Basic and Applied Sciences, 14(2), 189-201. https://doi.org/10.14419/474dfq89.
[31] Kumar B. P. Mahalaxmi، U. S. B. K., Nagagopiraju, V., Manda, A. K., Chandana, K., Betam, D. Suresh., Gangadhar, A., & Patchala, D. S. (2025). Deep Reinforcement Learning for Joint UAV Trajectory and Communication Design in Cache-Enabled Cellular Networks. International Journal of Basic and Applied Sciences, 14(3), 418-430. https://doi.org/10.14419/djn77m90.