Future-Proof Physical Layer Security for 6G Multicast Systems: Relay Selection, Antenna Diversity, and Cooperative Transmission Design

Authors

  • Enliang Li 1,* https://orcid.org/0000-0003-2956-2077
    • Formal Analysis
    • Data Curation
    • Conceptualization
    • Investigation
    • Project Administration
    • Writing – Original Draft Preparation
    • Writing – Review & Editing
    • Visualization
    • Supervision
    • Methodology
    • Resources
  • Ziyu Wang 2 https://orcid.org/0009-0004-9939-1810
    • Conceptualization
    • Methodology
    • Validation
    • Writing – Review & Editing
    • Formal Analysis
1 The Grainger College of Engineering Electrical & Computer Engineering, University of Illinois Urbana-Champaign, Urbana, US 61801
2 School of Business, Wake Forest University, Winston-Salem, NC 27109
Corresponding author: eli9@illinois.edu

DOI:

https://doi.org/10.63646/ZTWC8649

Keywords:

6G; physical layer security; multicast systems; relay selection; antenna diversity; cooperative transmission; eavesdropping; secure outage probability

Abstract

Future 6G multicast systems will carry immersive media, emergency coordination traffic, satellite-assisted services, cooperative sensing streams, and software-defined industrial control messages across heterogeneous radio environments. These services require not only high throughput but also secure transmission to many legitimate users under passive eavesdropping, mobility, blockage, and imperfect channel-state information. This paper develops a future-proof physical layer security framework for 6G multicast systems by integrating partial relay selection, antenna diversity, and cooperative transmission design. Unlike conventional multicast-security studies that focus on Rayleigh fading with fixed relay pools, the proposed framework interprets relay selection as a risk-aware and service-aware decision problem. The analysis is motivated by recent PRSF-based secure multicasting research but extends the discussion toward 6G deployment conditions such as cell-free access, reconfigurable surfaces, non-terrestrial links, and AI-assisted control. A compact analytical model is combined with a reproducible numerical experiment to study the probability of non-zero secrecy multicast capacity and secure outage probability. The results indicate that relay diversity and destination-antenna diversity jointly improve secrecy performance, while larger multicast groups, stronger eavesdropper populations, and higher target secrecy rates create measurable reliability penalties. The paper contributes a practical design roadmap that links physical layer security metrics with 6G architecture requirements, thereby offering a balanced approach to secure, scalable, and energy-aware multicast transmission.

How to Cite

Li, E., & Wang, Z. (2025). Future-Proof Physical Layer Security for 6G Multicast Systems: Relay Selection, Antenna Diversity, and Cooperative Transmission Design. Crossroads of Future Technologies, 1(1), 65-77. https://doi.org/10.63646/ZTWC8649

References

Badrudduza, A. S. M., Sarkar, M. Z. I., & Kundu, M. K. (2020). Enhancing security in multicasting through correlated Nakagami-m fading channels with opportunistic relaying. Physical Communication, 43, 101177. https://doi.org/10.1016/j.phycom.2020.101177

Basar, E., Di Renzo, M., De Rosny, J., Debbah, M., Alouini, M.-S., & Zhang, R. (2019). Wireless Communications Through Reconfigurable Intelligent Surfaces. IEEE Access, 7, 116753-116773. https://doi.org/10.1109/access.2019.2935192

Björnson, E., Larsson, E. G., & Marzetta, T. L. (2016). Massive MIMO: ten myths and one critical question. IEEE Communications Magazine, 54(2), 114-123. https://doi.org/10.1109/mcom.2016.7402270

Bjornson, E., & Sanguinetti, L. (2020). Scalable Cell-Free Massive MIMO Systems. IEEE Transactions on Communications, 68(7), 4247-4261. https://doi.org/10.1109/tcomm.2020.2987311

Bletsas, A., Khisti, A., Reed, D. P., & Lippman, A. (2006). A simple Cooperative diversity method based on network path selection. IEEE Journal on Selected Areas in Communications, 24(3), 659-672. https://doi.org/10.1109/jsac.2005.862417

Chaccour, C., Soorki, M. N., Saad, W., Bennis, M., Popovski, P., & Debbah, M. (2022). Seven Defining Features of Terahertz (THz) Wireless Systems: A Fellowship of Communication and Sensing. IEEE Communications Surveys & Tutorials, 24(2), 967-993. https://doi.org/10.1109/comst.2022.3143454

Chen, M., Challita, U., Saad, W., Yin, C., & Debbah, M. (2019). Artificial Neural Networks-Based Machine Learning for Wireless Networks: A Tutorial. IEEE Communications Surveys & Tutorials, 21(4), 3039-3071. https://doi.org/10.1109/comst.2019.2926625

Chen, X., Ng, D. W. K., Gerstacker, W. H., & Chen, H.-H. (2017). A Survey on Multiple-Antenna Techniques for Physical Layer Security. IEEE Communications Surveys & Tutorials, 19(2), 1027-1053. https://doi.org/10.1109/comst.2016.2633387

Csiszar, I., & Korner, J. (1978). Broadcast channels with confidential messages. IEEE Transactions on Information Theory, 24(3), 339-348. https://doi.org/10.1109/tit.1978.1055892

Dai, L., Wang, B., Yuan, Y., Han, S., Chih-lin, I., & Wang, Z. (2015). Non-orthogonal multiple access for 5G: solutions, challenges, opportunities, and future research trends. IEEE Communications Magazine, 53(9), 74-81. https://doi.org/10.1109/mcom.2015.7263349

Dang, S., Amin, O., Shihada, B., & Alouini, M.-S. (2020). What should 6G be?. Nature Electronics, 3(1), 20-29. https://doi.org/10.1038/s41928-019-0355-6

Deng, H., Wang, H.-M., Guo, W., & Wang, W. (2015). Secrecy Transmission With a Helper: To Relay or to Jam. IEEE Transactions on Information Forensics and Security, 10(2), 293-307. https://doi.org/10.1109/tifs.2014.2374356

Di Renzo, M., Zappone, A., Debbah, M., Alouini, M.-S., Yuen, C., de Rosny, J., & Tretyakov, S. (2020). Smart Radio Environments Empowered by Reconfigurable Intelligent Surfaces: How It Works, State of Research, and The Road Ahead. IEEE Journal on Selected Areas in Communications, 38(11), 2450-2525. https://doi.org/10.1109/jsac.2020.3007211

Ding, Z., Fan, P., & Poor, H. V. (2016). Impact of User Pairing on 5G Nonorthogonal Multiple-Access Downlink Transmissions. IEEE Transactions on Vehicular Technology, 65(8), 6010-6023. https://doi.org/10.1109/tvt.2015.2480766

Ding, Z., Leung, K. K., Goeckel, D. L., & Towsley, D. (2011). Opportunistic Relaying for Secrecy Communications: Cooperative Jamming vs. Relay Chatting. IEEE Transactions on Wireless Communications, 10(6), 1725-1729. https://doi.org/10.1109/twc.2011.040511.101694

Dong, L., Han, Z., Petropulu, A. P., & Poor, H. V. (2010). Improving Wireless Physical Layer Security via Cooperating Relays. IEEE Transactions on Signal Processing, 58(3), 1875-1888. https://doi.org/10.1109/tsp.2009.2038412

Fakoorian, S. A. A., & Swindlehurst, A. L. (2011). Solutions for the MIMO Gaussian Wiretap Channel With a Cooperative Jammer. IEEE Transactions on Signal Processing, 59(10), 5013-5022. https://doi.org/10.1109/tsp.2011.2161298

Geraci, G., Egan, M., Yuan, J., Razi, A., & Collings, I. B. (2012). Secrecy Sum-Rates for Multi-User MIMO Regularized Channel Inversion Precoding. IEEE Transactions on Communications, 60(11), 3472-3482. https://doi.org/10.1109/tcomm.2012.072612.110686

Giordani, M., Polese, M., Mezzavilla, M., Rangan, S., & Zorzi, M. (2020). Toward 6G Networks: Use Cases and Technologies. IEEE Communications Magazine, 58(3), 55-61. https://doi.org/10.1109/mcom.001.1900411

Goel, S., & Negi, R. (2008). Guaranteeing Secrecy using Artificial Noise. IEEE Transactions on Wireless Communications, 7(6), 2180-2189. https://doi.org/10.1109/twc.2008.060848

Gopala, P. K., Lai, L., & El Gamal, H. (2008). On the Secrecy Capacity of Fading Channels. IEEE Transactions on Information Theory, 54(10), 4687-4698. https://doi.org/10.1109/tit.2008.928990

He, X., & Yener, A. (2010). Cooperation With an Untrusted Relay: A Secrecy Perspective. IEEE Transactions on Information Theory, 56(8), 3807-3827. https://doi.org/10.1109/tit.2010.2050958

Hu, L., Wen, H., Wu, B., Pan, F., Liao, R.-F., Song, H., Tang, J., & Wang, X. (2018). Cooperative Jamming for Physical Layer Security Enhancement in Internet of Things. IEEE Internet of Things Journal, 5(1), 219-228. https://doi.org/10.1109/jiot.2017.2778185

Huang, C., Zappone, A., Alexandropoulos, G. C., Debbah, M., & Yuen, C. (2019). Reconfigurable Intelligent Surfaces for Energy Efficiency in Wireless Communication. IEEE Transactions on Wireless Communications, 18(8), 4157-4170. https://doi.org/10.1109/twc.2019.2922609

Huang, J., & Swindlehurst, A. L. (2012). Robust Secure Transmission in MISO Channels Based on Worst-Case Optimization. IEEE Transactions on Signal Processing, 60(4), 1696-1707. https://doi.org/10.1109/tsp.2011.2182344

Hwang, K.-S. (2017). Performance Analysis of Secrecy Outage Probability for AF-Based Partial Relay Selection with Outdated Channel Estimates. Mobile Information Systems, 2017, 1-8. https://doi.org/10.1155/2017/7461803

Ikki, S., & Ahmed, M. (2007). Performance Analysis of Cooperative Diversity Wireless Networks over Nakagami-m Fading Channel. IEEE Communications Letters, 11(4), 334-336. https://doi.org/10.1109/lcom.2007.348292

Khisti, A., & Wornell, G. W. (2010). Secure Transmission With Multiple Antennas I: The MISOME Wiretap Channel. IEEE Transactions on Information Theory, 56(7), 3088-3104. https://doi.org/10.1109/tit.2010.2048445

Krikidis, I., Timotheou, S., & Sasaki, S. (2012). RF Energy Transfer for Cooperative Networks: Data Relaying or Energy Harvesting?. IEEE Communications Letters, 16(11), 1772-1775. https://doi.org/10.1109/lcomm.2012.091712.121395

Laneman, J. N., Tse, D. N. C., & Wornell, G. W. (2004). Cooperative Diversity in Wireless Networks: Efficient Protocols and Outage Behavior. IEEE Transactions on Information Theory, 50(12), 3062-3080. https://doi.org/10.1109/tit.2004.838089

Larsson, E. G., Edfors, O., Tufvesson, F., & Marzetta, T. L. (2014). Massive MIMO for next generation wireless systems. IEEE Communications Magazine, 52(2), 186-195. https://doi.org/10.1109/mcom.2014.6736761

Letaief, K. B., Chen, W., Shi, Y., Zhang, J., & Zhang, Y.-J. A. (2019). The Roadmap to 6G: AI Empowered Wireless Networks. IEEE Communications Magazine, 57(8), 84-90. https://doi.org/10.1109/mcom.2019.1900271

Liang, Y., Poor, H. V., & Shamai, S. (2008). Secure Communication Over Fading Channels. IEEE Transactions on Information Theory, 54(6), 2470-2492. https://doi.org/10.1109/tit.2008.921678

Liaskos, C., Nie, S., Tsioliaridou, A., Pitsillides, A., Ioannidis, S., & Akyildiz, I. (2018). A New Wireless Communication Paradigm through Software-Controlled Metasurfaces. IEEE Communications Magazine, 56(9), 162-169. https://doi.org/10.1109/mcom.2018.1700659

Liu, Y., Ding, Z., Elkashlan, M., & Poor, H. V. (2016). Cooperative Non-orthogonal Multiple Access With Simultaneous Wireless Information and Power Transfer. IEEE Journal on Selected Areas in Communications, 34(4), 938-953. https://doi.org/10.1109/jsac.2016.2549378

Liu, Y., Liu, X., Mu, X., Hou, T., Xu, J., Di Renzo, M., & Al-Dhahir, N. (2021). Reconfigurable Intelligent Surfaces: Principles and Opportunities. IEEE Communications Surveys & Tutorials, 23(3), 1546-1577. https://doi.org/10.1109/comst.2021.3077737

Lyu, J., Zeng, Y., Zhang, R., & Lim, T. J. (2017). Placement Optimization of UAV-Mounted Mobile Base Stations. IEEE Communications Letters, 21(3), 604-607. https://doi.org/10.1109/lcomm.2016.2633248

Mao, Y., Clerckx, B., & Li, V. O. K. (2018). Rate-splitting multiple access for downlink communication systems: bridging, generalizing, and outperforming SDMA and NOMA. EURASIP Journal on Wireless Communications and Networking, 2018(1), 133. https://doi.org/10.1186/s13638-018-1104-7

Mao, Y., You, C., Zhang, J., Huang, K., & Letaief, K. B. (2017). A Survey on Mobile Edge Computing: The Communication Perspective. IEEE Communications Surveys & Tutorials, 19(4), 2322-2358. https://doi.org/10.1109/comst.2017.2745201

Mao, Y., Zhang, J., & Letaief, K. B. (2016). Dynamic Computation Offloading for Mobile-Edge Computing With Energy Harvesting Devices. IEEE Journal on Selected Areas in Communications, 34(12), 3590-3605. https://doi.org/10.1109/jsac.2016.2611964

Marzetta, T. L. (2010). Noncooperative Cellular Wireless with Unlimited Numbers of Base Station Antennas. IEEE Transactions on Wireless Communications, 9(11), 3590-3600. https://doi.org/10.1109/twc.2010.092810.091092

Men, J., Ge, J., & Zhang, C. (2017). Performance Analysis of Nonorthogonal Multiple Access for Relaying Networks Over Nakagami-m Fading Channels. IEEE Transactions on Vehicular Technology, 66(2), 1200-1208. https://doi.org/10.1109/tvt.2016.2555399

Mukherjee, A., Fakoorian, S. A. A., Huang, J., & Swindlehurst, A. L. (2014). Principles of Physical Layer Security in Multiuser Wireless Networks: A Survey. IEEE Communications Surveys & Tutorials, 16(3), 1550-1573. https://doi.org/10.1109/surv.2014.012314.00178

Mumtaz, S., Rodriguez, J., & Dai, L. (Eds.). (2017). mmWave Massive MIMO: A Paradigm for 5G. Academic Press. https://doi.org/10.1016/c2015-0-01250-3

Ngo, H. Q., Ashikhmin, A., Yang, H., Larsson, E. G., & Marzetta, T. L. (2017). Cell-Free Massive MIMO Versus Small Cells. IEEE Transactions on Wireless Communications, 16(3), 1834-1850. https://doi.org/10.1109/twc.2017.2655515

Oggier, F., & Hassibi, B. (2011). The Secrecy Capacity of the MIMO Wiretap Channel. IEEE Transactions on Information Theory, 57(8), 4961-4972. https://doi.org/10.1109/tit.2011.2158487

Porambage, P., Okwuibe, J., Liyanage, M., Ylianttila, M., & Taleb, T. (2018). Survey on Multi-Access Edge Computing for Internet of Things Realization. IEEE Communications Surveys & Tutorials, 20(4), 2961-2991. https://doi.org/10.1109/comst.2018.2849509

Rappaport, T. S., Xing, Y., Kanhere, O., Ju, S., Madanayake, A., Mandal, S., Alkhateeb, A., & Trichopoulos, G. C. (2019). Wireless Communications and Applications Above 100 GHz: Opportunities and Challenges for 6G and Beyond. IEEE Access, 7, 78729-78757. https://doi.org/10.1109/access.2019.2921522

Saad, W., Bennis, M., & Chen, M. (2020). A Vision of 6G Wireless Systems: Applications, Trends, Technologies, and Open Research Problems. IEEE Network, 34(3), 134-142. https://doi.org/10.1109/mnet.001.1900287

Sendonaris, A., Erkip, E., & Aazhang, B. (2003). User cooperation diversity-part I: system description. IEEE Transactions on Communications, 51(11), 1927-1938. https://doi.org/10.1109/tcomm.2003.818096

Shahriyer, S. M. S., Badrudduza, A. S. M., Shabab, S., Kundu, M. K., & Yu, H. (2021). Opportunistic Relay in Multicast Channels With Generalized Shadowed Fading Effects: A Physical Layer Security Perspective. IEEE Access, 9, 155726-155739. https://doi.org/10.1109/access.2021.3128572

Shakhatreh, H., Sawalmeh, A. H., Al-Fuqaha, A., Dou, Z., Almaita, E., Khalil, I., Othman, N. S., Khreishah, A., & Guizani, M. (2019). Unmanned Aerial Vehicles (UAVs): A Survey on Civil Applications and Key Research Challenges. IEEE Access, 7, 48572-48634. https://doi.org/10.1109/access.2019.2909530

Sharma, P. K., & Kim, D. I. (2020). Secure 3D Mobile UAV Relaying for Hybrid Satellite-Terrestrial Networks. IEEE Transactions on Wireless Communications, 19(4), 2770-2784. https://doi.org/10.1109/twc.2020.2968296

Sun, X., Ng, D. W. K., Ding, Z., Xu, Y., & Zhong, Z. (2019). Physical Layer Security in UAV Systems: Challenges and Opportunities. IEEE Wireless Communications, 26(5), 40-47. https://doi.org/10.1109/mwc.001.1900028

Tataria, H., Shafi, M., Molisch, A. F., Dohler, M., Sjöland, H., & Tufvesson, F. (2021). 6G Wireless Systems: Vision, Requirements, Challenges, Insights, and Opportunities. Proceedings of the IEEE, 109(7), 1166-1199. https://doi.org/10.1109/jproc.2021.3061701

Trappe, W. (2015). The challenges facing physical layer security. IEEE Communications Magazine, 53(6), 16-20. https://doi.org/10.1109/mcom.2015.7120011

Wang, C.-X., Bian, J., Sun, J., Zhang, W., & Zhang, M. (2018). A Survey of 5G Channel Measurements and Models. IEEE Communications Surveys & Tutorials, 20(4), 3142-3168. https://doi.org/10.1109/comst.2018.2862141

Wang, H.-M., Zheng, T.-X., Yuan, J., Towsley, D., & Lee, M. H. (2016). Physical Layer Security in Heterogeneous Cellular Networks. IEEE Transactions on Communications, 64(3), 1204-1219. https://doi.org/10.1109/tcomm.2016.2519402

Wu, Q., & Zhang, R. (2019). Intelligent Reflecting Surface Enhanced Wireless Network via Joint Active and Passive Beamforming. IEEE Transactions on Wireless Communications, 18(11), 5394-5409. https://doi.org/10.1109/twc.2019.2936025

Wyner, A. D. (1975). The Wire-Tap Channel. Bell System Technical Journal, 54(8), 1355-1387. https://doi.org/10.1002/j.1538-7305.1975.tb02040.x

Yan, S., Yang, N., Malaney, R., & Yuan, J. (2014). Transmit Antenna Selection with Alamouti Coding and Power Allocation in MIMO Wiretap Channels. IEEE Transactions on Wireless Communications, 13(3), 1656-1667. https://doi.org/10.1109/twc.2013.013014.131248

Yang, N., Yeoh, P. L., Elkashlan, M., Schober, R., & Collings, I. B. (2013). Transmit Antenna Selection for Security Enhancement in MIMO Wiretap Channels. IEEE Transactions on Communications, 61(1), 144-154. https://doi.org/10.1109/tcomm.2012.12.110670

You, X., Wang, C.-X., Huang, J., Gao, X., Zhang, Z., Wang, M., Huang, Y., Zhang, C., Jiang, Y., Wang, J., Zhu, M., Sheng, B., Wang, D., Pan, Z., Zhu, P., Yang, Y., Liu, Z., Zhang, P., Tao, X., ... Liang, Y.-C. (2021). Towards 6G wireless communication networks: vision, enabling technologies, and new paradigm shifts. Science China Information Sciences, 64(1), 110301. https://doi.org/10.1007/s11432-020-2955-6

Zeng, Y., Zhang, R., & Lim, T. J. (2016). Wireless communications with unmanned aerial vehicles: opportunities and challenges. IEEE Communications Magazine, 54(5), 36-42. https://doi.org/10.1109/mcom.2016.7470933

Zhang, J., Bjornson, E., Matthaiou, M., Ng, D. W. K., Yang, H., & Love, D. J. (2020). Prospective Multiple Antenna Technologies for Beyond 5G. IEEE Journal on Selected Areas in Communications, 38(8), 1637-1660. https://doi.org/10.1109/jsac.2020.3000826

Zhang, J., Duong, T. Q., Marshall, A., & Woods, R. (2016). Key Generation From Wireless Channels: A Review. IEEE Access, 4, 614-626. https://doi.org/10.1109/access.2016.2521718

Zhang, Z., Xiao, Y., Ma, Z., Xiao, M., Ding, Z., Lei, X., Karagiannidis, G. K., & Fan, P. (2019). 6G Wireless Networks: Vision, Requirements, Architecture, and Key Technologies. IEEE Vehicular Technology Magazine, 14(3), 28-41. https://doi.org/10.1109/mvt.2019.2921208

Zhou, X., McKay, M. R., Maham, B., & Hjorungnes, A. (2011). Rethinking the Secrecy Outage Formulation: A Secure Transmission Design Perspective. IEEE Communications Letters, 15(3), 302-304. https://doi.org/10.1109/lcomm.2011.011811.102433

Zhou, Y., Pan, C., Yeoh, P. L., Wang, K., Elkashlan, M., Vucetic, B., & Li, Y. (2020). Secure Communications for UAV-Enabled Mobile Edge Computing Systems. IEEE Transactions on Communications, 68(1), 376-388. https://doi.org/10.1109/tcomm.2019.2947921

Zou, Y., Wang, X., & Shen, W. (2013). Optimal Relay Selection for Physical-Layer Security in Cooperative Wireless Networks. IEEE Journal on Selected Areas in Communications, 31(10), 2099-2111. https://doi.org/10.1109/jsac.2013.131011

Zou, Y., Zhu, J., Wang, X., & Hanzo, L. (2016). A Survey on Wireless Security: Technical Challenges, Recent Advances, and Future Trends. Proceedings of the IEEE, 104(9), 1727-1765. https://doi.org/10.1109/jproc.2016.2558521