Internet of Things and Sensors Networks in 5G Wireless CommunicationsPrinted Edition of the Special Issue Published in Sensorswww.mdpi.com/journal/sensorsLei Zhang, Guodong Zhao and Muhammad Ali ImranEdited by Internet of Things and Sensors Networks in 5G Wireless Communications • Lei Zhang, Guodong Zhao and Muhammad Ali ImranInternet of Things and Sensors Networks in 5G Wireless CommunicationsInternet of Things and Sensors Networks in 5G Wireless CommunicationsSpecial Issue EditorsLei ZhangGuodong ZhaoMuhammad Ali ImranMDPI • Basel • Beijing • Wuhan • Barcelona • BelgradeGuodong Zhao University of Glasgow UKSpecial Issue Editors Lei ZhangUniversity of GlasgowUKMuhammad Ali ImranUniversity of Glasgow UKEditorial OfﬁceMDPISt. Alban-Anlage 66 4052 Basel, SwitzerlandThis is a reprint of articles from the Special Issue published online in the open access journal Sensors(ISSN 1424-8220) from 2019 to 2020 (available at: https://www.mdpi.com/journal/sensors/specialissues/IOT SN5G)For citation purposes, cite each article independently as indicated on the article page online and asindicated below:LastName, A.A.; LastName, B.B.; LastName, C.C. Article Title. Journal Name Year, Article Number,Page Range.ISBN 978-3-03928-148-0 (Pbk)ISBN 978-3-03928-149-7 (PDF)c© 2020 by the authors. Articles in this book are Open Access and distributed under the CreativeCommons Attribution (CC BY) license, which allows users to download, copy and build uponpublished articles, as long as the author and publisher are properly credited, which ensuresmaximumdissemination and a wider impact of our publications.The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commonslicense CC BY-NC-ND.ContentsAbout the Special Issue Editors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiRabeea Basir, Saad Qaisar, Mudassar Ali, Monther Aldwairi, Muhammad Ikram Ashraf,Aamir Mahmood and Mikael GidlundFog Computing Enabling Industrial Internet of Things: State-of-the-Art andResearch ChallengesReprinted from: Sensors 2019, 19, 4807, doi:10.3390/s19214807 . . . . . . . . . . . . . . . . . . . . 1Collins Burton Mwakwat, Hassan Malik, Muhammad Mahtab Alam, Yannick Le Moullec,Sven Parand and Shahid MumtazNarrowband Internet of Things (NB-IoT): From Physical (PHY) and Media Access Control(MAC) Layers PerspectivesReprinted from: Sensors 2019, 19, 2613, doi:10.3390/s19112613 . . . . . . . . . . . . . . . . . . . . 39Ahmed Adel Aly, Hussein M. ELAttar, Hesham ElBadawy and Wael AbbasAggregated Throughput Prediction for Collated Massive Machine-Type Communications in 5GWireless NetworksReprinted from: Sensors 2019, 19, 3651, doi:10.3390/s19173651 . . . . . . . . . . . . . . . . . . . . 73Wenjun Hou, Song Li, Yanjing Sun, Jiasi Zhou and Nannan LuInterference-Aware Subcarrier Allocation for Massive Machine-Type Communication in5G-Enabled Internet of ThingsReprinted from: Sensors 2019, 19, 4530, doi:10.3390/s19204530 . . . . . . . . . . . . . . . . . . . . 93Muhammad Asad Ullah, Junnaid Iqbal, Arliones Hoeller, Richard Demo Souza and Hirley AlvesK-Means Spreading Factor Allocation for Large-Scale LoRa NetworksReprinted from: Sensors 2019, 19, 4723, doi:10.3390/s19214723 . . . . . . . . . . . . . . . . . . . . 106Shuang Zhang and Guixia KangUser Association and Power Control for Energy Efﬁciency Maximization in M2M-EnabledUplink Heterogeneous Networks with NOMAReprinted from: Sensors 2019, 19, 5307, doi:10.3390/s19235307 . . . . . . . . . . . . . . . . . . . . 125Jingyun Sun, Rongke Liu and Enrico PaoliniA Dynamic Access Probability Adjustment Strategy for Coded Random Access SchemesReprinted from: Sensors 2019, 19, 4206, doi:10.3390/s19194206 . . . . . . . . . . . . . . . . . . . . 143M. Carmen Lucas-Estan˜, Javier Gozalvez and Miguel SepulcreOn the Capacity of 5G NR Grant-Free Scheduling with Shared Radio Resources to SupportUltra-Reliable and Low-Latency CommunicationsReprinted from: Sensors 2019, 19, 3575, doi:10.3390/s19163575 . . . . . . . . . . . . . . . . . . . . 161Jiewen Deng, Wanrong Sun, Lei Guan, Nan Zhao, Muhammad Bilal Khan, Aifeng Ren,Jianxun Zhao, Xiaodong Yang and Qammer H. AbbasiNoninvasive Suspicious Liquid Detection Using Wireless SignalsReprinted from: Sensors 2019, 19, 4086, doi:10.3390/s19194086 . . . . . . . . . . . . . . . . . . . . 179Mohammad Kazem Chamran, Kok-Lim Alvin Yau, Raﬁdah M. D. Noor and Richard WongA Distributed Testbed for 5G Scenarios: An Experimental StudyReprinted from: Sensors 2020, 20, 18, doi:10.3390/s20010018 . . . . . . . . . . . . . . . . . . . . . 190vAbout the Special Issue EditorsLei Zhang (Dr.) is a Lecturer at the University of Glasgow, UK. He received his Ph.D. from theUniversity of Shefﬁeld, UK. He worked as a research engineer in the Huawei CommunicationTechnology Laboratory (CT Lab), and a research fellow in the 5G Innovation Centre (5GIC),Institute of Communications (ICS), University of Surrey, UK. His research interests broadly lie incommunications and networks, including wireless blockchain networks, radio access network slicing(RAN slicing), new air interface designs, Internet of Things (IoT), multi-antenna signal processing,and massive MIMO systems. He has 19 US/UK/EU/China granted/ﬁled patents on wirelesscommunications and has published over 100 peer-reviewed papers. Dr. Lei Zhang also holds avisiting position in 5GIC at the University of Surrey. He is an associate editor of IEEE ACCESS and asenior member of IEEE.Guodong Zhao (Dr.) received his B.E. degree from Xidian University, Xi’an, China, in 2005, andhis Ph.D. degree from Beihang University, Beijing, China, in 2011, both in Electrical Engineering.From 2011 to 2018, he was an associate professor at the University of Electronic Science andTechnology of China (UESTC), Chengdu, China. In 2018, he joined the University of Glasgowin the UK as a lecturer (assistant professor). He has 10 years of experience working on wirelesscommunications with international partners and he is a senior member of IEEE. He has publishedone book with Springer press and more than 50 peer-reviewed research papers (including more than10 IEEE transaction papers), had over 1700 citations in Google Scholar, and won a best paper awardin IEEE Globecom, 2012, and a best poster award in IEEE WCNC, 2018. His current research interestsare within the areas of wireless communication, control, and robotics.Muhammad Ali Imran (Prof.) is a Fellow of IET, Senior Member of IEEE and Senior Fellow ofthe Higher Education Academy UK. He is a Professor of Wireless Communication Systems, withresearch interests in self-organised networks, wireless networked control systems and wireless sensorsystems. He heads the Communications, Sensing and Imaging (CSI) research group at the Universityof Glasgow and is Dean at the University of Glasgow, UESTC. He is an Afﬁliate Professor at theUniversity of Oklahoma, USA and a visiting Professor at the 5G Innovation Centre, University ofSurrey, UK. He has over 20 years of combined academic and industry experience, with several leadingroles in multi-million GBP-funded projects. He has ﬁled 15 patents; has authored/co-authoredover 400 journal and conference publications; was editor of ﬁve books and author of more than20 book chapters; and successfully supervised over 40 postgraduate students at the Doctoral level.He has been a consultant for international projects and local companies in the area of self-organisednetworks. He has been interviewed by the BBC, Scottish television and many radio channels on thetopic of 5G technology.viisensorsReviewFog Computing Enabling Industrial Internet ofThings: State-of-the-Art and Research ChallengesRabeea Basir 1, Saad Qaisar 1, Mudassar Ali 1,2,* , Monther Aldwairi 3,Muhammad Ikram Ashraf 4, Aamir Mahmood 5 and Mikael Gidlund 51 School of Electrical Engineering and Computer Science, National University of Science and Technology,Islamabad 44000, Pakistan; rbasir.dphd17@seecs.edu.pk or rabeeabasir@gmail.com (R.B.);saad.qaisar@seecs.edu.pk (S.Q.)2 Department of Telecommunication Engineering, University of Engineering and Technology,Taxila 47050, Pakistan3 College of Technological Innovation, Zayed University, Abu Dhabi 144534, UAE; monther.aldwairi@zu.ac.ae4 Centre for Wireless Communication, University of Oulu, 90014 Oulu, Finland; ikram.ashraf@oulu.ﬁ orikramashraf@gmail.com5 Department of Information Systems and Technology, Mid Sweden University, 85170 Sundsvall, Sweden;aamir.mahmood@miun.se (A.M.); mikael.gidlund@miun.se (M.G.)* Correspondence: mudassar.ali@hotmail.com or mudassar.ali@seecs.edu.pkReceived: 14 August 2019; Accepted: 23 October 2019; Published: 5 November 2019Abstract: Industry is going through a transformation phase, enabling automation and dataexchange in manufacturing technologies and processes, and this transformation is called Industry4.0. Industrial Internet-of-Things (IIoT) applications require real-time processing, near-by storage,ultra-low latency, reliability and high data rate, all of which can be satisﬁed by fog computingarchitecture. With smart devices expected to grow exponentially, the need for an optimized fogcomputing architecture and protocols is crucial. Therein, efﬁcient, intelligent and decentralizedsolutions are required to ensure real-time connectivity, reliability and green communication. In thispaper, we provide a comprehensive review of methods and techniques in fog computing. Our focusis on fog infrastructure and protocols in the context of IIoT applications. This article has two mainresearch areas: In the ﬁrst half, we discuss the history of industrial revolution, application areas ofIIoT followed by key enabling technologies that act as building blocks for industrial transformation.In the second half, we focus on fog computing, providing solutions to critical challenges and as anenabler for IIoT application domains. Finally, open research challenges are discussed to enlighten fogcomputing aspects in different ﬁelds and technologies.Keywords: Industry 4.0; Internet of Things; Industrial Internet of Things; Cyber PhysicalSystem; cloud computing; fog computing; edge computing; smart devices; smart factory;industrial automation1. IntroductionRevolution in any realm is required with the passage of time. Every ﬁeld changes to go forwardwith better solutions dealing with the challenges of the era. Industrial Internet of Things (IIoT) isrevolutionizing the classical communication methodologies. With the emergence of smart devices(mobile, machines, sensors) coupled with a diverse range of applications requirements, IIoT is theway forward. It is expected that 26 billion IoT devices of heterogeneous capabilities will be installedto perform functions with different Quality-of-Service (QoS) requirements by 2020 [1]. IIoT givesrise to 4th industrial revolution based on Cyber-Physical Systems (CPS) with the need arising backin 2015 originated basically in Germany [2]. Industry 4.0 deﬁnes diverse use cases ranging fromSensors 2019, 19, 4807; doi:10.3390/s19214807 www.mdpi.com/journal/sensors1Sensors 2019, 19, 4807interconnected digital technologies, CPS, Mobile Cloud Computing (MCC) and Internet of Things (IoT)for promoting the whole industry in terms of efﬁciency, effectiveness, supporting heterogeneous data,higher production, automation, and integrating knowledge [3]. These key enabling technologies havebeen deployed to some extent in industrial domains such as healthcare, transportation, smart cities,micro-grids, and smart factory. This trend gives rise to intelligent, distributed and self-organizingsolutions to support these application domains.Deploying industry 4.0 involves three-layer implementation; physical layer, network layer, andintelligent-application layer [4]. The physical layer comprises identiﬁcation and location awarenessentities i.e. actuators, sensors, and terminal devices; the network layer comprises of the developmentof a network that can support industrial automation, network can be cellular, indoor, cloud orprivate. Factory automation and coordination are processed on the application layer. Infrared (IR),Radio-Frequency Identiﬁcation (RFID), Bluetooth, 6LoWPAN, IEEE 802.11 af, IEEE 802.11 a/b/n/acfor short range connectivity; Ultra-Wideband (UWB), cellular (2G, 3G, 4G, LTE-MTC, 5G), Sigfox, Longrange (LoRa) for long range connectivity, are a few of the majorly used communication standards forIIoT [5,6].The future of automation is based on decentralized intelligence in which all machinescan communicate with one another to arrive at independent or consensus inference, calledMachine-to-Machine (M2M) communication. These decentralized intelligent solutions play a vitalrole in industry 4.0 digital transformation. The decentralized solutions provide ﬂexibility and quickdecision assistance over centralized solutions. For M2M communication, 802.11ah technology hasevolved in the recent past. Exchanging machine data demands real-time communication ensuringlatency, security, reliability, bandwidth and privacy measures in all IIoT domains. To satisfy thesecritical requirements, there is a need to explore new enabling solutions that support these applications.In the future, 5G cellular technology will support such heterogeneous networks with massive numberof IIoT devices. It is anticipated that future 5G networks not only provide ﬂexibility but can optimizethe usage of available resources of bandwidth, power, energy, connectivity to different applications atthe same time [7].In the last decade, computation and processing requirements of end users have increasedexponentially. It has become increasingly challenging for designers to scale the processing anddata storage capabilities for users within the given device size and battery constraints. To meet thesegrowing requirements, researchers have come up with the solution to ofﬂoad services to a centralizedlocation known as the cloud. Cloud computing is an alternative for data computation, storageand management. It supports intensive computation and manages heterogeneous devices of nextgeneration networks [8–10]. Additionally, cloud computing architecture involves the direct connectionbetween devices and the cloud server. Practically, we are beginning to understand the connectionbetween and the enormous number of IIoT devices and a single cloud server. However, cloud-basedsystems are unable to meet the requirement such as heavy data computation, real-time devicecontrol, security and management results in insufﬁcient support of IIoT application requirements [11].Considering a wide variety of IoT scenarios, some of the challenges [10–16] in cloud computing arelisted below:• Large distance between the cloud and edge devices causes propagation and transmission delays.• Large computational load on a single cloud server causes processing and queuing delays.• Increased number of smart devices has hindered meeting the bandwidth requirements.• Enormous number of smart devices will bring scalability, speed, and computational issues.• Wireless medium between cloud and smart devices brings resource management issues.• Heterogeneity property of smart devices in terms of accessing technology will bring difﬁculty inhandling at the cloud.• Mobility of IoT devices bring service availability issues, cloud server may not be able to provideservices due to network congestion and failure.2Sensors 2019, 19, 4807• Security is a very critical thread, as the cloud is exposed to the whole world over the public internet.• Computing ofﬂoading every-time at cloud causes a loss in energy and battery lifetime.• Although data storage at cloud brings beneﬁts to application developers, they should be carefulof integrity and authentication demands of IIoT applications.• Cloud computing is a centralized and complex architecture for real-time applications of IIoT.All these limitations require a change, how and where we process data. These challenges motivateus to explore new decentralized approaches/solutions in IIoT domains. A new concept of fogcomputing is introduced by Bonomi et al. [15] for handling data locally at the network edge inorder to overcome the limitations of cloud architecture. Fog computing complements existing cloudarchitecture and has addressed the issue of latency and bandwidth efﬁciency [17]. Because of itsdistributed architecture, it calls for a strong check on QoS requirements to make it useful. Fog ismainly based on distributed networking with ubiquitous pervasive computing. It comprises smallscale data centers or a group of computers known as cloudlets (fog clouds) that provide services todevices located in close proximity [17,18]. The initial installation cost, latency, and energy consumptionis far less as compared to that of the cloud, but the operational cost varies. Fog architecture canleverage computations either from dedicated edge servers or adhoc infrastructure. For promoting IIoTarchitecture with fog computing as a key enabler technology, a group of fog clouds can also be used.In fog computing, data processing in single server (fog cloud) helps in achieving real-time andreliable communication. It puts the safety and security of personal data back into our premises.Furthermore, a cost effective approach can be used in fog computing such that data transmissionand storage fees can be reduced based on service premises. Therefore, fog computing has thepotential to provide affordable solutions for large IIoT projects. Instead of being restricted to onlyone expensive cloud connection, fog computing gives the freedom to choose any hardware fromInformation Technology (IT) solutions. It supports all existing legacy devices and non-IIoT devices thatnever intended to be the part of IIoT application. This is not only economical but also more ﬂexible.When it comes to speed, fog computing allows real-time processing and supports to process dataas fast as our local system. Fog can be managed securely from remote places. It can be scaled andupdated dynamically. It gives more security, better performance, and lower costs. Fog incorporatespositive attributes of cloud and provides beneﬁts that may support future IIoT applications [18–23].Fog computing and edge computing being extended form of cloud computing gives solutions tothe challenges faced by cloud computing that is attractive for IIoT real-time applications. The termsfog computing and edge computing are often used by industry interchangeably. Both these computingtechnologies bring computing and processing capabilities near the vicinity where data originates.Edge computing complements fog computing by bringing computation to one of the devices of anetwork. This device is named as E-node and is close to the data. E-node has more power, computationcapabilities and intelligent controllers, such as programmable automation controllers (PAC). Presenceof E-node in edge computing improves latency, reliability, security and privacy issues [24,25]. E-nodeacts as an interface/bridge between the data sources and the cloud. The basic architecture for fognetwork is given in Figure 1 depicting fog cloud serving as a middle layer between the cloud server andsmart end-devices. Figure 1 demonstrates a basic idea of cloud, fog and edge computing promotingdifferent IIoT application domains.Fog is a relatively new paradigm that brings new challenges in terms of efﬁcient and scalablenetwork architecture. It is expected that it will gradually develop over the next few years for realizingthe Industry 4.0. Challenges, such as energy conservation, real-time communication, efﬁcient spectrumuse, cache memory on edge devices and optimized allocation of resources are open issues that needto be addressed for future automation. Without such considerations, guaranteed QoS requirementsof IoT devices may not be fulﬁlled. In the future, solutions to these challenges must be provided byresearchers for the development of the industrial revolution. This paper is written with an aim togive a summarized version of existing solutions using fog computing acting as an enabler for IIoTapplications.3Sensors 2019, 19, 4807Figure 1. Generalized view: IIoT application domains with cloud, fog and edge computing.The paper is organized as follows; Section 2 brieﬂy introduces IIoT. Beneﬁts of IIoT applicationsin daily life and their critical requirements are brieﬂy explained in Section 3. Section 4 presentsprotocol/solution proposed by various researchers promoting fog computing as an enabling technologyfor IIoT development. Section 5 describes challenges and solutions in communication and networkingproposed in the literature to use fog computing in IIoT. Section 6 lists down several open researchissues in fog computing. Finally, the paper is concluded in Section 7. The ﬂow of this survey paper isshown in Figure 2.Figure 2. Flow of the paper.4Sensors 2019, 19, 48072. Evolution and Enablers of Industrial Internet of ThingsAs discussed earlier, IIoT or Industry 4.0 is a new emerging term for future industry, whichinvolves many key enabling technologies and applications of IoT. In this section, Industry 4.0evolutional phases, IoT connectivity technologies, beneﬁts from IIoT and key enabling technologiesthat endorse industrial revolution are brieﬂy explained.2.1. Industry 4.0-EvolutionThe industrial revolution with the passage of time has many phases according to the requirementand challenges of the respective era. Figure 3 gives an idea about evolution towards Industry 4.0 withits elements.Figure 3. Evolution towards Industry 4.0/IIoT.Industry 1.0: At the end of the 18th century, the 1st industrial revolution started with the helpof water and steam power, which systematizes the factory ﬂoor. First, the mechanical weaving loomwas established in 1784, and the ﬁrst mechanical system was built thorough mechanical productionfacilities.Industry 2.0: In the beginning of the 20th century, the 2nd industrial revolution startedusing electrical energy. The ﬁrst assembly line using electrical energy was established in 1870.The introduction of mass production in industry 2.0 enhanced the industry.Industry 3.0: Beginning of the 1970s i.e., in 1969, the ﬁrst control system using programminglanguage was established. Industry was slowly shifted to automation using information technologyand micro-electronics’s applications. This is the 3rd industrial revolution [26].Industry 4.0: This previous industrial revolutions give rise to the development of industry4.0. Industry 4.0 contributes a revolution to all domains comprising economic, academic, research,industrial and manufacturing sectors. There is a huge impact of the industrial revolution onthe manufacturing processes of many ﬁelds. Implementing industry 4.0 demands change inmany technologies namely automation, identiﬁcation, computer, network communication, digitalmanufacturing, production process, production control management, decision making, judgment,sensing and analysis [27]. In the future, the manufacturing industry is expected to change on a largescale because of all new generation networks and interfaces offered by the environment of industry4.0. This transformation is already in process in many industrial sectors. Up till now, for the fourthindustrial revolution, exponentially growing technologies are sensor technology, artiﬁcial intelligence,5Sensors 2019, 19, 4807machine learning, robotics, nanotechnology, and 3D printing [28]. These technologies were inventeddecades ago, but their minimum cost and exponential growth will shape industry 4.0. To change theindustrial process, researchers are focusing on providing the evolved form of these technologies interms of ﬂexibility and fast computational process. All this automation in industry is very importantfor the economic growth of a country.2.2. Industry 4.0-ConceptIncreasing progress is witnessed in automation of industry using advancement in digitization,networking and new communication technologies, satisfying themarket and consumer requirements [29].Lenze SE, Corporate Communications Public Relations use idea of machine modularization. According tothe demand of market/consumers, different modules are added or removed during the manufacturingprocess; machines are retooled smartly using smart communication technologies A cloud solution wasgiven by Lenze, which is secure as no customer wants to share their demands and production details.Details about the customer’s machine are saved in a cloud and he can investigate all the system detailsespecially faults. This cloud solution is vulnerable to hackers, Lenze, along with another company, haveprovided a secure solution that is acceptable for today’s industry [30].Charlotta Johnsson explained the idea of industry 4.0 using four terms, these are smart devicesand smart production processes with horizontally and vertically integrated manufacturing systems.Smart devices result in the production of intelligent products, these products do self-monitoring,self-controlling and self-manufacturing, have a uniquely identiﬁable ID, know how to solve and achievegoals [31]. The intelligent production process comprises smart starting and ending of manufacturingprocesses. Vertical and horizontal integration means all the steps during the smart/intelligentproduction process are integrated throughout the life cycle i.e., from starting phase to ending phase [30].Industry 4.0 results in a faster manufacturing process, product development and improvesthe handling of complex environments inside an industry. The term ﬁrst originated in Germanynamed as Industrie 4.0; in United States term used for this fourth generation is Smart Manufacturing,Chinese researchers have used term China 2020. Industrial Digitisation is the term used in Swedenfor transformation of industry to automation [32]. It is believed that this industrial revolution willincrease global competitiveness, preserve the domestic manufacturing industry and will have a hugeimpact on the business market as well. Now many countries around the globe have taken initiativesfor automation in industries.2.3. Industry 4.0-Merging CPS and IoTThe Industry 4.0 environment is comprised of the Internet of data, Internet of things, Internetof people and Internet of services. Interface of Industry 4.0 with existing smart infrastructure suchas smart buildings, smart homes, smart grids, smart logistics, social web, and business web build aCPS system. This revolution will merge the real and virtual world on the basis of CPS. A CPS systemhas a computer-based algorithm that integrates the Internet and its users. It is simply digitization,in which these systems make connection of information technology with electronic/mechanical devicecomponents that exchange information among each other using a network. Using computer-basedalgorithms, CPS brings software and hardware components working in an automated and controlledmanner to perform a certain task without human’s assistance. The basic visualization of a CPSis given in Figure 4. After collection and analysis of big data, CPS can increase performance interms of high-quality, low-cost goods production. With the advancement in sensors and computingtechnologies, various CPSs are emerging. CPS has evolved to Cyber Physical Production Systems(CPPS) to encourage the development and production process of Industry 4.0 [33]. CPPS combinesphysical smart IoT devices, networking technologies to compute in the production process. Robotics,remote machinery control and diagnosis, smart devices, heavy industry, transportation, health andcondition monitoring, energy production, smart cities, and food manufacturing are IIoT services enableby CPS/CPPS architecture. Figure 5 represents the comparative analysis between CPS and IoT in6Sensors 2019, 19, 4807the form of a Venn diagram. The similarities between the two give support for the development ofIndustry 4.0/IIoT.Figure 4. A cyber physical system architecture.Figure 5. Comparison of CPS and IoT; supporting Industry 4.0 development.7Sensors 2019, 19, 4807The layered architecture of Industry 4.0 is given in Figure 3, which involves the common attributesof CPS and IoT. The physical or sensing layer should be designed to the extent that IIoT applicationscan sense/control information from physical environment and integrates with hardware sensors andactuators accordingly. The second layer should be optimized to provide a reliable connection tosupport data transfer over a communication medium (wired/wireless). So far, IIoT applications usedwired medium to provide solutions. In the near future, the wireless medium is required because ofshifting from centralized to decentralized solutions. Connectivity technologies, such as NB-IoT/5Gand beyond 5G over different architecture such as SDN, NFV, cloud computing or fog computing willgive solution to different applications. The intelligent-application layer, providing services to usershas to be optimized in terms of service production, satisfaction, interaction, and management.2.4. Industry 4.0-Key Enabling TechnologiesWith the use of advance technologies of wireless communication, diverse new emerging protocolsand architectures are supporting automated industry 4.0 development. Resources can be efﬁcientlyused after integration of communication technology and big data processing in real time, this willresult in better performance. Industry 4.0 development involves many communication technologies;however, big data, IoT, 5G, mobile computing and cloud/fog/edge computing are the key enablingtechnologies [2,34,35]. An extensive range of IIoT projects have been deployed in domains of buildingautomation, manufacturing systems, health care systems, transportation systems, processing food andagricultural systems in the past few years. Reaching a common task in an IIoT application; sensing,integration, and communication are main steps. RFID tags are used for sensing, network topologiesand protocols are used for communication. All these smart devices are associated with each otherusing internet. Many connectivity technologies are available for supporting IIoT applications. Criticalrequirements of IIoT applications have many open challenges in all domains (smart grid, smart cities,smart devices, D2D, healthcare) such as capacity, real-time connectivity, remote maintenance andtopology of communication networks. Figure 6 gives a general overview of technologies to connectthings to the Internet, representing short-range and long-range wireless technologies. All technologieswork differently with aim of low-latency, low-power consumption, low-bandwidth requirement,and reliable communication.Figure 6. Different connectivity technologies in IIoT.8Sensors 2019, 19, 4807For connecting IIoT devices, all technologies have to work with the objective of maximumthroughput, minimum power consumption, minimum transmission delay, and maximum transmissiondistance range. 2G, 3G, 4G, LTE are cellular technologies that were used for long range connectivityin wireless wide area networks (WWAN). IIoT application’s critical requirements and the increasingnumber of smart devices need additional resources for connectivity. Increase in smart devices resultsin more data processing for which connectivity technology is moving towards 5G. The 3rd GenerationPartnership Project (3GPP) proposed Extended Coverage-Global System for Mobile Communicationsfor the Internet of Things (EC-GSM-IoT) and Narrowband-Internet of Things (NB-IoT) for supportingM2M, enhanced MTC (eMTC), massive MTC (mMTC) and critical MTC (cMTC) communicationnetworks for IIoT applications [5]. 5G cellular technology gives super low latency, ultra-reliable andhigh availability to cMTC applications (industrial application and control, remote surgery, remotetraining, remote manufacturing, and trafﬁc safety and control). Low cost, low energy, and the massivenumber of intelligent devices in smart agriculture, smart meter, tracking, ﬂeet management, andlogistics domain are supported by 5G as well. 5G is beneﬁcial for IIoT applications comprising frommMTC, cMTC to enhanced mobile broadband. The distributed model of IIoT applications require amassive amount of data rate with minimum latency, 5G technology gives 10 Gbps with 1 ms latency.5G is use case driven communication technology for upcoming IIoT applications.For distributed ultra-low-latency and reliable connectivity in IIoT applications, 5G-IoT is anemerging solution. 5G-IoT scenario extends capabilities of IoT smart devices used in all domains.Recent research is focusing on low-latency, end-to-end reliability, and low energy consumption forboth uplink and downlink communication. There is a lot of potential in research on IIoT with5G communication technologies, to overcome challenges. This research will help in the industrialrevolution. With the evolution of Industry 4.0, 5G is rapidly evolving in order to meet the requirementof IIoT applications mainly real time functioning, energy efﬁciency, less power consumption, sharedspectrum regulation, reliable communication, and handing massive amount of data. Almost 90% ofneeds met using ﬁxed line 3G and 4G cellular technologies, but need for deployment of industrialrevolution can be fulﬁlled using 5G. 5G as an enabler of industry 4.0 gives multi-channel, capability,multi-network management, operating both local and global networks, supporting heterogeneousnetworks [27]. Mobile computing and cloud computing brings accurate data for IIoT application andprovide efﬁciency to industry 4.0 infrastructure. Details of cloud computing in comparison with fogand edge computing is explained in the next section.2.5. Industry 4.0-Building BlocksFourth manufacturing revolution, i.e., digital industrial technology provides services in industriesthat involves data exchange among machines making more efﬁcient and fast processes. In words,the IIoT can be deﬁned as: Devices with centralized controllers, sensors, battery and memory attributes willinteract with each other using Artiﬁcial Intelligence (AI) and Machine Learning (ML) algorithms. Real timeconnection is possible using decentralized analytics and decision making of these devices. This sectionwill give building blocks that are used in transforming industry 4.0 development.2.5.1. Simulation, Autonomous RobotsA virtual model of a physical world which comprises machines, humans, and products can beinterpreted in real time technology named as simulation. Every newproduct or updating process in availableproducts for any machine can be verified, tested and optimized via simulation-based applications. It willresult in increasing the quality of machinery and save the resources in the physical world. An autonomousrobot collects information from its environment and learns from it and does work in the future without theinvolvement of humans using its self-learning algorithms (machine learning). These robots will transformthe industries into automated industry. This technology will have a large impact on the industrial revolution.These robots are cheap and more capable of doing tasks efficiently.9Sensors 2019, 19, 48072.5.2. Big Data and Analytics, Horizontal and Vertical System IntegrationDifferent systems, ranging from customer to enterprise-level systems, have to collect, manageand evaluate the big amount of data. Three main goals of big data analytics result in the reductionof cost, efﬁcient decision making and emerging new services and products. Industry 4.0 involvesdigital transformation in vertical and horizontal value chain networks. These networks result inthe integration of customer and enterprise systems of companies, departments and business market-exchanging data. These value chain processes should be transparent and ﬂexible with real-timefunctioning constraint.2.5.3. Additive ManufacturingAdditive manufacturing is the process in which a 3D model is manufactured by joining the rawmaterials, usually layer by layer. It is the opposite of subtractive manufacturing in which raw materialis carved to create a 3D model.2.5.4. Augmented RealityThe idea of taking decisions remotely in real-time results in improving work procedures and willbe implemented in the future as Augmented Reality (AR). In this augmented reality-based systemssend repairing requirements or selection of new components.2.5.5. Cyber-securityExponential increase in connections among devices in industry 4.0 will increase threats to systems,networks, and processes. Cyber-security is a process that prevents unwanted intruders from accessing,destroying, interrupting or changing sensitive information about company/organization as well asbusiness market networks; gives them secure and reliable communication systems.2.5.6. Cloud ComputingCloud computing is centralized and complex technology that supports high speed, highperformance, ﬂexible resource use and dynamic allocation in a network. As IIoT applicationrequirements are low latency, high speed and reliable communication, privacy and security, efﬁcientallocation of resources and energy-efﬁcient communication technology. Some limitations regardinguse of cloud computing for IIoT applications are:• Conﬁdential data and personal information of an industry should not be shared with outsiders.• Security and privacy are in high demand by an industry from the cloud service provider.• Data location on the basis of geographic follows rules and regulations. It also helps in securing theinformation.• High load demands high-speed internet connectivity. This processing causes delays in communication.• Memory and storage capacity may get exhausted because of many applications simultaneouslyaccessing a single cloud server.• Context awareness is required for speedy processes.• Different standards cause problems in exchanging data, information, services, and applicationsamong different clouds at different locations.• Recovery and back-up update are required for industrial processing and decision making, cloudcomputing will cause delay.2.5.7. Fog ComputingFog computing or “fogging” is an extended form of cloud computing, in respect of industrialrevolution giving applications and services (low latency and high processing) to autonomousheterogeneous devices inside an industry [36]. The idea is to bring processing, storage, maintenance10Sensors 2019, 19, 4807and intelligence control to the proximity of data devices. Inside industry 4.0, there is criticalrequirement of real-time services with high data processing, maximum capacity and scalability. Fogcomputing gives the best solutions for such an environment because of its signiﬁcant beneﬁts overcloud computing. Extension of cloud computing, aims to minimize the burden on the cloud byintroducing network edge computing concept.For industrial automation, real-time services and decision making processes require low latencyand enhanced cache memory. Required performance parameters are mobility, real time applications,low-latency, location-awareness, number of nodes and cache-enabled edge devices on this basis ofgeographical distribution. Virtualized nodes frequently known as cloudlets or fog nodes are placedbetween clouds of internet and end user devices. Fog computing provides services and applications asa cloud does with better QoS parameters performance covering critical requirements of IIoT. Importantadvantages of fog computing that inﬂuence its use for IIoT are:• Data storage on network edge nodes eliminates the transmission delay by removing the need foraccessing data from far-away clouds.• Fog computing supports to process and analyze the data on faster speed for IIoT applications.• Data storage on edge nodes will reduce the processing and computing delay.• Cache enabled nodes will prevent transmission of irrelevant information over the network.• Can give support to all IoT applications e.g., smart grids, smart cities, D2D, Vehicular Ad-hocnetworks (VANETS) using edge networking concept.• Provides ﬁltered and required interaction between end devices and cloud service providers.Fog computing is the building stone to provide solutions for more efﬁcient, effective andmanageable communication way for the massive number of smart IoT devices in the near future. Fogcomputing with extra features as compared to cloud computing in terms of latency, security, locationawareness, location, and number of server nodes, real-time connectivity and mobility is a promisingenabler for industrial automation.2.5.8. Edge ComputingIntroduction of enormous smart devices making an industrial revolution in all domains, causesextensive data processing, computation and burden of trafﬁc on a single server either a cloud serveror cloudlet. This motivates researchers to develop a new computing technology named as edgecomputing. The idea is to develop embedded automation controllers on devices named as edge-node(e-node) in the literature. This device is intelligent, with low processing power, better hardwaresecurity. Edge computing is an extending form of previous fog and cloud computing technologies.It comprises peer-to-peer networking, self-organizing network, and remotely manageable server.It gives following advantages:• Encourages real-time connectivity.• Overall network trafﬁc reduces, as some computation is done on the edge of the network.• Enhances security by encryption of data near to the network core.• Optimize the resource usage.IIoT applications have critical communication requirements. Cloud computing, fog computing,and edge computing platforms need to be optimized for better, efﬁcient results. Cloud computing canbe used where there is no high requirement of real-time connections, privacy, and security. On a localarea network, fog computing uses a centralized system which interacts between the network andcloud server, whereas edge computing does computation on embedded systems of the network.Edge computing has direct interaction with sensors and actuators. The need for cloud, fog and edgecomputing architectures is increased with the growth of the IIoT application. To increase the use ofIIoT smart devices, researchers are focusing on fog or edge computing paradigms which results inindustrial development.11Sensors 2019, 19, 48073. Industrial Internet of Things Applications and RequirementsInformation about the occurrence of faults, components, inventory, different demands anddifferent orders continuously needs to be shared among smart devices/processes resulting in improvedefﬁciency, tracking, capacity use, quality of production and development in industries. From IIoTperspective; smart cities, smart factories, and smart products are important IIoT beneﬁcial examples.The basic three-layer architecture of IoT as discussed in Section 2.3, need to be evolved according tothe requirements of speciﬁc IIoT applications.3.1. IIoT-ApplicationsFrom the identiﬁcation of faults to solutions via communication and networking technologies,every step needs to be optimized and has research potential. Possessing attributes of intelligence,reliability, safety, sustainability, privacy, and efﬁciency; these applications are called smart in theliterature. This revolution results in the development of new infrastructure. According to [37], smartcity, smart factory and smart product are the main applications of industry 4.0.3.1.1. Smart City ApplicationsAbout 6 billion people are expected to be part of cities of the earth in 2050 [38]. With the adventof technology infrastructure, this increase in population will result in more data origination anddemand for services. This big data origination is called Big Data. To develop future smart city,there are many domains that need to be intelligent, such as the smart home, smart ofﬁce, smartinstitution, smart health-care centers, smart agriculture, and smart transportation. All these domainshave different IIoT applications with different requirements. The development policy of a smart cityhas six factors namely, smart economy, smart mobility, smart environment, smart people, smart living,and smart governance [2]. Numerous research has been carried out in the domain of smart cities. Forinstance, [39] presents a framework with which, smart cities can overcome current limitations. Thissmart city transformation will take time.3.1.2. Smart Factory ApplicationsSmart industry comprises distributed automated systems and robotics. This future smart factory flooris possible using ML algorithms and AI technology. These automated devices are integrated with sensors,actuators, microchips, autonomous systems, and controllers. Relying upon CPS and IoT technologiesfor evolution industrial processes required ultra-reliable and low-latency communication (URLLC).For monitoring, managing and controlling such environment, IoT nodes can handle bounded latency ofmillisecond scale. These applications are characterized using latency, jitter, energy consumption, workloadparameters. M2M and D2D are emerging supporting technologies for smart factory development. In thesmart factory manufacturing process, machines will have high-level of automation and self-optimizationattributes. These attributes will fulfill complex requirements of products. This has open issues regardingnetwork communication technologies, number of devices, security, and cost.3.1.3. Smart Product ApplicationsIoT, cloud computing, big data, cloud computing and production time are drivers of industry 4.0development. The products in industry 4.0 are smart because they are integrated with sensors andmicrochips. Existing production systems need to be integratedwith industry 4.0 architecture (IoT+CPS+WSN).This integration will allow communication interaction between human beings and products [40].3.2. IIoT Application Design ParametersTo the increasing demand of customers and market requirements, the manufacturing industryis now facing problems in achieving desired goals. Industry 4.0 came up with an innovative idea ofautomation inside in the industry increasing the production process ﬂexibly. Industry 4.0 key point is12Sensors 2019, 19, 4807M2M communication, in which machines communicate with each other over the Internet. Developingand manufacturing industry, communication of intelligent machines with each other using differenttechnologies depending on the coverage area, give rise to high production and self-regulation of themanufacturing process in an industry. These are the goals of industry 4.0; all this is assured by ITsystems as they provide enabling smart technologies. The integration of these heterogeneous devicesin a network with other devices and existing communication technologies is also the main requirementfor designers. Every IIoT application has critical design goal requirements to improve QoS in providingsolutions. These design parameters are:• Energy & Long Battery Life: Overall network energy should be preserved for better and efﬁcientoutcomes. Smart devices should have enough battery storage so that they can use for long time.• Latency: Some IIoT applications are time-sensitive, a bound should be there to limit all types ofdelays including processing, propagation, transmission, and computation.• Throughput: Amount of data for processing is different for different applications. It shouldsatisfy the application requirement.• Network Topology: How the number of servers (cloud, fog, e-node) and smart devices are placedin a network for better QoS requirements.• Reliability: Solutions by IIoT applications demand reliable real-time connectivity.• Security, Safety & Privacy: These are very demanding and major requirements for all IIoTapplications. For example, inside a smart factory there should be privacy and security such that noone can access the private information. For healthcare applications, patient’s information shouldbe safe and not easily accessible and changeable. 3A’s; Authentication, Access, and Authorizationare steps involved in the strictly secure system. The demand of end to end communication in IIoTapplications requires privacy of data as well. Sensors and actuators should be safe from intrudersas well as environmental hazards.• Low Cost: Smart devices used for IIoT applications should be low cost so that doesn′t affect theCAPEX/OPEX. Deployment involved in industry 4.0 should not be so much that will cause lossin marketplace.• Long Coverage: A device should be capable enough to cover the desired range.• Standardization: So far, there is no such network standardization and is an open challengefor researchers.• Integration: IIoT applications are composed of heterogeneous devices and hybrid networks, thereare a lot of issues in integration.• Communication/Enabling Technology: Communication technology for supporting IIoTapplication should provide assured performance services.• Device Maintenance: Heterogeneous device in an industry 4.0 environment, require constantdevice management as devices are connected with each other and the Internet. Software DeﬁnedNetworking (SDN) is used for such failure and changing maintenance issues of devices.• Monitoring Network: Wireless, environmental and mobility nature may cause a change innetwork topology which requires the system to be monitored and managed frequently.• Conﬁguration & Management of System: Self-conﬁgurable, self-control, reconﬁgurationfunctionality in addition of new devices in network.• Trafﬁc congestion & Overload: Smart devices will be increased with time in any IIoT application.System should be able to adjust according to the trafﬁc burden and data requirement.• Mobility: IIoT applications, such as transportation, inside industry and healthcare devices, havethe property of mobility from one place to another.• Scalability: Scalability brings many issues, some are: How many numbers of smart devices areenough to support an industrial application environment? or how many devices are served by aserver easily? how to optimally design a system under energy/spectrum issues?13Sensors 2019, 19, 4807• Heterogeneity & Interoperability: Heterogeneous smart IIoT devices have to communicate andcollect information among themselves and the Internet. This integration is an issue to solve.Standardization is required for interoperability of IIoT devices.• Performance: There is always a performance trade-off among these QoS requirements. Thereshould be an optimized, supportive, and efﬁcient trade-off among the factors affectingperformance. Performance maintenance solutions are required for future automation.3.3. IIoT Applications in Relation to Cloud, Fog and Edge ComputingAll IIoT applications require critical QoS parameters in order to produce beneﬁts in every ﬁeld.Fog term used in computer science is an extension of cloud architecture which offers services ofcloud to edge devices. It is seen as a new cloud or it will replace the cloud in future but it is just anarchitecture that complements cloud architecture in order to provide solutions for critical applications.Entry points in any network are called edge devices. Entry points are part of second and third layer,it has hardware devices named as switches, routers and WAN devices. Cloud is a centralized solutionand fog brings solutions at a distributed level by bringing data storage and its computation near theedge of a network. It allows getting services in the proximity of IoT devices. Fog combines servicesprovided by cloud and IIoT applications; or it enables IIoT applications.Cloud architecture is efﬁcient, beneﬁcial and provides solutions. It offers services after storingdata in remote centers from the Internet. These remote data centers face less delay and computationas compared to internet. It saves the cost of physical resources, makes connections more reliableand results in an increase in efﬁciency and performance. It is a ﬂexible, innovative framework in thenetworking ﬁeld. It helps in accessing resources anywhere anytime. Question is why fog architectureis needed when cloud is already the best solution to many issues? There are certain issues andrequirements of IoT applications and fog can provide solutions. First and foremost, the low-latencyrequirement of IIoT applications in every ﬁeld can be attained effectively. It stores data in the proximityof users. Propagation delay between cloud and users will be reduced; computation delay due to hugedata trafﬁc at cloud can be reduced. It supports time-sensitive tasks effectively. Second IoT challengewas using network bandwidth in an optimized manner, fog being in the center of cloud servers andend devices. It helps in less usage of bandwidth; data doesn′t have to travel fog-to-cloud distance.Popular content is available at the network edge. This will also result in minimizing cost, lifespan ofdevices, energy consumption and complexity during every demand which goes on the users-to-cloudpath. Security issues can be seen on the fog server; it can act as a proxy-server controller. Privacy andsafety of data is another important IIoT devices requirement. Fog helps in monitoring such tasks.Exponential increase in IIoT applications in every ﬁeld making world a cutting edge technologyparadigm. Figure 7 shows both these architectures; cloud computing in which IoT devices are directlyconnected to data centers and cloud server and fog computing in which fog server is in the middle ofcloud and IoT devices. Fog works on network edge which improves speed, computing capabilities andprovides distributed and better solutions. Fog has complemented the cloud architecture in many ways.Cloud is a centralized solution while fog can work in both a centralized and decentralized manner.Over large geographical area a group of fog nodes can be monitored and managed in a centralized way.Cloud size is large as compared to fog, as it has massive storage of data from the Internet. Fog size is aﬂexible parameter that can be altered according to the demand of users. For example, for a vehicle tinysized fog would be enough while for an institution many small fogs can work in a form of network ormay be a large-sized fog would be sufﬁcient. Fog has fewer deployment complexities as compared tothe cloud. Similarly, cloud management is tricky and more time taking as compared to a fog becauseof its ﬂexibility. This ﬂexibility will support mobility in networks.14Sensors 2019, 19, 4807Figure7.Industry4.0,IIoTapplicationsversuscloudandfogcomputing.15Sensors 2019, 19, 4807Because of architecture complexity, the cloud can only be operated by technical experts whilefog can be managed and operated by little human effort. Large companies control cloud networks,while fog can be controlled by small as well as large companies. Low-latency critical IIoT applicationswill be handled by fog, while applications that can tolerate large delays leverage the cloud services.Internet connection between user IoT devices and a cloud server should be reliable for the entire timeof connection, while in fog architecture connection with cloud is not necessarily required for the wholetime. Bandwidth requirement increases for both frameworks as the number of IoT devices is increasingexponentially. It provides awareness property, which means it knows about the requirements ofcustomers and will provide solutions accordingly. It can be placed anywhere between the cloud anduser nodes according to the demands. Performance and efﬁciency parameters of IIoT applicationservices can be enhanced using fog framework. In addition, the scalability issue can be handledusing it. Data storage at network edge will result in minimization of service delays and supportsreal-time processing. The big amount of data produced from heterogeneous smart devices requireshuge data storage and computation at the cloud server. IIoT applications requires data storage attheir backbone. This big data is giving rise to edge computing for future. Edge computing is alsocalled Mobile Edge Computing (MEC). This technology supports IIoT applications by building betteroperational connectivity. It brings cloud computing capabilities at the devices that are present on theedge of a network, these devices named as edge servers or edge devices. The edge devices are part ofthe Internet and participate in processing and computation near the data location. This technology isan industrial initiative by the European Telecommunication Standards Institute (ETSI) [41].4. Protocols/AlgorithmsProtocol is a basic set of rules that deﬁnes how communication happen between different devicesin a network. Protocols have to be devised intelligently in order to achieve the deﬁned goals.As the future is industrial automation in industry 4.0 revolutionized era, there is an exponentialincrease in smart, intelligent devices in IIoT applications. A major requirement is that the emergingnetwork protocols must meet required goals in performance affecting parameters, such as energyefﬁciency, latency minimization, spectrum efﬁciency, cache memory maximization, and bandwidthuse requirements. A summarized discussion on pre-existing protocols in the context of fog computingproposed by researchers is given below in following subsections.4.1. RoutingDong et al. [42] have introduced redundant fog loops for WSNs. The proposed fog loop-basedscheme has two main steps. Creation of fogs using loop paths is the ﬁrst step, while the secondmechanism creates fog nodes in the source node areas along with many other interfering fogs withinthe network. This proposed scheme has helped in ﬁnding the exact location of the source nodein terms of energy efﬁciency and privacy. Results were compared to the efﬁciency offered by thePhantom Routing Scheme (PRS). The proposed scheme gives improved efﬁciency by 4 folds and canalso improve the privacy and security up to 8 folds.Since fog computing lowers latency and offers energy saving, they are tailor-made for dealingwith WSNs [43]. Sensors in WSNs are resource-constrained, therefore energy efﬁciency is an importantissue. It needs to be addressed for the network to increase network life-time of operation and workingefﬁciently for a prolonged period of time. Sensors in a form of clusters collected data and send to thebase station using energy-efﬁcient routing protocols. In this approach, using multi-hop communication,data is transferred to the sink/destination node. The nodes acting as a cluster head are used formulti-hop communications. For networking, apart from the routing problem, another issue thatneeds to be catered is increasing network lifetime. Network lifetime can be effectively increased byoptimizing energy and power consumption at nodes. Some examples of these routing protocols areLow-Energy-Adaptive-Clustering-Hierarchy (LEACH) and Stable-Election-Protocol (SEP). LEACHprotocol involves Deterministic Cluster Selection Head and abbreviated as LEACH-DCHS [44].16Sensors 2019, 19, 4807Handy et al. [45] have proposed LEACH in which rotation of the cluster heads in a randomizedmanner is used for energy distribution to the nodes evenly. These algorithms haven′t been used forfog computing yet, but can be used in future incorporating fog computing. SEP was proposed bySmaragdakis et al. [46], prolongs the period of stability of wireless sensor networks. A modiﬁedversion of SEP (M-SEP) proposed by Singh et al. [47] is suitable for heterogeneous WSNs. Some ofthe nodes in this approach might have a greater chance of getting selected as a cluster head and havemore energy. Moreover, in large WSNs, the data to be processed from heterogeneous devices is inlarge volumes. Processing time is signiﬁcantly large, so an alternate approach is to use fog computing.The time required to process large amount of data gets considerably reduced using fog as the sensors inthe network get interconnected with the Internet in order to make smart by making them autonomousin making decisions.Another routing protocol that is energy-efﬁcient named as new-SEP is proposed byNaranjo et al. [48], prolongs the stability of sensor networks supported by fog more than SEP.Optimal clustering nodes are elected by considering various sensor node features such as the ratioof heterogeneity in the network, residual energy, distance between cluster heads. Results werecompared with baseline schemes of LEACH and SEP, the proposed scheme performs better in termsof energy-preservation and network lifetime. Considering increasing the network’s stability periods,N-SEP performs better than LEACH (50 percent) and SEP (25 percent). An efﬁcient route optimizationalgorithm was presented in [49] to address the mobility control issue in fog-based SDN networks.The proposed SDN-enabled fog computing architecture had three-layered structure namely, fog layer,network layer and application layer. Results showed that there is a great improvement in networkperformance. A three-layered plane architecture was proposed to generate efﬁcient routing pathsby the authors in [50] for data-center-based heterogeneous networks, using tensor decompositionmethods. These three planes do three different tasks, edge plane considers the trafﬁc, bandwidth anddelay requirements; fog planes computes and controls the available paths and ﬁnally cloud plane dothe routing.4.2. Resource AllocationOne way to meet the growing IIoT application requirements is to use fog computing. Interactionsin real time takes place in fog application rather than batch processing. Services supported by foginclude mobility, heterogeneity, working with cloud to extend cloud services, user optimization,etc. Resource allocation or resource management in any network is one of the biggest challenges.This deﬁnes new protocols for networking and communication. Many researchers have donework in this context and already proposed some protocols. Resource allocation was done witha speciﬁc deﬁned objective, it can be maximizing energy efﬁciency or throughput; minimizinglatency or power consumption or network cost. A joint min-max optimization problem of resourceallocation and ofﬂoading decision making was proposed by the authors in [51]. They have proposedComputation-Ofﬂoading-Decision-Making and Resource-Allocation Algorithm (CORA) to minimizethe maximal network cost (delay & energy consumption). Results were explained using fractionalprogramming theory and lagrangian dual decomposition. Fog Radio Access Networks (F-RAN),an extension of the Services provided by Cloud Radio Access Network (C-RAN), has gathered attentionglobally and several advantages can be taken by providing functions of baseband signal processingnear the edge or making the edge devices cache-enabled. F-RAN is the best example in which we areusing edge devices as well as network devices (access points) as fog nodes to achieve the best results.Moreover, due to the cooperative communication, beneﬁts of C-RAN such as enhanced spectral andenergy efﬁciency are also conserved in F-RAN. Resource allocation is a challenge for the upcomingF-RAN. Authors in [52] have proposed a Stackelberg equilibrium (SE) for a hierarchical problemof network slicing customization between global radio resource manager (GRRM) and local radioresource managers (LRRMs). A game is formulated to alleviate the burdens in GRRM and LRRM;GRRM assigned resources to each slice and then LRRM in every slice provides resources to UEs. UEs17Sensors 2019, 19, 4807could be divided into clusters on basis of respective objective functions and accordingly get resources.Authors have provide two algorithms for LRRM 1 slice and LRRM 2 slice with an objective function ofmaximizing high data rate and minimizing latency, respectively.To meet low latency, high throughput and connectivity requirements in future RANs,non-orthogonal multiple access (NOMA) is a promising technique. Zhang et al. in [53] have formulateda problem for maximizing the net utility under interference constraint for resource allocation problemfor NOMA-based FRAN system model. Results were compared with the conventional orthogonalfrequency division modulation (OFDM) technique. Madsen and Albeanu [54] discuss platformsinvolving fog computing in which there is communication going on between the smart devices,fog, and cloud. A model for internet applications in future is presented by Hong et al. [55] inwhich applications are delay-sensitive and distributed geographically. For improving the renderingperformance of a webpage, Zu et al. [56] exploit information available only at the network’s edge.Companies that want to deliver content such as Netﬂix uses fog computing to reach their geographicallydistributed customers. As indicated by [57], ensuring signiﬁcantly large streamed data to be deliveredin the proximity of the end-user (customer), is done using fog computing.Minimizing the energy consumed in geographically distributed applications and resourceallocation using fog computing is discussed in [58] for video services. Size of fog nodes deploymentshows the application demand in that region. To maximize social welfare, some of the user data needsto be controlled. Optimization on large scale is also possible using proximal and distributed algorithms.The algorithm proposed gives a near-optimal solution.4.3. Load BalancingWhen one edge device has to do a lot of work e.g., computing tasks all by itself, it consumesresources. However, this can be reduced with the help of distributed architecture in which load isevenly distributed among the edge devices present in the network. This distribution of load is calledload balancing. Load Balancing in combination with fog computing makes a formidable combination.A fog network can provide a platform for cooperation and coordination between edge devices in aload-balanced network. Authors have proposed a load balancing ofﬂoading algorithm for latencyminimization in Vehicular fog computing (VFC). VFC is an integration of vehicular networks and fogcomputing technology, which is an efﬁcient ﬁeld to achieve real-time and location-aware for vehiclesin a smart city. Zhaolong et al. in [59] has formulated a response time minimization problem using athree-layered decentralized network system model to balance the trafﬁc among vehicles. These threelayers are cloud layer with a high computing cloud data center, cloudlet layer processed the receiveddata from vehicles before sending to the cloud center. Lower layer has vehicle clusters, clusters aremade for trafﬁc balancing. The fog-vehicle interface manages and alleviates the trafﬁc load.Trafﬁc overhead results in inefﬁcient resource management, which is also a challenge to besolved for healthcare applications. WSNs based health monitoring systems have become a convenientchoice as elderly people can frequently require health services. The basic requirements for healthcareapplications are energy efﬁciency, high-response time, low-latency and real-time connection. Fogcomputing can be considered to be an important enabling technology for such time-sensitiveapplications. Authors in [60] provide a very signiﬁcant critical review of existing solutions provided byresearchers in the healthcare domain using fog computing. The authors in [61] proposed a fog-cloudhybrid solution to load-balancing problem. If a client’s requirement is more critical, it will be handledby cloud otherwise servicing is done by foglets. Results shows that network utility can be enhancedin terms of latency and load balancing. They used iFogSim tool for experiments. Forough et al. [62]proposed an Energy Balancing Algorithm (EBA) for Fog-IoT networks to reduce delay and energyconsumption. The authors have proposed two optimization problems, ﬁrst one is to ﬁnd an optimaltransmission rate and power for terminal nodes (TNs) and the second one is to ﬁnd an efﬁcienttopology between TNs and fog nodes (FNs). The system model was designed under the constraint ofchannel conditions between TN and corresponding FN. A major challenge in edge computing is the18Sensors 2019, 19, 4807inefﬁcient deployment of resources as it reduces the overall efﬁciency of the network. This results indeﬁnite increase in power consumption. A low service block is preferred for maintaining lower latency,which will otherwise be detrimental to the high-performance requirements of edge computing. Unlikecloud computing, fog computing has a limitation of resources, which can be overcome if we allowcooperation between various data centers. A cooperative scheme by Beraldi et al. [63] is proposed inwhich data centers near the edge exchange processing requests and shares the load of highly loadeddata centers. The request arriving at a busy data center is forwarded to any other data center havingthe request buffer partially ﬁlled. The proposed scheme maintains a threshold of requests with thehelp of Markov Chains to make sure that load is equally distributed among all the data centers. Thisleads to much-improved performance as lightly loaded data centers can absorb the burden of heavilyloaded ones during peak hours.Full use of edge resources cannot be done by cloud computing polymerization calculation [64].The edge devices are not part of the cloud computing, which is undesirable for delay-sensitiverequirements. Ningning et al. [65] have explored how fog computing can turn the nodes or edgesinto virtual machines using Cloud Atomization Technology to improve on this problem. The authorsuse graph partitioning for developing an efﬁcient load balancing algorithm. Consequently, a ﬂexiblenetwork can be built by making fog networking possible after atomizing the cloud and eventuallyreduce the cost of the system that were high before implementing the load balancing algorithm.Deng et al. [66] investigate power consumption and transmission delay trade-off. Problem of workloadallocation is formulated to obtain the minimal power consumption subject to the constraint of servicedelay. Three sub-problems result after the decomposition of the primal problem. Based on theobtained results, it is shown that cloud performance is enhanced by using fog computing. Moreover,latency is reduced and bandwidth can be saved by sacriﬁcing some of the computational resources.Computational clusters are formed on the basis of cooperation behavior between the Small Cells(SC) to share computational resources. However, the cooperation is dependent upon many factorssuch as resource availability, resource allocation, delay constraints of the application, distribution ofcomputational load and size of the cluster. The joint distribution of resources for the mobile end-usersand the cloud is the main objective problem of this framework.All the data that is being frequently accessed by the edge devices is stored at the Radio Units (RU),which considerably decreases the overall delay in the network of Fog-Radio Access Networks (F-RAN).However, in such systems, the energy efﬁciency aspect has always been a matter of serious concerndue to the addition of extra smart components in the system. In [67], the authors have proposed anovel scheme by designing a green network in which an efﬁcient algorithm is incorporated to optimizethe selection of RU. Furthermore, the algorithm also jointly optimizes the formation of clusteringand beamforming while maintaining the QoS and balancing the load of each of active backhaul as ameasure of its capacity. While copying data from the database, edge devices may interfere with eachother. To avoid this mishap, data replication techniques are used for copying data electronically fromthe main database where the data of all the users is hosted. Uniform distribution of data and processingis crucial over the network which in turn helps in managing the large amount of data and workloadwith efﬁciency. Fog computing not only helps in achieving higher efﬁciency, it also helps in balancingload across distributed platforms and achieving higher energy efﬁciency due to fewer performancebottlenecks. Verma et al. [68] have focused on making a network that is less dependent on cloudcomputing and bringing the storage and processing capabilities near the edge devices. The results aresimulated with the help of CloudSim by testing different geographically separated servers and theirconﬁgurations and then make comparisons between cloud and fog computing for various attributes.5. Challenges with SolutionsIn general, there are many challenges towards industry 4.0 digital transformation. For M2Mcommunication, reliable and stable connectivity with bounded delays is a mandatory requirement.Real-time communication is on the higher priority for this fourth revolution, which brings many19Sensors 2019, 19, 4807technical challenges for future network development. A well-designed network architecture canincrease the sustainability and performance of the entire system. The journey towards industry 4.0is on its way and until now there are no such standards, regulations, and certiﬁcation to followthem.Although fog becoming a developing key enabling technology for IoT architecture, it still facessome issues while integrating into the current architecture. Complex software applications andsolutions are needed to achieve an efﬁcient fog network. Fog network will be analyzed in termsof key performance measurements, which are bandwidth use, energy consumption, low latency,maximum throughput, and resource management. In this section, critical technical communicationand networking challenges in the context of fog computing for IIoT applications are listed. All of themhave potential for future work.• Power Consumption/Energy Efﬁciency: Several smart devices supporting an IIoT applicationwill consume a massive amount of energy on a different scale according to their requirements.Ensuring network QoS with minimum energy consumption of smart IoT devices, fog nodes andcloud in an optimized way is an open challenge for every upcoming future IIoT application.• Throughput/Rate/Capacity: Throughput or network bandwidth, data rate and storage capacitydepends on how much data is used and where data is stored in a fog network. This data placementon fog nodes or edge devices or cloud server has effects on cost, delays, bandwidth, and networkcoverage. The optimal placement of data on cloud server or fog cloudlet is one of the criticaltechnical challenges for fog-IoT architecture.• Spectrum Use/Resource Allocation: Geographically separated fog, cloud nodes and theirinterconnection makes the backbone of any network that relies on ofﬂoading services. Mostof the cloud computing interconnection mechanisms are not enough for fog networking due totheir limitations including relying on a centralized cloud which cannot fulﬁll the latency andlocation awareness requirements of distributed devices, etc. Fog computing must encompassfeatures, such as multi-tenancy, scalability, heterogeneity and quick resource provisioning.An architecture including fog and cloud computing must meet all these requirements for whichresource allocation/use is the most critical challenge for better network performance. It has effectson all other QoS parameters.• Latency: IIoT applications requirement is real-time connectivity. All applications aretime-sensitive and require real-time streaming rather than batch processing. Fog computing givesa better result for such decentralized solutions. It gives low latency with reliable connectivityand mobility. Optimized placement of data centers, resource allocation, network architecture,energy consumption of nodes, and storage capacity of nodes have impact on latency. Latency fora network is the sum of transmission, processing, propagation and queuing delays. To achievethe low-latency requirement, there is need mitigate all types of delays.• Cache Enabled Edge Devices: Caching content locally, reduces the access delay time andincreases the energy and spectral usage efﬁciency. Since the Internet has multiple bottleneckswhile accessing data from across continents and oceans, caching does not have to be dependent onany of these bottlenecks and instead makes the same data available locally. Furthermore, cachingincredibly reduces the load on backhaul links since they do not have to be used anymore foraccessing data. Since all the users have to access the data from the same centralized location(internet/cloud server), a certain degree of fairness is needed to avoid inefﬁciency in accessingdata. Since all the backhauls have certain capacity constraints, there is a need for an efﬁcient loadbalancing mechanism to overcome this issue.Further, we discuss existing solutions to these mentioned challenges.5.1. Power Consumption/Energy EfﬁciencyObjects are extensively being connected together using the IoT technologies. Heterogeneous smartobjects, in the context of hardware and software, can perform efﬁciently in the availability of memory20Sensors 2019, 19, 4807and high computational power. Network’s complexity increases day by day, because of scalabilityissues of smart objects supporting IIoT applications.Virtualization has played a very crucial role in improving the overall efﬁciency of the data centersand now research has also been done on how virtualization can help in fog networks. Virtualizationmakes it easiest to deploy fog functionalities on an existing node (by isolating and securing fog servicesin a virtual machine or container). It also helps in conserving energy by efﬁciently consolidatingthe tasks on a single fog node. A comparison in terms of power consumption was studied in [69]between C-RAN and F-RAN using network function virtualization (NFV) technology. The authorshave formulated a mixed integer linear programming (MILP) problem, results for F-RAN are 30%more improved in terms of power-saving as compared to C-RAN.Roca et al. [70] have developed a platform using Fog-Function Virtualization, which worksand builds on the concept of Network Function Virtualization (NFV) for multiple IoT applications.This, further with the help of node constellation creation, helps in easy deployment and reducesthe cost considerably as less energy is required for running the system due to efﬁciently performingvirtualization. Task scheduling is necessary as it helps with the load balancing aspect of networkingand can provide services to multiple users. Cooperative games between the containers and brokers arestudied for energy-efﬁcient task selection algorithm. Kaur et al. [71] have achieved efﬁciency withthe help of container-as-a-service (CaaS). Lightweight containers have been used which considerablyreduce the energy consumption by a container migration techniques. This kind of virtualization ismore cost-efﬁcient for distributed architecture, where a large number of devices with different runningapplications/processes can be allocated to resources efﬁciently. Results achieved by the authors provethat the system is more energy-efﬁcient.Graph-Based Heuristic algorithms were proposed by the authors in [72]. Type of problem is theinteger linear programming (ILP) problem, the objective is to increase the energy efﬁciency underassociation and capacity threshold constraints for hybrid cloud-fog RAN (CF-RAN) architecture.Minimization of latency and power consumption is also addressed in the proposed system model.Fog computing along with NFV gives excellent outcomes in terms of reducing latency and powerconsumption. Energy can be saved by incorporating techniques, such as Message Queue TelemetryTransport (MQTT) in a fog-based environment [22]. In this scheme, the number of transmissions isreduced to save energy of the end devices. Using energy-efﬁcient routing protocols is imperativeto achieve energy efﬁciency. MQTT supports sensor data in real time due to its many-to-manycommunication nature. The concept of MQTT focuses on introducing another layer between the fogand cloud with lower complexity. MQTT broker place is at the fog layer. The intermediate layer isresponsible for predicting the future measurements, and acts as a gateway for the upper layer. It helpsto ofﬂoad the computationally expensive tasks from the cloud to save in the storage memory of thefog server. This results in a reduced number of transmissions as the update only occurs in case of amismatch.5.2. Throughput/Rate/CapacityFor network designing, a new paradigm known as Socially-Aware-Networking (SAN) has been ofmajor interest [73]. To achieve efﬁcient performance, SAN brings the human behavior and CPStogether via intelligently designing of a network. This design should be adaptable as well forall environments. The resources available to mobile devices differ depending on the models andspeciﬁcations. This resource availability results in a group of mobile devices that might be sufﬁcientin terms of processing and storage parameters. Group of some might not be self-sufﬁcient. The bestsolution to this problem is given by SAN and Fog-Radio Access Networking (F-RAN).D2D communication comprises the direct sharing of contents among mobile devices. Directsharing is a key feature supporting D2D communication. For achieving efﬁcient performance results,an imperative design of network embedded with all technologies is required. Research has shown thatthe system performance in terms of utility, throughput and energy efﬁciency is maximized using the21Sensors 2019, 19, 4807download mode with the help of branch and bound algorithm [74]. Klas et al. [75] and Cau et al. [76]worked on improving network efﬁciency Mobile Edge Computing (MEC). MEC is a domain underfog and focuses on providing cloud computing capabilities near the edge of network. Efforts andimprovements have been made in MEC in order for it to support 5G communications. Another targetof MEC is to achieve access to information provided by the radio network for application developmentand distribution of the content. It is predicted that data trafﬁc over mobile devices will increasemanifold in the next few years. Such increasing needs must be satisﬁed using efﬁcient mechanisms.A reliable service with an extremely low latency (URLLC) and high capacity is the foremostrequirement of IIoT networks. Throughput maximization of the F-RAN system is compared with threedifferent back-haul strategies for Small Cell Networks (SCN) in [77]. All these strategies namely, decodeand forward, direct transmission and C-RAN were studied under delay threshold, rate constraint, andbackhaul and fronthaul links. The authors proposed iterative algorithms for all strategies. For F-RAN,Pontois et al. in [78] formulated a non-convex optimization problem under fronthaul constraints. Ahybrid semi-distributed resource allocation algorithm was proposed by the authors for the proposedweighted sum-rate maximization problem. Results show that there is a trade-off between maximumthroughput and system latency. A multi-objective optimization problem was proposed by the authorsin [79]. They have proposed three parallel algorithms to improve latency, throughput and resourcemanagement. A queuing model was studied under task buffering, ofﬂoading and resource allocationalgorithms. Lyapunov drift was used by the authors to design the resource allocation policy. Inresults for better system performance, trade-off between latency and throughput is observed. [80]end-to-end performance is guaranteed after composition of problem as Multi-Constrained OptimalPath (MCOP). The authors propose a solution from the network architecture’s perspective and cloudservice relation. The proposed algorithm provides better results in terms of efﬁciency and effectiveness.QoS parameters (capacity, delay and cost) gets improved by the proposed network-cloud serviceprovisioning system model.5.3. Spectrum Use/Resource AllocationFog networking can provide a solution for resource management issue in future 5G networks.With a growing number of smart devices, the major issue is of spectrum use and resource allocation.Spectrum pricing and allocation scheme (SPAS) was proposed in [81] for F-RAN framework. Threeareas were deﬁned for the whole network, and double game theory was applied to give the solutionfor efﬁcient spectrum use. Proposed two algorithms are named as game-model (GM-SPAS) andmultiple-spectrum-reuse-technologies (MSRT-SPAS). Their presented results are effective in termsof revenue and efﬁciency. The problem of resource allocation with utility maximization objectiveunder QoS constraints was formulated in [82]. Analytic Hierarchy Process (AHP) was proposed by theauthors for ﬁnding the QoS requirements of IoT devices. Afterward, the matching theory was appliedfor user association.In [83], the authors have used the calculus of variation to ﬁnd out the optimal spatial density ofnodes such that the distribution of nodes can support the whole network. A Parallel ImperialisticCompetitive Algorithm (PICA) is used to determine the initial positions of the access points. WSNswithimmense scalability, immobility factor and low-deployment cost properties have limited processingand are resource-constrained. Smart Mobile Access Point (SMAP), which has been proposed byMajd et al. [84] aims at achieving higher resource use and energy efﬁciency with the adoption ofhierarchical placement of SMAP in WSNs using fog computing. Moreno-Vozmediano et al. [85] havepresented a framework for interconnection of fog and cloud computing, Hybrid Fog and Cloud (HFC),that provides effective, simple and productive resource provisioning. It also automates the processof conﬁguring various virtual networks that are part of the network for interconnecting importantcomponents. The proposed architecture covers the salient features, such as security and scalabilityalong with fog to fog, and fog to cloud communication mechanism.22Sensors 2019, 19, 4807Shojafar et al. [86] propose a resource scheduler for Networked Fog Centers (NetFCs) thatnot only provides services to the vehicular client but is also energy efﬁcient. NetFCs operate fromvehicular network’s edge via Infrastructure to Vehicle (I2V) mobile links to the vehicular clients thatare being served. Not only does the overall computation and communication energy efﬁciency getmaximized but also the network performs better in terms of QoS requirements. Moreover, hardQoS requirements induced by the application are met, such as reducing transmission rates, delay,and jitter. Resource scheduler is responsible for admission control, dispatching the allowed trafﬁcusing minimum energy and adaptively controlling the trafﬁc that is being injected into the mobileconnections. Few of the important characteristics of the scheduler include providing QoS guaranteesinduced by the application, implantation of the scheduler is both scalable and distributive.5.4. LatencyFog networking is a key paradigm to provide solutions to latency-sensitive future IIoT applications.Many researchers have contributed to cater latency minimization problem. Online Fog NetworkFormation Algorithm was proposed by Gilsoo et al. in [87] for minimizing the overall maximumlatency (communication and processing) of a fog network. The objective problem constitutes of thesum of two types of delays i.e., fog network formation and task distribution. A joint energy andlatency optimization (JELO) scheme for F-RAN was proposed in [88]. The joint optimization ofenergy consumption and latency was formulated as an integer-programming (IP) problem subject touser association, capacity and latency threshold. The proposed complex problem was divided intotwo sub-problems of knapsack and semi-assignment problem. The proposed algorithm gives betterresults as compared to the existing techniques in the literature. A latency minimization problem wasformulated for IoT-fog network under user association, workload and latency threshold constraints.The authors have proposed a matching-game theory for the proposed resource management problemfor network latency minimization [89].Over the last few years, the Internet of Vehicles (IoV) has been a matter of growing interest.Cloud computing provides high performing IoT services to IoV, still, there are many shortcomingswhen it comes to mobility support, latency and location awareness. Xiuli Hi et al. [90] haveintegrated fog computing into SDN. While fog Computing helps with the latency of the network, SDNprovides ﬂexibility in the centralized control and providing the complete global knowledge of thenetwork. Vehicle to Vehicle (V2V), Vehicle to Infrastructure (V2I) and Vehicle to Base station (V2B)communication can be supported by a weighted undirected graph having Road Side Units (RSUs).The simulation results given by the aforementioned authors decrease the latency and achieve a higherQoS. The scheme proposed by Skarlat et al. [91] tackles an optimization problem, providing a solutionthat there is no trade-off between communication and using resources and energy for computation.The applied system model demonstrates that in optimization scenarios, reduced average round-triptime (RTT) and delays up to 39 percent can be achieved using fog. The scheme also considers anindependent cooperative number of working nodes.For the smart city, Fairness Cooperation Algorithm (FCA) was proposed by Dong et al. [92], for thejoint optimization problem. The authors have formulated a convex-non-linear programming problemof minimizing the total system cost (delay and energy consumption), subject to power, workload andcomputation capacity threshold constraints. QoE and fairness of users under FCA was compared withbaseline algorithm (BA) and distributed optimization algorithm (DOA).5.5. Cache Enabled Edge DevicesFog networking is a vast ﬁeld, which uses both smart objects and the already deployed networkinfrastructure. System can ensure QoS either by using already deployed network devices or byoptimally dividing the tasks among edge devices or using both at the same time. Cache placementin edge devices/fog nodes in a network improves the efﬁciency of a system in terms of low powerconsumption, low-latency, high throughput, and efﬁcient spectrum use. To deal with the congestion23Sensors 2019, 19, 4807problem at the backhaul link of F-RAN, authors have proposed a projection gradient method in [93].To increase the throughput, they have used stochastic geometry to derive a close-form of successfultransmission probability (STP). Afterward, the optimal placement of cache over the network wasdone. To fulﬁll the delay requirements, authors in [94] have proposed a decentralized asynchronouscoding caching scheme. The result shows that the proposed algorithm is more efﬁcient than existingalgorithms present in the literature. A convex optimization problem of minimizing the worst-casefronthaul delay was formulated by the authors. Depending on application delay requirements, theproposed coded scheme gives synchronous and asynchronous transmission methods. The problemof user association with a fog node on the basis of lower latency is formulated in [95]. The authorshave formulated the problem using game theory, for which they have used a proactive caching schemeand Boltzmann-Gibbs learning algorithm for solution. Latency of proposed fog network is the sum ofcomputing and queuing delays.Different researchers have worked on the F-RAN design considering various aspects. For a greensystem, as discussed by Chen et al. [67,96], minimization of energy consumption is done by consideringan F-RAN that is cache-enabled for the selection of Remote Radio Head (RRHs). For balancingfront-haul trafﬁc, Park et al. [97] have discussed a scheme to deliver data from Base Band Unit (BBU)and RRHs. Content placement problem involving caching has been discussed in X Peng et al. [98] andDai et al. [99]. F-RANs accommodate caching in the road-side units. Di Chen et al. [100] have workedon maximizing the Signal to Interference Noise Ratio (SINR) to ensure fairness while jointly optimizingcluster formation and multicast beamforming. The objective function in this work has non-convexconstraints and is collectively an NP-hard problem. Sensor-cloud system gives solutions to manyapplications in a smart city. The system was developed by the integration of CPS and cloud computing.Besides many beneﬁts, a major problem is coupling resource management, which was discussedin [101]. They have introduced a fog layer between sensor and cloud layer, which emphasizes theservices. Firstly, authors have proposed an algorithm for caching at fog layer, afterwards, Hungarianalgorithm was extended to deal with the optimal use of resources on basis of maximum matching.The proposed algorithms result in the minimization of latency for sustainable services.6. Open Research IIoT Application Domains, Fog Computing as an EnablerIn Industry 4.0, the digital transformation of the industry requires research development in allﬁelds (smart city, D2D communication, transportation, healthcare, etc.). These all IIoT domains havesame critical issues in communication and networking. Even though, fog computing has umpteenthapplications in multiple research areas, few notable applications in respective of IIoT domain havebeen listed in Table 1 to gain an idea about the diverse use of fog computing. The main objective isachieving maximum beneﬁts/solutions using fog computing for this industrial revolution at efﬁcientoptimized QoS measurements. Many authors have proposed solutions in past years to networkingand communication challenges in order to leverage beneﬁts using fog computing in all domains ofIIoT. Some has started to provide prototypes for supporting their research, Table 2 has listed somecase studies for various areas of open research. In this section, some of the past research worksare summarized for readers to get an idea about the integration of fog computing with differentpre-existing network architectures. This integration brings solutions to support IIoT applications, yetthere are many open research areas in all ﬁelds that need to be solved in the coming future towards theindustrial development.24Sensors 2019, 19, 4807Table1.LiteratureReview:R.A=ResourceAllocation,L=Latency,E=Energy,T/R/C=Throughput/Rate/Capacity,Cc=Cache,P=Power,H=Handover,B=Bandwidth,S=Security,T.L=TransmissionLink.Ref.NoIIoTApplicationDomainR.ALET/R/CCcPHBST.LArchitecture[49]MobilityRoutingSDN[50]BigDataAnalyticsRoutingHetNets[51]SmartIoTdevicesDownlinkCloudComputing[52]SmartIoTdevicesDownlinkRANs[53]BigDataAnalyticsDownlinkNOMA+RANs[59]VANETSDownlinkCloudComputing[61]HealthcareDownlink+UplinkCloudComputing[62]SmartIoTdevicesDownlinkFog-IoT[69]5GnetworkDownlinkNFV+RANs[72]VirtualizedPassiveOpticalNetworks(VPON)/5GDownlink+UplinkRANs+CloudComputing[77]SmallCellNetworks(SCNs)/5GUplinkRANs[78]5GnetworkDownlinkRANs[79]DownlinkCloudComputing[81]5GnetworkDownlinkRANs[82]HeterogeneousIoTapplicationsDownlinkHetNets[102]VANETsVFC[84]SmartmonitoringsystemsDownlinkWSN+CPS[85]SecurityRoutingVN+CloudComputing[87]HeterogeneousIoTapplicationsDownlinkCloudComputing[88]Time-sensitiveIoTapplicationsUplinkRANs[89]HeterogeneousIoTapplicationsDownlinkCloudComputing[93]WirelessnetworkRANs[94]DownlinkRANs[95]5GnetworkDownlinkFog-IoT[103]MicrogridDownlinkVM+CloudComputing[104]Security+MicrogridsDownlinkCloudComputing[105]MicrogridDownlinkCloudComputing[101]SmartcityRoutingCPS+CloudComputing[92]SmartcityDownlinkFog-IoT[106]MultimediaDownlinkCloudComputing[107]SecureandtimesavingmultimediaRoutingICN[108]SecureIoTapplicationsDownlinkD2D[109]DownlinkD2D+RANs[110]5Gmobilenetwork+V2GservicesRoutingV2G[111]VANETsIoT+ITS25Sensors 2019, 19, 4807Table1.Cont.Ref.NoIIoTApplicationDomainR.ALET/R/CCcPHBST.LArchitecture[112]Mobility+VANETsDownlinkCloudComputing[113]Mobility+VANETsDownlinkFog-Ues[114]Mobility+SmartcityRANs+CloudComptinig[115]VANETsRoutingSDN[116]HeterogeneousIoTapplicationsRoutingSDN+Blockchain[117]e-HealthcareDownlinkBlockchain[118]Cooperative+securehealthcareRoutingFog+IoT[119]Big-DataAnalytics+securityCloudComputing[120]SmarthomeacasestudyCloudComputing[121]SmartcityvideoapplicationsRoutingCloudComputing26Sensors 2019, 19, 4807Table 2. Case studies, Fog Computing as an enabler.Ref.NoIIoT ApplicationDomain Case Study: Key Focus[122] smart city Smart city solutions have been deployed in cities, such as Barcelona and Venice,to make further advancements in e-governance[123]smart trafﬁccontrol and healthmonitoringTo increase ﬂexibility in a fog computing in the context of Complex eventprocessing (CEP), a case study is presented. The methodology, called“mechanism transitions”, is used to study how and where a query should beprocessed and how this decision affects the performance.[124] smart cityFog Computing Architecture Network (FOCAN) is presented to give low-latencyand energy-efﬁcient solution for smart city applications. It manages differentapplication’s requirements by categorizing the trafﬁc type and its ﬂow.[125] city, factory,building, homeUsing an open-source platform Distributed Node-RED (DNR), authors havepresented how applications can be decomposed and deployed. They buildprototype for scalability and dynamic nature solutions using the networksimulator Omnet++.[126] smart pipelinemonitoringA sequential machine learning algorithm on every layer of fog-cloud architecture,sensors and Markov model are used to monitor, control and detection ofhazardous events of a pipeline system. A working prototype was constructedto observe 12 distinct events. This prototype could be used for future city-widepipeline safety measurements[127] smarttransportationThe extended policy management to support secure travel to user’s is presentedby the authors. Four different route guiding scenarios are explained; namelydepending on trafﬁc condition, emergency connected vehicles (ECV), connectedvehicle (CV) and probable collision detection.[128] smarttransportationSmart transportation framework is proposed for Vehicle to Vehicle (V2V)communication by the authors, on basis of the current trafﬁc situation (road andvehicle’s condition, capacity).[129] big data Case study named as “Streamcloud” is presented to provide real-timeenergy-efﬁcient solution.[130] healthcarePersonalized missing data resilient decision-making approach is validated on areal human subject trial on maternity health. Data missing in critical applicationsis a very crucial challenge, that needs to be solved.[131] healthcare Table 2 in the mentioned paper gives some projects for healthcare monitoringsupported by fog computing, cloud computing, and IoT.[132] healthcareA demo test-bed is developed on edge-IoT architecture for e-healthcareapplications. Proposed EH-IoT gives better results towards bandwidth andlatency requirements. The article also presents the beneﬁts leveraging from IoTand edge computing from an industrial perspective.[133] cardiac diseases A case study using Electrocardiogram (ECG) feature is discussed in the article tomonitor health in real time.6.1. Micro-Grids (MGs)Multiple loads and distributed renewable energy resources combine to form an electrical systemnamed as MG. This energy is stored in the storage devices, though, it can have signiﬁcant powerlosses when power is exchanged between different MGs. It increases reliability and efﬁciency of thesystem. In [104] authors implemented their proposed framework on a test MG system. Performancewas observed for the proposed three-layer fog computing system. A convex linear programmingproblem was formulated with an objective of minimizing the total cost in terms of power consumption.The constraints are total load threshold which was calculated using the power balance equation.The equation has three types of powers with threshold range namely, dispatchable, non-dispatchableand grid power. Graph theory was applied to the proposed system and a fast consensus-basedalgorithm by taking advantage of fog computing was proposed. An optimization problem of powerdemand problem was discussed in [105] for hybrid fog-cloud system. The high volume of data by the27Sensors 2019, 19, 4807number of smart devices results computation delays at cloud also it involves more power consumption.This power consumption can be optimized after load balancing among fog nodes and cloud server.Jalali et al. [134] have discussed various ways for the deployment of IoT nodes in an energy-efﬁcientway. The authors have used MGs and fog computing as an enabler with an aim to reduce the energyconsumption by the IoT speciﬁc applications. Energy consumption of various types has been discussedi.e., energy consumed during computation and balancing trafﬁc on fog as well as the cloud. Renewableenergy is stored in MGs, which can be used for further processing. Dynamic decisions such as weatherforecasting or renewable energy availability causes dependent energy-saving processes.6.2. Smart-Grids (SGs)Due to an increase in the number of devices required by a user at home, residential usage ofelectricity has been increasing with time. The proposed modern solutions, such as SGs, greatlyimprove the reliability, efﬁciency, and sustainability of the system in an automated way. Usingcommunication technologies, SGs works on gathered information about consumer’s and supplier’sbehavior. As SGs are distributed systems implemented on a very large scale, fog computing is ideal todeal with such a scenario. For this purpose, Foteini Beligianni et al. [135] have discussed how IoTsand fog computing can be integrated into the power systems without compromising user privacy.The proposed architecture works on demand-response service to ensure privacy by using standardprotocols and open-source resources. For devising a power plan for all the devices, a load scheduler isplaced in the system while data management performs all the necessary actions that are required toprocess the data. The software architecture of the proposed solution is based on Lambda architecturewhich embodies edge computing. To meet the demands of real time processing, collecting information,computing and storing the data generated by multiple smart meters, the aforementioned researchershave proposed a fog-based solution. Fog networking acts as a bridge between the cloud and the SG.With fog networking, geographically distributed smart meters can be employed. Latency is reducedwhile improving location awareness and privacy for SG. Furthermore, Rao et al. [136] have workedon how to maintain the QoS by minimizing the expense of electricity using coordination betweendata centers.6.3. MultimediaMultimedia communication involves a large amount of audio and video data, being producedby smart IIoT devices. This ever-increasing amount of multimedia content brings new challenges.Low-latency and energy-efﬁcient solutions are required to provide services. A direct consequence ofincreasing multimedia trafﬁc is overburdening of the already strained mobile access network channels.The essential components of a multimedia communication system can be visualized with the help ofFigure 1. The authors in [106], have formulated a cost minimization optimization problem, this costdepends on scheduler decisions. Scheduler takes decisions regarding distribution and states (open orclose) of fog nodes. This cost minimization problem alternatively converted into multimedia user’s(MMU) response time minimization problem under capacity, coverage zone area and associationconstraints. The authors have introduced a fog node to resolve the resource management and latencyissues between cloud and MMUs. Using Stackelberg game, an online resource allocation scheme wasproposed. The enormous amount of data production for smart applications, need security and privacyas well. The authors have proposed a chaotic cryptographic method to ensure security along withlow-latency requirements for Information-Centric Multimedia Network (ICMN) [107]. The speed ofencryption and decryption of multimedia streams in the cryptographic method, was enhanced usingfog computing.6.4. Device to Device (D2D) CommunicationResearchers have to ﬁnd a new way in which devices can independently work without using theexisting cellular infrastructure. The demand for new infrastructure development is due to the increase28Sensors 2019, 19, 4807in the number of smart devices that causes scarcity in spectrum and resources. This idea gives rise toD2D communication technology in which devices are part of the Internet and can communicate witheach other without the involvement of internet infrastructure. This direct communication will result inless usage of the wireless spectrum. A device is itself constrained-bounded in terms of parameters, suchas energy or resources, hence cloud and fog computing technologies support them. These emergingtechnologies enhance processing capabilities and result in efﬁcient system formation. Cloud servicesfor D2D communications have performance bottleneck as devices use direct communication, hencean alternative fog service is required. Fog with the same set of services, provide better performance.Fog networking act as an alternative of cloud, as it provides desired requirements in the vicinity ofedge devices. Researchers have proposed D2D fogging to address energy-efﬁcient task ofﬂoading inD2D communication [137]. The proposed method includes a framework for task ofﬂoading and usesassistance from the network for D2D. Mobile users use the communication and computation resourcesof each other dynamically. After discussing several security issues in fog computing, authors haveproposed three lightweight anonymous authentication protocols (LAAPs) [108].The proposed scheme with the aid of D2D communication, is feasible for IoT devices whichare resource-limited. Li et al. [109] proposed a F-community architecture for F-RAN followed by adata caching scheme. The caching scheme for UEs in D2D aided F-RAN helps in reduction of delays.Nodes in the system with higher chances of being selected as the central nodes store the most popularcontent in their cache. For access to any data, a user receives data from the cache stored previously.Kaur et al. [138] have proposed cachinMobile, which, compared to the cloud, can meet the low latencyand energy efﬁciency requirements by using the elastic services provided by the nodes at the edge.Energy efﬁciency and low latency requirements is a major issue in this technique as the resources atthe edge and the devices that are mobile. This approach has proposed to use D2D communicationto carry out communication at a short distance and save network resources. CachinMobile not onlyimproves energy efﬁciency but also maintains QoS.6.5. Vehicular Ad-hoc Networks (VANETs)This area is named as Intelligent Transportation System (ITS), which is an emerging area withmany open issues that need to be solved. A transportation service and automobile service managementinvolves the controlling and monitoring of the transportation network. ITS is designed in such away that this system can satisfy the required QoS parameters of the transportation network. TheseQoS requirements involve reliable connection, efﬁcient performance, safety, and privacy requirement,mobility and scalability requirement. A transportation network system aided with ITS technologyhas components that can be optimized. These are Global Positing System (GPS), RFID sensor tagsand readers, road-side equipment for example trafﬁc lights, signals, road bank cameras, cars. TheseITS subsystems integrated with IoT technology elements will help in monitoring and managing oftransportation environment; distribution of vehicles according to the scenario; manufacturing of ITS;shipping, tracking and monitoring of physical objects.Enormous number of such physical objects embedded with data processing capabilities, integratedwith RFID sensors along with networking technologies will promote IIoT applications. These IIoTapplications will help in monitoring the exact original location as well as destination location ofvehicle or aero-plane or ships along with the followed path trafﬁc condition or environment effectsor any emergency road situation. Many authors have done research in designing ITS supportivetransportation networks optimizing wireless communication technologies, RFID tags or antennas.The use of IoT technology in the transportation industry and systems results in new research domainssuch as vehicular ad-hoc networks (VANETs), internet of vehicles (IoV) and vehicle to grid (V2G).These domains are very promising areas of research. In recent times the integration of VANETswith fog computing results in better efﬁcient system designing. Services of automotive connectivityarchitecture can be improved using fog computing as an enabler for smart transportation (ITS), suchas the one shown in Figure 1 (Transportation). Efﬁciency is improved in terms of minimizing the29Sensors 2019, 19, 4807latency for time-sensitive applications using fog computing. Being an integral part of the ITS, VANETshave many applications. Computation and communication demands are hard to meet in the cloudarchitecture. Incorporating fog networking not only fulﬁlls the demands but also improves latency,location awareness and energy efﬁciency. For power management, there is a need for new technologies.For this problem Vehicle to Grid (V2G) is a recent concept with open issues, that uses renewable energy.When renewable energy is available it is used to charge plug-in electric vehicles. Otherwise, theseelectric vehicles are used as the source of energy. The use of alternative energy source will reducethe power burden on the grid during peak hours. Owners of these plug-in electric vehicles got paidby electric companies using metering systems. It provides some relief to electric companies in termsof payment, trafﬁc load, and energy consumption during peak hours. In smart grids, a distributedarchitecture with storage and processing capabilities is need to be deployed, as there is a high factor ofmobility in V2G. To implement the V2G services in the 5G network, a hybrid fog and cloud architecturewas proposed by researchers in [110]. Open issues, such as energy efﬁciency, resource management,security, and privacy are needed to be addressed in the future to improve system efﬁciency. Securityis a big challenge in VANETS, fog computing integration is smart transportation brings solution tothis challenge. Ma et al. [111] have designed a new authenticated key agreement (AKA) protocol forfog-based VANETs.The authors in [112] compared fog and cloud computing performance in a real VANETenvironment. Results shows that fog computing gives better services for real-time scenario applications,namely trafﬁc detection and time estimation. Fog computing performs well because it supportsthe main attributes of VANETs that are location-awareness, mobility and real-time communication.Integration of the Internet of Things with VANETs gives rise to Internet of Vehicles (IoV) is a matter ofgrowing interest over the last few years [139].6.6. Big-Data AnalyticsAs the next revolutionized era comprises enormous smart devices that supports IIoT applications,this causes the generation of a signiﬁcant amount of data. Due to the widespread acceptance of fogcomputing, signiﬁcant research has been carried out in the domain of big-data analytics. Recentadvancements show an indication that fog networking has the potential to provide solutions in thisﬁeld. Fog computing, as an extension of cloud computing, gives solutions to problems such aslocation-awareness, mobility, big-data analytics, and cyber threats. Author in [119] have designed athree-layer architecture of IoT-Fog-Cloud that supports big-data analytics and security applications.Author summarized cyber attack types and compared the existing security solutions. The authorsin [140] provide a review article on fog computing challenges in the context of big IoT analytics.6.7. Software Deﬁned Networking (SDN)SDN is an evolving concept to remotely control the entire network from a centralized location.Distributed networks can be integrated with SDN architecture for more granularity of control inthe network. Since edge computing is done near the edge devices, there is a high probability thatthe edge devices are mobile due to the exponential evolution of smartphones and other handhelddevices. Similarly, the ever-changing requirements of resources for each user brings a certain degreeof dynamism in the network which needs to be catered to ensure efﬁcient operation. Since fog is anew concept and it cannot completely replace the existing cloud architecture, both fog and cloud workhand in hand for smooth operation of the network. The interplay between fog networking and cloudcomputing is always there when both work in an integrated fashion. The four-layer architecture wasstudied for end-to-end delay, energy consumption and packet loss ratio in [115]. The authors formulatemixed integer programming (MIP) problem of minimizing the energy consumption under data rate,bandwidth and delay threshold constraints. A new routing protocol was proposed for VANETapplications using SDN and fog computing, named as Energy Efﬁcient Multicast routing protocol(EEMSFV). SDN controller, OpenFlow switches and fog computing works under two algorithms,30Sensors 2019, 19, 4807priority based scheduling algorithm for classifying the trafﬁc message type (emergency or safetyapplications) and a classiﬁcation algorithm to schedule the requests.7. ConclusionsIndustry 4.0, a revolutionized era which will have a massive number of smart devices that willsupport IIoT applications in every ﬁeld. This deployment of smart devices will change all domains ofhuman life’s perspective. IIoT applications will provide solutions to all ﬁelds, such as transportation,healthcare, food supply chain, education, and industry. These IIoT applications will provide efﬁcient,effective solutions for future networks. There are challenges in communication and networking interms of latency, bandwidth, resource allocation, and storage.All these advanced IIoT applications will create a huge amount of data, causing a burden on thecloud. Even though cloud computing provides services to the edge devices, it incurs huge latency,resource allocation challenges, caching placement problems, energy consumption. These issues aredetrimental to the QoS aspect of a network. Fog as an extension of cloud provides a platform tocompute, control, store and manage these IIoT devices. In the future, it will reshape all sectorsinvolving IIoT applications, with the integration of important existing communication technologiesnamely, CPS, SDN, NFV, 5G, D2D. This brings computation, resource management, and storagechallenges. In this paper, ﬁrst, we gave an overview of IIoT applications and its enabling technologiesused for new revolutionized era. The pre existing protocols and solutions to challenges related to fogcomputing are summarized. In the end, we have mentioned open research IIoT domains, in which fogcomputing can act as an enabler. We have previewed the work carried out by numerous researchersincorporating fog computing to provide services to IIoT edge devices leveraging towards Industry 4.0way. Critical review of some existing work is summarized in the table, which can be used to ﬁnd openresearch challenges. Towards the development of this industrial transformative epoch, most of theresearch work is still uncertain and waiting. This is an interesting era to discover what fog computingmay contribute to the world of automation in the coming future.Author Contributions: R.B. is the main author for this survey article, who wrote the original draft.M.A. (Mudassar Ali), S.Q., M.A. (Monther Aldwairi), M.I.A., and A.M. contributed in terms of conceptualization,organization and validation of the article. A.M., M.G. and M.A. (Monther Aldwairi) also contributed toacquire funds.Funding: This work was supported by the Swedish Knowledge Foundation under Grant 20180178.Acknowledgments: This work was supported by Zayed University Research Ofﬁce, Research Cluster Award# R17079.Conﬂicts of Interest: The authors declare no conﬂict of interest.References1. Rivera, J.; Goasduff, L. Gartner says a thirty-fold increase in internet-connected physical devicesby 2020 will signiﬁcantly alter how the supply chain operates. Gartner 2014. Available online:https://www.gartner.com/en/newsroom/press-releases/2014-03-24-gartner-says-a-thirty-fold-increase-in-internet-connected-physical-devices-by-2020-will-signiﬁcantly-alter-how-the-supply-chain-operates(accessed on 19 July 2019).2. Roblek, V.; Meško, M.; Krapež, A. A complex view of industry 4.0. Sage Open 2016, 6. [CrossRef]3. Thames, L.; Schaefer, D. Software-deﬁned cloud manufacturing for industry 4.0. Procedia CIRP 2016,52, 12–17. [CrossRef]4. Varghese, A.; Tandur, D. Wireless requirements and challenges in Industry 4.0. In Proceedings ofthe 2014 International Conference on Contemporary Computing and Informatics (IC3I), Mysore, India,27–29 November 2014; pp. 634–638.5. Akpakwu, G.A.; Silva, B.J.; Hancke, G.P.; Abu-Mahfouz, A.M. A survey on 5G networks for the Internet ofThings: Communication technologies and challenges. IEEE Access 2018, 6, 3619–3647. [CrossRef]31Sensors 2019, 19, 48076. Vangelista, L.; Zanella, A.; Zorzi, M. Long-range IoT technologies: The dawn of LoRaTM. In Future AccessEnablers of Ubiquitous and Intelligent Infrastructures; Springer: Cham, Switzerland, 2015; pp. 51–58.7. Osseiran, A.; Boccardi, F.; Braun, V.; Kusume, K.; Marsch, P.; Maternia, M.; Queseth, O.; Schellmann, M.;Schotten, H.; Taoka, H.; et al. Scenarios for 5G mobile and wireless communications: The vision of theMETIS project. IEEE Commun. Mag. 2014, 52, 26–35. [CrossRef]8. Christensen, J.H. Using RESTful web-services and cloud computing to create next generation mobileapplications. In Proceedings of the 24th ACM SIGPLAN Conference Companion on Object OrientedProgramming Systems Languages and Applications, Orlando, FL, USA, 25–29 October 2009; pp. 627–634.9. Buyya, R.; Yeo, C.S.; Venugopal, S.; Broberg, J.; Brandic, I. Cloud computing and emerging IT platforms:Vision, hype, and reality for delivering computing as the 5th utility. Future Gener. Comput. Syst. 2009,25, 599–616. [CrossRef]10. Atlam, H.F.; Alenezi, A.; Alharthi, A.; Walters, R.J.; Wills, G.B. Integration of cloud computing with internetof things: Challenges and open issues. In Proceedings of the 2017 IEEE International Conference onInternet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber,Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData), Exeter, UK, 21–23 June 2017;pp. 670–675.11. Ai, Y.; Peng, M.; Zhang, K. Edge computing technologies for Internet of Things: A primer.Digit. Commun. Netw. 2018, 4, 77–86. [CrossRef]12. Peter, N. Fog computing and its real time applications. Int. J. Emerg. Technol. Adv. Eng. 2015, 5, 266–269.13. Aazam, M.; Huh, E.N. Fog computing and smart gateway based communication for cloud of things.In Proceedings of the 2014 International Conference on Future Internet of Things and Cloud, Barcelona,Spain, 24–29 August 2014; pp. 464–470.14. Atlam, H.F.; Alenezi, A.; Walters, R.J.; Wills, G.B.; Daniel, J. Developing an adaptive Risk-based accesscontrol model for the Internet of Things. In Proceedings of the 2017 IEEE International Conference onInternet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber,Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData), Exeter, UK, 21–23 June 2017;pp. 655–661.15. Bonomi, F.; Milito, R.; Zhu, J.; Addepalli, S. Fog computing and its role in the Internet of things.In Proceedings of the First Edition of the MCC Workshop on Mobile Cloud Computing, Helsinki, Finland,13–17 August 2012; pp. 13–16.16. Wen, Z.; Yang, R.; Garraghan, P.; Lin, T.; Xu, J.; Rovatsos, M. Fog orchestration for internet of things services.IEEE Internet Comput. 2017, 21, 16–24. [CrossRef]17. Verma, M.; Bhardwaj, N.; Yadav, A.K. Real time efﬁcient scheduling algorithm for load balancing in fogcomputing environment. Int. J. Inf. Technol. Comput. Sci 2016, 8, 1–10. [CrossRef]18. Fog Computing and the Internet of Things: Extend the Cloud to Where the Things Are. White Paper. 2016.Available online: https://www.cisco.com/c/dam/en_us/solutions/trends/iot/docs/computing-overview.pdf (accessed on 19 September 2019).19. Atlam, H.; Walters, R.; Wills, G. Fog computing and the Internet of things: A review. Big Data Cogn. Comput.2018, 2, 10. [CrossRef]20. Chiang, M.; Zhang, T. Fog and IoT: An overview of research opportunities. IEEE Internet Things J. 2016,3, 854–864. [CrossRef]21. Liu, Y.; Fieldsend, J.E.; Min, G. A framework of fog computing: Architecture, challenges, and optimization.IEEE Access 2017, 5, 25445–25454. [CrossRef]22. Peralta, G.; Iglesias-Urkia, M.; Barcelo, M.; Gomez, R.; Moran, A.; Bilbao, J. Fog computing based efﬁcientIoT scheme for the Industry 4.0. In Proceedings of the 2017 IEEE International Workshop of Electronics,Control, Measurement, Signals and their Application to Mechatronics (ECMSM), Donostia-San Sebastian,Spain, 24–26 May 2017; pp. 1–6.23. Bonomi, F.; Milito, R.; Natarajan, P.; Zhu, J. Fog computing: A platform for internet of things and analytics.In Big Data and Internet of Things: A Roadmap for Smart Environments; Springer: New York, NY, USA, 2014;pp. 169–186.24. Agarwal, S.; Yadav, S.; Yadav, A.K. An efﬁcient architecture and algorithm for resource provisioning in fogcomputing. Int. J. Inf. Eng. Electron. Bus. 2016, 8, 48. [CrossRef]32Sensors 2019, 19, 480725. Ketel, M. Fog-cloud services for iot. In Proceedings of the of the SouthEast Conference, Kennesaw, GA, USA,13–15 April 2017; pp. 262–264.26. Georgakopoulos, D.; Jayaraman, P.P.; Fazia, M.; Villari, M.; Ranjan, R. Internet of Things and edge cloudcomputing roadmap for manufacturing. IEEE Cloud Comput. 2016, 3, 66–73. [CrossRef]27. Lu, Y. Industry 4.0: A survey on technologies, applications and open research issues. J. Ind. Inf. Integr. 2017,6, 1–10. [CrossRef]28. Camarillo, A.; Ríos, J.; Althoff, K.D. Product Lifecycle Management as Data Repository for ManufacturingProblem Solving. Materials 2018, 11, 1469. [CrossRef]29. Obst, M.; Holm, T.; Urbas, L.; Fay, A.; Kreft, S.; Hempen, U.; Albers, T. Semantic description of processmodules. In Proceedings of the 2015 IEEE 20th Conference on Emerging Technologies & Factory Automation(ETFA), Luxembourg, 8–11 September 2015; pp. 1–8.30. Li, B.; Zhao, Z.; Guan, Y.; Ai, N.; Dong, X.; Wu, B. Task Placement Across Multiple Public Clouds WithDeadline Constraints for Smart Factory. IEEE Access 2018, 6, 1560–1564. [CrossRef]31. Chen, H.; Abbas, R.; Cheng, P.; Shirvanimoghaddam, M.; Hardjawana, W.; Bao, W.; Li, Y.; Vucetic, B.Ultra-reliable low latency cellular networks: Use cases, challenges and approaches. IEEE Commun. Mag.2018, 56, 119–125. [CrossRef]32. Ladiges, J.; Fay, A.; Holm, T.; Hempen, U.; Urbas, L.; Obst, M.; Albers, T. Integration of modular processunits into process control systems. IEEE Trans. Ind. Appl. 2018, 54, 1870–1880. [CrossRef]33. Vogel-Heuser, B.; Diedrich, C.; Pantförder, D.; Göhner, P. Coupling heterogeneous production systems by amulti-agent based cyber-physical production system. In Proceedings of the 2014 12th IEEE InternationalConference on Industrial Informatics (INDIN), Porto Alegre, Brazil, 27–30 July 2014; pp. 713–719.34. Wan, J.; Tang, S.; Shu, Z.; Li, D.; Wang, S.; Imran, M.; Vasilakos, A.V. Software-deﬁned industrial internet ofthings in the context of industry 4.0. IEEE Sens. J. 2016, 16, 7373–7380. [CrossRef]35. Gruber, F.E. Industry 4.0: A best practice project of the automotive industry. In Proceedings of theIFIP International Conference on Digital Product and Process Development Systems, Dresden, Germany,10–11 October 2013; pp. 36–40.36. Vaquero, L.M.; Rodero-Merino, L. Finding your way in the fog: Towards a comprehensive deﬁnition of fogcomputing. ACM SIGCOMM Comput. Commun. Rev. 2014, 44, 27–32. [CrossRef]37. Stock, T.; Seliger, G. Opportunities of sustainable manufacturing in industry 4.0. Procedia Cirp 2016,40, 536–541. [CrossRef]38. Elliott, J.A. An Introduction to Sustainable Development; Routledge: Abingdon, UK, 2012.39. Bibri, S.E.; Krogstie, J. Smart sustainable cities of the future: An extensive interdisciplinary literature review.Sustain. Cities Soc. 2017, 31, 183–212. [CrossRef]40. Schlechtendahl, J.; Keinert, M.; Kretschmer, F.; Lechler, A.; Verl, A. Making existing production systemsIndustry 4.0-ready. Prod. Eng. 2015, 9, 143–148. [CrossRef]41. Mouradian, C.; Naboulsi, D.; Yangui, S.; Glitho, R.H.; Morrow, M.J.; Polakos, P.A. A comprehensive surveyon fog computing: State-of-the-art and research challenges. IEEE Commun. Surv. Tutor. 2017, 20, 416–464.[CrossRef]42. Dong, M.; Ota, K.; Liu, A. Preserving source-location privacy through redundant fog loop for wirelesssensor networks. In Proceedings of the 2015 IEEE International Conference on Computer and InformationTechnology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing;Pervasive Intelligence and Computing (CIT/IUCC/DASC/PICOM), Liverpool, UK, 26–28 October 2015;pp. 1835–1842.43. Huang, L.; Li, G.; Wu, J.; Li, L.; Li, J.; Morello, R. Software-deﬁned QoS provisioning for fog computingadvanced wireless sensor networks. In Proceedings of the 2016 IEEE SENSORS, Orlando, FL, USA,30 October–3 November 2016; pp. 1–3.44. Liu, Y.; Gao, J.; Jia, Y.; Zhu, L. A cluster maintenance algorithm based on LEACH-DCHS protoclol.In Proceedings of the International Conference on Networking, Architecture, and Storage, 2008. NAS’08,Chongqing, China, 12–14 June 2008; pp. 165–166.45. Handy, M.; Haase, M.; Timmermann, D. Low energy adaptive clustering hierarchy with deterministiccluster-head selection. In Proceedings of the 4th International Workshop on Mobile and WirelessCommunications Network, Stockholm, Sweden, 9–11 September 2002; pp. 368–372.33Sensors 2019, 19, 480746. Smaragdakis, G.; Matta, I.; Bestavros, A. Sep: A Stable Election Protocol for Clustered Heterogeneous WirelessSensor Networks; Technical Report; Boston University Computer Science Department: Boston, MA, USA, 2004.47. Singh, D.; Panda, C.K. Performance analysis of modiﬁed stable election protocol in heterogeneous wsn.In Proceedings of the 2015 International Conference on Electrical, Electronics, Signals, Communication andOptimization (EESCO), Visakhapatnam, India, 24–25 January 2015; pp. 1–5.48. Naranjo, P.G.V.; Shojafar, M.; Abraham, A.; Baccarelli, E. A new stable election-based routing algorithm topreserve aliveness and energy in fog-supported wireless sensor networks. In Proceedings of the 2016 IEEEInternational Conference on Systems, Man, and Cybernetics (SMC), Budapest, Hungary, 9–12 October 2016;pp. 2413–2418.49. Bi, Y.; Han, G.; Lin, C.; Deng, Q.; Guo, L.; Li, F. Mobility support for fog computing: An SDN approach.IEEE Commun. Mag. 2018, 56, 53–59. [CrossRef]50. Wang, X.; Yang, L.T.; Kuang, L.; Liu, X.; Zhang, Q.; Deen, M.J. A tensor-based big-data-driven routingrecommendation approach for heterogeneous networks. IEEE Netw. 2019, 33, 64–69. [CrossRef]51. Du, J.; Zhao, L.; Feng, J.; Chu, X. Computation ofﬂoading and resource allocation in mixed fog/cloudcomputing systems with min-max fairness guarantee. IEEE Trans. Commun. 2018, 66, 1594–1608. [CrossRef]52. Sun, Y.; Peng, M.; Mao, S.; Yan, S. Hierarchical radio resource allocation for network slicing in fog radioaccess networks. IEEE Trans. Veh. Technol. 2019, 68, 3866–3881. [CrossRef]53. Zhang, H.; Qiu, Y.; Long, K.; Karagiannidis, G.K.; Wang, X.; Nallanathan, A. Resource allocation inNOMA-based fog radio access networks. IEEE Wirel. Commun. 2018, 25, 110–115. [CrossRef]54. Madsen, H.; Burtschy, B.; Albeanu, G.; Popentiu-Vladicescu, F. Reliability in the utility computing era:Towards reliable fog computing. In Proceedings of the 2013 20th International Conference on Systems,Signals and Image Processing (IWSSIP), Bucharest, Romania, 7–9 July 2013; pp. 43–46.55. Hong, K.; Lillethun, D.; Ramachandran, U.; Ottenwälder, B.; Koldehofe, B. Mobile fog: A programmingmodel for large-scale applications on the internet of things. In Proceedings of the Second ACM SIGCOMMWorkshop on Mobile cloud Computing, Hong Kong, China, 12–16 August 2013; pp. 15–20.56. Zhu, J.; Chan, D.S.; Prabhu, M.S.; Natarajan, P.; Hu, H.; Bonomi, F. Improving web sites performanceusing edge servers in fog computing architecture. In Proceedings of the 2013 IEEE Seventh InternationalSymposium on Service-Oriented System Engineering, Redwood City, CA, USA, 25–28 March 2013;pp. 320–323.57. Rudenko, E. Fog Computing Is a New Concept of Data Distribution. Available online: https://xcluesiv.com/fog-computing-is-a-new-concept-of-data-distribution/ (accessed on 19 July 2019).58. Do, C.T.; Tran, N.H.; Pham, C.; Alam, M.G.R.; Son, J.H.; Hong, C.S. A proximal algorithm for joint resourceallocation and minimizing carbon footprint in geo-distributed fog computing. In Proceedings of the 2015International Conference on Information Networking (ICOIN), Siem Reap, Cambodia, 12–14 January 2015;pp. 324–329.59. Ning, Z.; Huang, J.; Wang, X. Vehicular fog computing: Enabling real-time trafﬁc management for smartcities. IEEE Wirel. Commun. 2019, 26, 87–93. [CrossRef]60. Mutlag, A.A.; Ghani, M.K.A.; Arunkumar, N.a.; Mohamed, M.A.; Mohd, O. Enabling technologies for fogcomputing in healthcare IoT systems. Future Gener. Comput. Syst. 2019, 90, 62–78. [CrossRef]61. Khattak, H.A.; Arshad, H.; ul Islam, S.; Ahmed, G.; Jabbar, S.; Sharif, A.M.; Khalid, S. Utilization and loadbalancing in fog servers for health applications. EURASIP J. Wirel. Commun. Netw. 2019, 2019, 91. [CrossRef]62. Abkenar, F.S.; Jamalipour, A. EBA: Energy Balancing Algorithm for Fog-IoT Networks. IEEE Internet Things J.2019, 6, 6843–6849. [CrossRef]63. Beraldi, R.; Mtibaa, A.; Alnuweiri, H. Cooperative load balancing scheme for edge computing resources.In Proceedings of the 2017 Second International Conference on Fog and Mobile Edge Computing (FMEC),Valencia, Spain, 8–11 May 2017; pp. 94–100.64. Armbrust, M.; Fox, A.; Grifﬁth, R.; Joseph, A.D.; Katz, R.; Konwinski, A.; Lee, G.; Patterson, D.; Rabkin, A.;Stoica, I.; et al. A view of cloud computing. Commun. ACM 2010, 53, 50–58. [CrossRef]65. Ningning, S.; Chao, G.; Xingshuo, A.; Qiang, Z. Fog computing dynamic load balancing mechanism basedon graph repartitioning. China Commun. 2016, 13, 156–164. [CrossRef]66. Deng, R.; Lu, R.; Lai, C.; Luan, T.H.; Liang, H. Optimal workload allocation in fog-cloud computing towardbalanced delay and power consumption. IEEE Internet Things J. 2016, 3, 1171–1181. [CrossRef]34Sensors 2019, 19, 480767. Chen, D.; Kuehn, V. Adaptive radio unit selection and load balancing in the downlink of Fog radio accessnetwork. In Proceedings of the 2016 IEEE Global Communications Conference (GLOBECOM), Washington,DC, USA, 4–8 December 2016; pp. 1–7.68. Verma, S.; Yadav, A.K.; Motwani, D.; Raw, R.; Singh, H.K. An efﬁcient data replication and load balancingtechnique for fog computing environment. In Proceedings of the 2016 3rd International Conferenceon Computing for Sustainable Global Development (INDIACom), New Delhi, India, 16–18 March 2016;pp. 2888–2895.69. Zeng, Y.; Al-Quzweeni, A.; Elgorashi, T.E.; Elmirghani, J.M. Energy Efﬁcient virtualization framework for5G F-RAN. arXiv 2019, arXiv:1904.02481.70. Roca, D.; Quiroga, J.V.; Valero, M.; Nemirovsky, M. Fog function virtualization: A ﬂexible solution foriot applications. In Proceedings of the 2017 Second International Conference on Fog and Mobile EdgeComputing (FMEC), Valencia, Spain, 8–11 May 2017; pp. 74–80.71. Kaur, K.; Dhand, T.; Kumar, N.; Zeadally, S. Container-as-a-service at the edge: Trade-off between energyefﬁciency and service availability at fog nano data centers. IEEE Wirel. Commun. 2017, 24, 48–56. [CrossRef]72. Tinini, R.I.; Batista, D.M.; Figueiredo, G.B.; Tornatore, M.; Mukherjee, B. Low-latency and energy-efﬁcientBBU placement and VPON formation in virtualized cloud-fog RAN. IEEE/OSA J. Opt. Commun. Netw. 2019,11, B37–B48. [CrossRef]73. Xia, F.; Liu, L.; Li, J.; Ma, J.; Vasilakos, A.V. Socially aware networking: A survey. IEEE Syst. J. 2015,9, 904–921. [CrossRef]74. Zhang, C.; Sun, Y.; Mo, Y.; Zhang, Y.; Bu, S. Social-aware content downloading for fog radio access networkssupported device-to-device communications. In Proceedings of the 2016 IEEE International Conference onUbiquitous Wireless Broadband (ICUWB), Nanjing, China, 16–19 October 2016; pp. 1–4.75. Klas, G.I. Fog Computing and Mobile Edge cloud Gain Momentum Open Fog Consortium, Etsi Mec andCloudlets. 2015. Available online: https://yucianga.info/wp-content/uploads/2015/11/15_11_22-_Fog_computing_and_mobile_edge_cloud_gain_momentum_Open_Fog_Consortium-ETSI_MEC-Cloudlets_v1_1.pdf (accessed on 19 September 2019).76. Cau, E.; Corici, M.; Bellavista, P.; Foschini, L.; Carella, G.; Edmonds, A.; Bohnert, T.M. Efﬁcient exploitationof mobile edge computing for virtualized 5G in EPC architectures. In Proceedings of the 2016 4th IEEEInternational Conference on Mobile Cloud Computing, Services, and Engineering (MobileCloud), Oxford,UK, 29 March–1 April 2016; pp. 100–109.77. Li, Z.; Sichitiu, M.L.; Qiu, X. Fog Radio Access Network: A New Wireless Backhaul Architecture for SmallCell Networks. IEEE Access 2018, 7, 14150–14161. [CrossRef]78. Pontois, N.; Kaneko, M.; Dinh, T.H.L.; Boukhatem, L. User pre-scheduling and beamforming with outdatedCSI in 5G fog radio access networks. In Proceedings of the 2018 IEEE Global Communications Conference(GLOBECOM), Abu Dhabi, UAE, 9–13 December 2018; pp. 1–6.79. Li, L.; Guan, Q.; Jin, L.; Guo, M. Resource allocation and task ofﬂoading for heterogeneous real-time taskswith uncertain duration time in a fog queueing system. IEEE Access 2019, 7, 9912–9925. [CrossRef]80. Huang, J.; Duan, Q.; Guo, S.; Yan, Y.; Yu, S. Converged network-cloud service composition with end-to-endperformance guarantee. IEEE Trans. Cloud Comput. 2015, 6, 545–557. [CrossRef]81. Yu, Y.; Liu, S.; Tian, Z.; Wang, S. A dynamic distributed spectrum allocation mechanism based on gamemodel in fog radio access networks. China Commun. 2019, 16, 12–21.82. Abedin, S.F.; Alam, M.G.R.; Kazmi, S.A.; Tran, N.H.; Niyato, D.; Hong, C.S. Resource allocation forultra-reliable and enhanced mobile broadband IoT applications in fog network. IEEE Trans. Commun. 2018,67, 489–502. [CrossRef]83. Majd, A.; Sahebi, G.; Daneshtalab, M.; Plosila, J.; Lotﬁ, S.; Tenhunen, H. Parallel imperialist competitivealgorithms. Concurr. Comput. Pract. Exp. 2018, 30, e4393. [CrossRef]84. Majd, A.; Sahebi, G.; Daneshtalab, M.; Plosila, J.; Tenhunen, H. Hierarchal Placement of Smart Mobile AccessPoints in Wireless Sensor Networks Using Fog Computing. In Proceedings of the 2017 25th EuromicroInternational Conference on Parallel, Distributed and Network-based Processing (PDP), St. Petersburg,Russia, 6–8 March 2017; pp. 176–180.85. Moreno-Vozmediano, R.; Montero, R.S.; Huedo, E.; Llorente, I.M. Cross-site virtual network in cloud andfog computing. IEEE Cloud Comput. 2017, 4, 46–53. [CrossRef]35Sensors 2019, 19, 480786. Shojafar, M.; Cordeschi, N.; Baccarelli, E. Energy-efﬁcient adaptive resource management for real-timevehicular cloud services. IEEE Trans. Cloud Comput. 2016, 7, 196–209. [CrossRef]87. Lee, G.; Saad, W.; Bennis, M. An online optimization framework for distributed fog network formation withminimal latency. IEEE Trans. Wirel. Commun. 2019, 18, 2244–2258. [CrossRef]88. Vu, D.N.; Dao, N.N.; Jang, Y.; Na, W.; Kwon, Y.B.; Kang, H.; Jung, J.J.; Cho, S. Joint energy and latencyoptimization for upstream IoT ofﬂoading services in fog radio access networks. Trans. Emerg. Telecommun.Technol. 2019, 30, e3497. [CrossRef]89. Ali, M.; Riaz, N.; Ashraf, M.I.; Qaisar, S.; Naeem, M. Joint cloudlet selection and latency minimization in fognetworks. IEEE Trans. Ind. Inform. 2018, 14, 4055–4063. [CrossRef]90. He, X.; Ren, Z.; Shi, C.; Fang, J. A novel load balancing strategy of software-deﬁned cloud/fog networkingin the Internet of vehicles. China Commun. 2016, 13, 140–149. [CrossRef]91. Skarlat, O.; Schulte, S.; Borkowski, M.; Leitner, P. Resource provisioning for IoT services in the fog.In Proceedings of the 2016 IEEE 9th Conference on Service-Oriented Computing and Applications (SOCA),Macau, China, 4–6 November 2016; pp. 32–39.92. Dong, Y.; Guo, S.; Liu, J.; Yang, Y. Energy-Efﬁcient Fair Cooperation Fog Computing in Mobile EdgeNetworks for Smart City. IEEE Internet Things J. 2019, 6, 7543–7554. [CrossRef]93. Wang, R.; Li, R.; Wang, P.; Liu, E. Analysis and Optimization of Caching in Fog Radio Access Networks.IEEE Trans. Veh. Technol. 2019, 68, 8279–8283. [CrossRef]94. Jiang, Y.; Huang, W.; Bennis, M.; Zheng, F. Decentralized Asynchronous Coded Caching Design andPerformance Analysis in Fog Radio Access Networks. IEEE Trans. Mob. Comput. 2019. [CrossRef]95. Abouaomar, A.; Elmachkour, M.; Kobbane, A.; Tembine, H.; Ayaida, M. Users-Fogs association within acache context in 5G networks: Coalition game model. In Proceedings of the 2018 IEEE Symposium onComputers and Communications (ISCC), Natal, Brazil, 25–28 June 2018; pp. 14–19.96. Chen, D.; Schedler, S.; Kuehn, V. Backhaul trafﬁc balancing and dynamic content-centric clustering for thedownlink of Fog Radio Access Network. In Proceedings of the 2016 IEEE 17th International Workshop onSignal Processing Advances in Wireless Communications (SPAWC), Edinburgh, UK, 3–6 July 2016; pp. 1–5.97. Park, S.H.; Simeone, O.; Shamai, S. Joint optimization of cloud and edge processing for fog radio accessnetworks. In Proceedings of the 2016 IEEE International Symposium on Information Theory (ISIT), Barcelona,Spain, 10–15 July 2016; pp. 315–319.98. Peng, X.; Shen, J.C.; Zhang, J.; Letaief, K.B. Joint data assignment and beamforming for backhaullimited caching networks. In Proceedings of the 2014 IEEE 25th Annual International Symposium onPersonal, Indoor, and Mobile Radio Communication (PIMRC), Washington, DC, USA, 2–5 September 2014;pp. 1370–1374.99. Dai, B.; Yu, W. Joint user association and content placement for cache-enabled wireless access networks.In Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), Shanghai, China, 20–25 March 2016; pp. 3521–3525.100. Chen, D.; Kuehn, V. Weighted max-min fairness oriented load-balancing and clustering for multicastcache-enabled F-RAN. In Proceedings of the 2016 9th International Symposium on Turbo Codes and IterativeInformation Processing (ISTC), Brest, France, 5–9 September 2016; pp. 395–399.101. Wang, T.; Liang, Y.; Jia, W.; Arif, M.; Liu, A.; Xie, M. Coupling resource management based on fog computingin smart city systems. J. Netw. Comput. Appl. 2019, 135, 11–19. [CrossRef]102. Zhou, Z.; Liu, P.; Feng, J.; Zhang, Y.; Mumtaz, S.; Rodriguez, J. Computation resource allocation andtask assignment optimization in vehicular fog computing: A contract-matching approach. IEEE Trans.Veh. Technol. 2019, 68, 3113–3125. [CrossRef]103. Javaid, S.; Javaid, N.; Saba, T.; Wadud, Z.; Rehman, A.; Haseeb, A. Intelligent resource allocation in residentialbuildings using consumer to fog to cloud based framework. Energies 2019, 12, 815. [CrossRef]104. Tajalli, S.Z.; Tajalli, S.A.M.; Kavousi-Fard, A.; Niknam, T.; Dabbaghjamanesh, M.; Mehraeen, S. A SecureDistributed Cloud-Fog Based Framework for Economic Operation of Microgrids. In Proceedings of the 2019IEEE Texas Power and Energy Conference (TPEC), College Station, TX, USA, 7–8 February 2019; pp. 1–6.105. Barros, E.B.C.; Dionísio Machado Filho, L.; Batista, B.G.; Kuehne, B.T.; Peixoto, M.L.M. Fog ComputingModel to Orchestrate the Consumption and Production of Energy in Microgrids. Sensors 2019, 19, 2642.[CrossRef] [PubMed]36Sensors 2019, 19, 4807106. Jie, Y.; Li, M.; Guo, C.; Chen, L. Game-theoretic online resource allocation scheme on fog computing formobile multimedia users. China Commun. 2019, 16, 22–31.107. Zhou, Y.; Shen, Q.; Dong, M.; Ota, K.; Wu, J. Chaos-Based Delay-Constrained Green SecurityCommunications for Fog-Enabled Information-Centric Multimedia Network. In Proceedings of the 2019IEEE 89th Vehicular Technology Conference (VTC2019-Spring), Kuala Lumpur, Malaysia, 28 April–1 May2019; pp. 1–6.108. Gope, P. LAAP: Lightweight Anonymous Authentication Protocol for D2D-Aided Fog Computing Paradigm.Comput. Secur. 2019, 86, 223–237. [CrossRef]109. Li, Z.; Chen, J.; Zhang, Z. Socially Aware Caching in D2D Enabled Fog Radio Access Networks. IEEE Access2019, 7, 84293–84303. [CrossRef]110. Tao, M.; Ota, K.; Dong, M. Foud: Integrating fog and cloud for 5G-enabled V2G networks. IEEE Netw. 2017,31, 8–13. [CrossRef]111. Ma, M.; He, D.; Wang, H.; Kumar, N.; Choo, K.K.R. An Efﬁcient and Provably-Secure Authenticated KeyAgreement Protocol for Fog-Based Vehicular Ad-Hoc Networks. IEEE Internet Things J. 2019, 6, 8065–8075.[CrossRef]112. Pereira, J.; Ricardo, L.; Luís, M.; Senna, C.; Sargento, S. Assessing the reliability of fog computing for smartmobility applications in VANETs. Future Gener. Comput. Syst. 2019, 94, 317–332. [CrossRef]113. Wang, D.; Liu, Z.; Wang, X.; Lan, Y. Mobility-Aware Task Ofﬂoading and Migration Schemes in FogComputing Networks. IEEE Access 2019, 7, 43356–43368. [CrossRef]114. Chen, Y.S.; Tsai, Y.T. A mobility management using follow-me cloud-cloudlet in fog-computing-based RANsfor smart cities. Sensors 2018, 18, 489. [CrossRef]115. Kadhim, A.J.; Seno, S.A.H. Energy-efﬁcient multicast routing protocol based on SDN and fog computing forvehicular networks. Ad Hoc Netw. 2019, 84, 68–81. [CrossRef]116. Muthanna, A.; Ateya, A.A.; Khakimov, A.; Gudkova, I.; Abuarqoub, A.; Samouylov, K.; Koucheryavy,A. Secure IoT Network Structure Based on Distributed Fog Computing, with SDN/Blockchain. J. Sens.Actuator Netw. 2019, 8, 15. [CrossRef]117. Islam, N.; Faheem, Y.; Din, I.U.; Talha, M.; Guizani, M.; Khalil, M. A blockchain-based fog computingframework for activity recognition as an application to e-Healthcare services. Future Gener. Comput. Syst.2019, 100, 569–578. [CrossRef]118. Tang, W.; Zhang, K.; Zhang, D.; Ren, J.; Zhang, Y.; Shen, X.S. Fog-Enabled Smart Health: Toward Cooperativeand Secure Healthcare Service Provision. IEEE Commun. Mag. 2019, 57, 42–48. [CrossRef]119. Moustafa, N. A Systemic IoT-Fog-Cloud Architecture for Big-Data Analytics and Cyber Security Systems:A Review of Fog Computing. arXiv 2019, arXiv:1906.01055.120. Yassine, A.; Singh, S.; Hossain, M.S.; Muhammad, G. IoT big data analytics for smart homes with fog andcloud computing. Future Gener. Comput. Syst. 2019, 91, 563–573. [CrossRef]121. Nasir, M.; Muhammad, K.; Lloret, J.; Sangaiah, A.K.; Sajjad, M. Fog computing enabled cost-effectivedistributed summarization of surveillance videos for smart cities. J. Parallel Distrib. Comput. 2019,126, 161–170. [CrossRef]122. Mora, L.; Bolici, R.; Deakin, M. The ﬁrst two decades of smart-city research: A bibliometric analysis.J. Urban Technol. 2017, 24, 3–27. [CrossRef]123. Luthra, M.; Koldehofe, B.; Steinmetz, R. Transitions for Increased Flexibility in Fog Computing: A CaseStudy on Complex Event Processing. Inform. Spektrum 2019, 42, 244–255. [CrossRef]124. Naranjo, P.G.V.; Pooranian, Z.; Shojafar, M.; Conti, M.; Buyya, R. FOCAN: A Fog-supported smart citynetwork architecture for management of applications in the Internet of Everything environments. J. ParallelDistrib. Comput. 2019, 132, 274–283. [CrossRef]125. Giang, N.K.; Lea, R.; Leung, V.C. Developing applications in large scale, dynamic fog computing: A casestudy. Softw. Pract. Exp. 2019, doi: 10.1002/spe.2695. [CrossRef]126. Tang, B.; Chen, Z.; Hefferman, G.; Wei, T.; He, H.; Yang, Q. A hierarchical distributed fog computingarchitecture for big data analysis in smart cities. In Proceedings of the ASE BigData & SocialInformatics2015, Kaohsiung, Taiwan , 7–9 October 2015; p. 28.127. Dsouza, C.; Ahn, G.J.; Taguinod, M. Policy-driven security management for fog computing: Preliminaryframework and a case study. In Proceedings of the 2014 IEEE 15th International Conference on InformationReuse and Integration (IEEE IRI 2014), Redwood City, CA, USA, 13–15 August 2014; pp. 16–23.37Sensors 2019, 19, 4807128. Aamir, M.; Masroor, S.; Ali, Z.A.; Ting, B.T. Sustainable Framework for Smart Transportation System: A CaseStudy of Karachi. Wirel. Pers. Commun. 2019, 106, 27–40. [CrossRef]129. Baccarelli, E.; Cordeschi, N.; Mei, A.; Panella, M.; Shojafar, M.; Stefa, J. Energy-efﬁcient dynamic trafﬁcofﬂoading and reconﬁguration of networked data centers for big data stream mobile computing: Review,challenges, and a case study. IEEE Netw. 2016, 30, 54–61. [CrossRef]130. Azimi, I.; Pahikkala, T.; Rahmani, A.M.; Niela-Vilén, H.; Axelin, A.; Liljeberg, P. Missing data resilientdecision-making for healthcare IoT through personalization: A case study on maternal health. Future Gener.Comput. Syst. 2019, 96, 297–308. [CrossRef]131. Kumari, A.; Tanwar, S.; Tyagi, S.; Kumar, N. Fog computing for Healthcare 4.0 environment: Opportunitiesand challenges. Comput. Electr. Eng. 2018, 72, 1–13. [CrossRef]132. Ray, P.P.; Dash, D.; De, D. Edge computing for Internet of Things: A survey, e-healthcare case study andfuture direction. J. Netw. Comput. Appl. 2019, 140, 1–22. [CrossRef]133. Gia, T.N.; Jiang, M.; Rahmani, A.M.; Westerlund, T.; Liljeberg, P.; Tenhunen, H. Fog computing in healthcareinternet of things: A case study on ecg feature extraction. In Proceedings of the 2015 IEEE InternationalConference on Computer and Information Technology; Ubiquitous Computing and Communications;Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing, Liverpool, UK,26–28 October 2015; pp. 356–363.134. Jalali, F.; Vishwanath, A.; De Hoog, J.; Suits, F. Interconnecting Fog computing and microgrids for greeningIoT. In Proceedings of the 2016 IEEE Innovative Smart Grid Technologies-Asia (ISGT-Asia), Melbourne, VIC,Australia, 28 November–1 December 2016; pp. 693–698.135. Beligianni, F.; Alamaniotis, M.; Fevgas, A.; Tsompanopoulou, P.; Bozanis, P.; Tsoukalas, L.H. An internetof things architecture for preserving privacy of energy consumption. In Proceedings of the MediterraneanConference on Power Generation, Transmission, Distribution and Energy Conversion (MedPower 2016),Belgrade, Serbia, 6–9 November 2016.136. Rao, L.; Liu, X.; Ilic, M.D.; Liu, J. Distributed coordination of internet data centers under multiregionalelectricity markets. Proc. IEEE 2012, 100, 269–282.137. Pu, L.; Chen, X.; Xu, J.; Fu, X. D2D fogging: An energy-efﬁcient and incentive-aware task ofﬂoadingframework via network-assisted D2D collaboration. IEEE J. Sel. Areas Commun. 2016, 34, 3887–3901.[CrossRef]138. Wang, S.; Huang, X.; Liu, Y.; Yu, R. CachinMobile: An energy-efﬁcient users caching scheme for fogcomputing. In Proceedings of the 2016 IEEE/CIC international conference on communications in China(ICCC), Chengdu, China, 27–29 July 2016; pp. 1–6.139. Yang, F.; Wang, S.; Li, J.; Liu, Z.; Sun, Q. An overview of internet of vehicles. China Commun. 2014, 11, 1–15.[CrossRef]140. Anawar, M.R.; Wang, S.; Azam Zia, M.; Jadoon, A.K.; Akram, U.; Raza, S. Fog computing: An overview ofbig IoT data analytics. Wirel. Commun. Mob. Comput. 2018, 2018, 7157192. [CrossRef]c© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open accessarticle distributed under the terms and conditions of the Creative Commons Attribution(CC BY) license (http://creativecommons.org/licenses/by/4.0/).38sensorsArticleNarrowband Internet of Things (NB-IoT):From Physical (PHY) and Media Access Control(MAC) Layers PerspectivesCollins Burton Mwakwata 1,*, Hassan Malik 1, Muhammad Mahtab Alam 1,Yannick Le Moullec 1, Sven Parand 2 and Shahid Mumtaz 31 Thomas Johann Seebeck Department of Electronics, Tallinn University of Technology (TalTech),Ehitajate tee-5, 19086 Tallinn, Estonia; hassan.malik@taltech.ee (H.M.);muhammad.alam@taltech.ee (M.M.A.); yannick.lemoullec@taltech.ee (Y.L.M.)2 Telia Estonia Ltd., 10616 Tallinn, Estonia; sven.parand@telia.ee3 Instituto de Telecomunicações, 1049-001 Aveiro, Portugal; smumtaz@av.it.pt* Correspondence: collins.burton@taltech.ee; Tel.: +372-58-663569Received: 27 April 2019; Accepted: 5 June 2019; Published: 8 June 2019Abstract: Narrowband internet of things (NB-IoT) is a recent cellular radio access technologybased on Long-Term Evolution (LTE) introduced by Third-Generation Partnership Project (3GPP)for Low-Power Wide-Area Networks (LPWAN). The main aim of NB-IoT is to support massivemachine-type communication (mMTC) and enable low-power, low-cost, and low-data-ratecommunication. NB-IoT is based on LTE design with some changes to meet the mMTC requirements.For example, in the physical (PHY) layer only single-antenna and low-order modulations aresupported, and in the Medium Access Control (MAC) layers only one physical resource blockis allocated for resource scheduling. The aim of this survey is to provide a comprehensive overviewof the design changes brought in the NB-IoT standardization along with the detailed researchdevelopments from the perspectives of Physical and MAC layers. The survey also includes anoverview of Evolved Packet Core (EPC) changes to support the Service Capability Exposure Function(SCEF) to manage both IP and non-IP data packets through Control Plane (CP) and User Plane (UP),the possible deployment scenarios of NB-IoT in future Heterogeneous Wireless Networks (HetNet).Finally, existing and emerging research challenges in this direction are presented to motivate futureresearch activities.Keywords: narrowband; IoT; PHY; NB-IoT; MAC; deployment; survey; mMTC; 5G1. IntroductionAccording to Information Handling Services (IHS) technology forecast, the Internet of Things(IoT) market is expected to grow to billions of devices by 2020 [1]. Massive connections are expectedto respond to different IoT use cases such as smart city, smart wearables, smart home, etc. [2].For these applications, latency-insensitive devices can be positioned in hard-to-reach areas and donot require high throughput or frequent reporting. Therefore, to cope with such tremendous IoTtrends, the Third-Generation Partnership Project (3GPP) introduced the Narrowband Internet ofThings (NB-IoT) standard as a communication technology enabler. NB-IoT is categorized as one ofthe licensed Low-Power Wide-Area Networks (LPWAN) cellular technologies based on Long-TermEvolution (LTE) with long range and low cost. In the LPWAN category, there exist other licensedtechnologies, i.e., Long-Term Evolution Category M1 (LTE-M), and unlicensed technologies, i.e., LongRange (LoRa), SigFox, Ingenu, etc. [3–7], but they are not the focus of the current work since they arenot based on cellular technology.Sensors 2019, 19, 2613; doi:10.3390/s19112613 www.mdpi.com/journal/sensors39Sensors 2019, 19, 2613The term Narrowband refers to NB-IoT’s bandwidth of maximum 200 kHz thanks to which itcan coexist either in the Global System for Mobile Communications (GSM) spectrum or by occupyingone of the legacy LTE Physical Resource Blocks (PRBs) as in-band or as guard-band. Since it coexistsin the LTE spectrum, NB-IoT follows the legacy LTE numerologies as it uses Orthogonal FrequencyDivision Multiplexing (OFDM) and Single-Carrier Frequency Division Multiple Access (SC-FDMA)in the downlink and uplink transmission schemes, respectively. Some modiﬁcations in the physical(PHY) and medium access control (MAC) layers are implemented to support the long-range massivemachine-type (mMTC) connections with low power, low data rates, low complexity, and hencelow cost. However, despite its low complexity, this new radio access technology (RAT) deliversbetter performance in terms of the supported number of devices, and coverage enhancements forlatency-insensitive applications with maximum coupling loss (MCL) of about 20 dB higher than LTE(i.e., 164 dB) [5–11].With ﬂexible deployment as well as the possibility to implement over-the-air (OTA) ﬁrmwareupgrades, many telecommunication operators across the globe (as shown in Figure 1) deployedNB-IoT to test its practical feasibility on diverse use cases with real-life trials such as connected sheepin Norway [12], smart metering and tracking in Brazil [13], NB-IoT at sea in Norway [14], smart city inLas Vegas, USA [15], etc. The trials are enabled by different NB-IoT software and hardware solutionsfrom different chip or module vendors such as Skyworks [16], Media tek [17], Neul (Huawei) [18],Quectel [19], Nordic Semiconductors [20], Intel [21], Sequans [22], Qualcomm [23], Siera wireless [24],Samsung [25], Altair [26], U-Blox [27], and so on.????????????????? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????Figure 1. The geographical representation of countries with the ongoing NB-IoT real-life deploymentsfor diverse use cases (May 2019).40Sensors 2019, 19, 2613The availability of such commercial off-the-shelf solutions speeds up the adoption of NB-IoT.For this reason, numerous studies addressing segmented enhancement criteria including surveyarticles emerged to analyze NB-IoT performance and implementation. Table 1 presents, in a nutshell,the main differences and similarities between this survey and the other existing ones by displaying thekey focus features.Table 1. Summarized comparison of this survey’s contribution with respect to the existing surveys.Survey The Third Generation Partnership Project Layers Deployment Strategies[Ref] Rel 13 Rel 14 Rel 15 Rel 16 Physical Media Access Control[28] 2017 [29] 2017 [30] 2017 [31] 2017 [32] 2018 [33] 2019 This survey For example, in [32], the authors surveyed the development path of MTC and elaborated theNB-IoT evolution in Release 13. Similarly, in [28], the authors discussed the Release 13 features andcompared its performance with respect to other communication technologies such as LTE-M, SigFox,Lora and Wireless-Fidelity (WiFi), etc. In [29,30], the authors gave an overview of NB-IoT Release 14;however, in [30], the authors elaborated more on the expectations for NB-IoT Release 15 agenda.In [31], the authors presented a survey on the NB-IoT downlink scheduling issues by highlightingthe associated scheduling process in terms of offset index selection. In [33], the authors surveyedthe uplink and downlink performance evaluation of NB-IoT systems by analyzing the main causesof latency, trade-off between throughput and free resources, channel occupancy etc. with respect toRelease 13 and Release 14 updates.In contrast to the above surveys, this paper presents:• A comprehensive survey of NB-IoT, from Release 13 to the ongoing Release 16 prospects.• An all-inclusive overview of the state of the art of PHY and MAC layers by addressing the keyimprovement concerns in terms of challenges and the corresponding potential solutions.• The possible NB-IoT deployment strategies for synchronous and asynchronous network structuresin HetNet scenarios to foster the NB-IoT coexistence with legacy technologies as well as with theﬁfth generation (5G) networks.• Discussion on the open research challenges to motivate future research directions.To the best of the authors’ knowledge, this is the ﬁrst survey that covers broadly theseabove-mentioned contributions and hence will facilitate the reader’s knowledge related to NB-IoTfrom standardization, ongoing research, and its practical implementation.The rest of this paper is organized as follows: Section 2 discusses NB-IoT standards by elaboratingthe key design changes and the related ongoing enhancements. Section 3 presents the state of the artof NB-IoT protocol stack by detailing the PHY layer and MAC layer features. Section 4 discusses theopen research questions and their potential solutions, and the conclusion is drawn in Section 5.2. Narrowband-IoT Standard and ReleasesEarly in 2014, the LPWAN market rapidly developed thanks to the emergence of IoT. Realizingthe need and potential for new communication ways, 3GPP started a feasibility study on cellularsystem support for an ultra-low complexity and low throughput IoT solution referred to as cellularIoT. In May 2014, Huawei and Vodafone proposed the Narrowband Machine to Machine (NB-M2M) to3GPP as a study item to cope with the IoT market needs. Additional telecom industrial players gotinterested and later the same year Qualcomm proposed narrowband orthogonal frequency division41Sensors 2019, 19, 2613multiplexing (NB-OFDM). In May 2015, 3GPP merged the two proposals (i.e., NB-M2M and NB-OFDM)and formed the Narrowband Cellular IoT (NB-CIoT). Eight months later, Ericsson proposed theNarrowband Long-Term Evolution NB-LTE. In September 2015, 3GPP included all proposals as a workitem for Release 13. The key difference between NB-CIoT and NB-LTE was the number of the reusedlegacy LTE network resources to support interoperability. In June 2016 NB-IoT was recognized asa new clean slate RAT. Only further improvement changes were allowed and implemented thereafter.In this regard, this section presents the main NB-IoT design changes from Release 13 untiltoday that enabled the massive IoT connections with the corresponding solutions to respond to theadopted NB-IoT objectives. The enhancement features are classiﬁed following the objectives thatare presented in the releases which would make it easier for the readers to refer back to the ofﬁcial3GPP documents [8,9,34–38].2.1. Release 133GPP introduced the following techniques in NB-IoT Release 13 to enable cellular massive IoTdeployment for diverse use cases with low power, low complexity, and hence low cost. The introducedfeatures and their corresponding objectives are as follows.2.1.1. Mode of OperationWith the limited bandwidth requirement, NB-IoT can be deployed in three different modes i.e.,standalone, in-band, and guard-band, as depicted in Figure 2. In in-band and guard-band modes,NB-IoT occupies one PRBs of 180 KHz in LTE spectrum both in the downlink and uplink. It can also beallocated as standalone where it occupies the 200 KHz bandwidth by “refarming” the GSM spectrum.These ﬂexible deployment possibilities enable fast integration and coexistence with legacy LTE andGSM systems.??????????????????????????? ??????????????????????????????????????????????????????????????????????????????????? ????????????????????????Figure 2. Narrow band Interet of Things (NB-IoT) Flexible Allocation inside Long-Term Evolution(LTE) spectrum (in-band and guard-band) and when refarming the Global System for MobileCommunications (GSM) spectrum (standalone).2.1.2. Multi-Tone Transmission SupportTo reach the massive device deployment objective, NB-IoT introduces the allocation of ResourceUnits (RU) to multiple User Equipment (UE) contrary to LTE where the whole resource block is42Sensors 2019, 19, 2613allocated to a single UE in the uplink. In this regard, tones (frequency domain) with different durationare allocated to UEs. For the uplink transmission, each tone may either occupy 3.75 kHz or 15 kHzof transmission bandwidth based on the SC-FDMA scheme; for downlink NB-IoT uses 15 kHz oftransmission bandwidth with OFDM scheme as LTE. With 15 kHz spacing, NB-IoT can dedicate eithersingle-tone (8 ms) or multi-tone (3 tones, 6 tones, and 12 tones) to different UEs with the duration of4 ms, 2 ms, and 1 ms, respectively. On the other hand, the 3.75 kHz spacing supports only single-toneallocation to different users with 48 subcariers of 32 ms duration [11,39,40].2.1.3. Complexity and Cost Reduction TechniquesNB-IoT is required to have low complexity to reach the low-cost objective to facilitate massiveconnections. The features that were implemented to reach this objective include relaxed base-bandprocessing, low memory storage, and reduced radio-frequency (RF) components. In this regard,the system bandwidth is set as narrow as 180 kHz with reduced frequency and time synchronizationrequirement. Also, NB-IoT uses the restricted BPSK and QPSK modulation schemes with only oneantenna support both in uplink and downlink transmission.2.1.4. Power Reduction MethodNB-IoT devices are intended to have a 10 years battery life to support massive deployment withlimited human intervention. In this regard, two features i.e., Power Saving Mode (PSM), (from Release12), and extended Discontinuous Reception (eDRx) (new feature from Release 13) were supported.These features are intended to extend the UE’s battery longevity as follows:In PSM, the NB-IoT device is conﬁgured to completely sleep while remaining registered onlinebut cannot be reached by the base station signaling. In Release 13, the device can be in PSM mode forapproximately up to about 413 days. In eDRX, the device is in an inactive mode for a few minutes toa few hours only.In both cases, the partial or complete inability to receiving and sending different signals enhancethe battery life longevity; however, choosing either PSM, eDRX or both depends on the correspondinguse-case requirement. In this regard, the device can be synchronized to wake up from these modes byeither Real-Time Clock (RTC), triggering from sensors, or both.2.1.5. Physical Channels and SignalsNB-IoT adopts the same frame structure as LTE, with 1024 hyper frames, consisting of 1024 framesthat contain 10 subframes of two slots with a duration of 0.5 ms each in the time domain. Similarly,in the frequency domain, NB-IoT contains 12 subcarriers of 7 OFDM symbols mapped in each slot.In addition to that, when NB-IoT uses the 3.75 kHz spacing on the uplink, 48 subcarriers are used witha slot duration of 2 ms.The following channels and signals are used in the uplink:• Narrowband Physical Random Access Channel (NPRACH).• Narrowband Physical Uplink Shared Channel (NPUSCH).• Demodulation Reference Signal (DMRS).And the following are in the downlink frame:• Narrowband Physical Downlink Shared Channel (NPDSCH).• Narrowband Physical Downlink Control Channel (NPDCCH).• Narrowband Reference Signal (NRS).• Narrowband Primary Synchronization Signal (NPSS).• Narrowband Secondary Synchronization Signal (NSSS).• Narrowband Physical Broadcast Channel (NPBCH).43Sensors 2019, 19, 2613In general, NPRACH is used by UEs to perform initial access to the network, to requesttransmission resources, and to reconnect to the base station after a link failure. NPDSCH and NPUSCHare used to carry the downlink and uplink data packets transmissions, respectively. DMRS is usedfor uplink channel estimation accuracy. The UE acquires Master Information Block (MIB) fromNPBCH and System Information Block (SIBs) from the NPDCCH. The deﬁned MIB and SIB arebroadcasted once during 640 ms and 2560 ms intervals, respectively. The timing of the remainingSIBs is conﬁgured in SIB1-NB. NRS is used for cell search and initial system acquisition. NPSS andNSSS are used by the UE for its frequency and timing synchronization with the base station. Due tooverhead scheduling gaps in NPDCCH, the downlink and uplink peak data rates are ~250 kb/s and~2267 kb/s, respectively, [34,40–43].2.1.6. Coverage Enhancement MethodNB-IoT is designed to enhance coverage for the applications that are in hard-to-reach areas suchas deep indoors and basements. In this regard, NB-IoT delivers an additional coverage of 20 dB ascompared to the legacy LTE system. This corresponds to 164 dB of MCL. To enhance its coverage,NB-IoT uses up to 128 and 2048 retransmissions in uplink and downlink, respectively. Hence, thismakes NB-IoT suitable for use cases that are latency insensitive as it can tolerate up to 10 secondstransmission delay.2.2. Release 14 EnhancementsAfter the implementation of Release 13 features, studies erupted along with ﬁeld trials thatrevealed the need for further enhancements to improve the quality of service as well as user experience.In this regard, 3GPP introduced further enhancement features to NB-IoT.The enhancements features in Release 14 include positioning update, multicast services, and a newUE output power class in which the NB-IoT system throughput, mobility, service continuity andnon-anchor carrier operation are improved [29,30].2.2.1. Improved Positioning Technique3GPP Release 14 introduces an indoor advanced positioning method of observed time differenceof arrival (OTDOA) for NB-IoT to enhance UE position measurement of cell identity (CID). In OTDOAmethod, the UE measures the times of arrival (ToAs) of positioning reference signals (PRSs) receivedfrom different transmitters with respect to a reference node’s PRS transmission to form the referencesignal time difference (RSTD) measurements. In enhanced CID, the measurement requirements includethe base station receive (Rx) and transmit (Tx) time difference, reference signal received power (RSRP),and reference signal received quality (RSRQ).2.2.2. Multicast ServicesThe main objective of this mechanism is to optimize resources as well as transmission latencyby addressing the data to a group of UEs at the same time rather than sending it multiple times toseparate devices.Therefore in Release 14, Multimedia Broadcast Multicast Services (MBMS) is supported throughsingle-cell point-to-multipoint (SC-PTM). In general, SC-PTM is an efﬁcient dynamic mechanism foroptimal radio resource usage as it allows broadcast or multicast services to a speciﬁc group based onreal-time trafﬁc load and user requirement. SC-PTM uses NPDSCH by mapping Single-cell MBMSControl CHannel (SC-MCCH) and Single-Cell MBMS Trafﬁc CHannel (SC-MTCH) that carry controland data trafﬁc to the physical layer scheduled by using the downlink control information (DCI).44Sensors 2019, 19, 26132.2.3. New Power Class for Narrowband-IoT User EquipmentInstead of the two power classes of Release 13 (i.e., 20 dBm and 23 dBm), in Release 14,the maximum allowed device’s output power is reduced to 14 dBm. This has led to coveragerelaxation of 9 dB that corresponds to 155 dB MCL as compared to 164 dB MCL and hence reducesthe drained current. Technically, the use of the new power class facilitates the use of small coin-cellbatteries and hence can be suitable for limited-size devices and applications that need a small battery.The compensation of the reduced NB-IoT power is achieved by increasing the NB-IoT transmissiontime to maintain the same energy per bit as the UE in Release 13 achieves. The newly introducedpower class allows the serving base station to acquire the device power class during the establishmentof the connection.2.2.4. New Transport-Block-Size SupportContrary to Release 13 where NB-IoT supports relatively low data rates (~250 kb/s and ~226.7 kb/sin downlink and uplink, respectively), 3GPP Release 14 introduces a new NB-IoT device category whichsupports the improved data rates by enhancing the Transport Block Size (TBS) to 2536 bits. These datarates can be reached thanks to the ability to support a second Hybrid Automatic Repeat Request(HARQ) process. This second HARQ is useful for enhancing the reliability of the link for the UEs thatexperience favorable channel conditions. Implementation of this optional second HARQ process resultsin throughput gain as it reduces the overhead caused by NPDCCH scheduling gaps.2.2.5. Multicarrier OperationTo enable the massive NB-IoT deployment, in Release 14, NB-IoT can monitor paging and performrandom access on non-anchor carriers. With this feature, one or more non-anchor carriers are addedto the anchor carrier to carry out the synchronization and mobility measurements by using the NRS.Non-anchor carriers should also perform random access or paging when needed. Therefore, pagingoccasions and hence paging load will be spread over the anchor and non-anchor carriers and allcarriers can then monitor paging.2.2.6. User Equipment Mobility EnhancementFor the use cases that involve mobility, the temporary loss of radio interface impacts the system toa degree that can degrade link performance in terms of transmission errors. In this regard, 3GPP Release14 introduces the possibility of Radio Resource Control (RRC) re-establishment for NB-IoT UE thatsupports data transfer via the control plane, i.e., the UE will try to re-establish the connection on thatcell and resume the data transfer. This new RRC re-establishment feature hides the temporary loss ofthe radio interface to the upper layers.2.3. Release 15 EnhancementsOn top of all the enhancements that were introduced in Releases 13 and 14, the followingimprovements were introduced in Release 15 to satisfy the fast adoption of massive deployment withfurther improved quality of service.2.3.1. Latency ReductionIn Release 15, NB-IoT supports new features to further reduce the transmission delay as well as tofurther reduce the power consumption dissipated during long transmission requirements.In this regard, the NB-IoT UE is now able to support the physical layer Scheduling Request (SR)which is a special physical layer message to request the network to send the access grant (DCI format0) so that the UE can transmit the uplink data. Also, NB-IoT uses a wake-up (Wu) signal to wake upthe main receiver. This signal is transmitted in idle mode only when the UE is required to decodethe physical downlink control channel in paging occasions. Therefore, power consumption reduction45Sensors 2019, 19, 2613with the wake-up signal technique is larger when the UE wakes up from deep sleep more frequently(i.e., for shorter DRX/eDRX cycles). Also, signiﬁcant power consumption reduction is achievedeven when a common wake-up signal is used for a group of UEs. Quick RRC release and earlydata transmission during random access channel (RACH) procedure are supported to reduce the UEtransmission latency and hence power consumption.2.3.2. Semi-Persistent SchedulingTo enable better support of voice messages for the corresponding use cases, in Release 15,Semi-Persistent Scheduling (SPS) feature is introduced. In general, SPS is comprised of persistentscheduling for initial transmissions and dynamic scheduling for retransmissions. The base stationassigns speciﬁc resource units to be used for NB-IoT UE voice messages with speciﬁc interval to savecontrol plane overhead and hence optimize the radio resource usage. By principle, the base stationpreconﬁgures the UE with the Radio Network Temporary Identiﬁer (SPS-RNTI) which is used tospeciﬁcally differentiate one NB-IoT UE from another, or one radio channel from another. This SPSenables the NB-IoT data reception at a regular conﬁgured periodicity.2.3.3. Small Cell SupportTo further improve the capacity as well as coverage, in Release 15, NB-IoT supports small celldeployments. The downlink power to be reused for NB-IoT small cells is speciﬁed in section 16.2.2of TS 36.213 [44]. In general, NB-IoT UE is not allowed to transmit more power than the conﬁguredmaximum power, even if the conﬁgured power is lower than UE’s maximum capability. This is doneto avoid interference.On the other hand, to extend the IoT connectivity especially in remote and rural areas for usecases such as agriculture, logistics, and environmental monitoring, NB-IoT is now able to support upto 100 km range. According to Ericsson, this could be achieved with a software upgrade only, withoutany changes in the existing NB-IoT hardware [45].2.3.4. Enhanced User Equipment MeasurementsLike in legacy LTE systems, UE measurements are critical since the corresponding reporting ismainly used to characterize the reference signal of a given bandwidth.In Release 15, UE measurements are improved in a way that only NSSS additionally to NRSis deﬁned for radio resource management measurement enhancement. This means that NRS isdetermined by the resource elements that carry NSSS in the NSSS occasions that the UE measures,through which the cell search and initial cell acquisition are improved.2.3.5. Time Division Duplex (TDD) SupportIn Release 15, a new feature of TDD support is introduced with a new TDD frame structure (type 2).For both 3.75 kHz and 15 kHz spacing, some speciﬁed restrictions are introduced i.e., only a normalcyclic preﬁx is supported for NB-IoT transmission. To support some of the TDD conﬁgurations withfew downlink subframes, some of the system information (SI) can be transmitted on non-anchorcarriers. In this way, the UE will have reduced system information acquisition and search time,and hence reduced UE differentiation and access control [30,46,47].2.4. Release 16 Enhancement Prospects3GPP and many industrial players are involved in ongoing discussions for Release 16enhancements. The agenda includes the following objectives with their corresponding solutions.46Sensors 2019, 19, 26132.4.1. Grant-Free AccessMost of the power consumption takes place during the NB-IoT UE active time, i.e., during Txand Rx. In Release 16, the UE will be expected to transmit during RRC-Idle mode through Msg3 (RRCconnection request) without access grant. A UE in RRC connected mode can transmit data withoutgrant or with the simpliﬁed control-less grant. A further enhancement is on reducing NB-IoT signalingoverhead while guaranteeing the needed quality of service. These features will reduce both powerconsumption and latency. In Release 16, it is also proposed to further study other signal waveforms(i.e., FDMA) that require less orthogonality with more relaxed timing advance (TA) alignment ascompared to SC-FDMA.2.4.2. Simultaneous Multi-User TransmissionThe introduction of new schemes will enable simultaneous multi-user transmissions by usinga shared resource in the time and frequency domains, such as Code division multiplexing (CDM),and multi-user multiple inputs multiple outputs (MU-MIMO), without increasing the number ofantennae at the UE. In this regard, more dynamic access can also be achieved through enhanced basestation receiver for detection of multiple users that are using the same resource unit as cluster andhence be able to schedule them effectively. This is because, for the last releases, NB-IoT UE uses thestatic or semi-static conﬁguration of more resources for the unexpected application trafﬁc handling.Similarly, the introduction of NB-IoT transmission without grant will cause a collision of data packetsso dynamic handling of multiplexing is necessary.2.4.3. Enhanced Group Message MechanismIn Release 16, there should be more enhancements to support downlink command between usergroups and group RNTIs. This is because MBMS which was proposed in Release 14 is only efﬁcientfor large size downlink command message transmission and requires many UEs to be deployed.For example, the application layer common message can be very small but sent to many UEs undera small group of UEs hence making MBMS not efﬁcient for such applications.2.4.4. Inter-RAT Idle-Mode MobilityFor applications such as smart tracking of logistics that involve mobility, the NB-IoT UE may stillneed to be accessible even when moved to the area served by other base station.In this regard, 3GPP should introduce the new feature for NB-IoT UE support for inter-RATmobility during idle mode. The mentioned feature is introduced along with optional handoversupport during connected mode through procedure simpliﬁcation i.e., without dedicated signalingfor measurement control and report. This is because handover helps to reduce system informationreading time.2.4.5. Network Management Tool Enhancement to Improve UE DifferentiationNB-IoT UE is expected to be able to perform differentiation according to maximal tolerable delayper service to optimize the radio resource usage. This is because, in the last release, the UE can bedifferentiated according to trafﬁc model (periodic communication indicator, periodic time, scheduledcommunication time, trafﬁc proﬁle) and battery indication.Section 2 has presented the NB-IoT standard and the corresponding enhancements from Release 13until today. It has highlighted the main design changes and the corresponding further enhancements,i.e., deployment ﬂexibility, physical channels and signals, positioning, multicast, new power classes,improved data rates, multicarrier operations, mobility support, improved scheduling, NB-IoT smallcell support etc.47Sensors 2019, 19, 26133. Narrowband-IoT: Protocol StackThis section presents the NB-IoT protocol stack based on state of the art of the PHY and MAClayers to identify the knowledge gap and deﬁne future research directions. NB-IoT adopts the sameprotocol stack as the legacy LTE. However, some design changes in both PHY and MAC layers wereintroduced to support the massive long-range connections with up to additional 20 dB MCL than inlegacy technologies such as LTE, GSM, and GPRS. Those changes are described in what follows.3.1. Physical LayerOn the physical layer, NB-IoT adopts the same numerologies as legacy LTE along with OFDM andSC-FDMA signal waveforms in downlink and uplink, respectively. However, the resource schedulingunit in NB-IoT is the subcarrier (or tone) instead of PRB, to foster the network scalability by servingmultiple UEs in a 180 kHz bandwidth. The downlink and uplink frame structures are as depicted inFigures 3 and 4, respectively.?? ?? ?? ?? ?? ?? ?? ?? ?? ??????? ????????? ????????????????????????????????????????????????????? ???????????????????????????????????????????Figure 3. NB-IoT Downlink Frame Structure: subframe number 0 carries the Narrowband PhysicalBroadcast Channel (NPBCH), 1 to 4, and 6 to 8 carry the Narrowband Physical Downlink ControlChannel (NPDCCH)/Narrowband Physical Downlink Shared Channel (NPDSCH), and 5 and 9 carrythe Narrowband Primary Synchronization Signal (NPSS)/Narrowband Secondary SynchronizationSignal (NSSS) (A) When the subframe is carrying control channels and (B) when the subframe iscarrying data.In general, the base station uses DCI to specify the scheduling information for a downlink/uplinktransmission in NB-IoT. Then NB-IoT UE learns the deployment mode (standalone, in-band, orguard-band) as well as the cell identity through its initial acquisition, and it ﬁgures out which resourceelements are already used by LTE. This is the way by which the UE can map NPDCCH and NPDSCHsymbols to available resource elements. For example, in the downlink, NPDCCH is transmitted byaggregating the narrowband control elements (element 0 and element 1) where element 0 is occupied insubcarrier 0 to 5 and element 1 occupies subcarrier 6 to 11 in a subframe. The elements are determinedby the type of DCI which is carried by NPDCCH to deliver scheduling command. Either two DCIs canbe multiplexed in one subframe, or one DCI can be mapped in one subframe, corresponding to theaggregation level used [48]. However, NPDCCH, NPDSCH, and NRS cannot be mapped to the alreadyoccupied resource elements for LTE signals such as cell-speciﬁc reference symbols (CRS) and LTEphysical downlink control channel (PDCCH). When NB-IoT UE receives NPDCCH which carries DCI,it decodes it and uses the device’s scheduling feature (k0) to know the delay over which it will start toreceive NPDSCH. The scheduling information is used to identify the allocated resources over NPDSCH48Sensors 2019, 19, 2613and NPUSCH, respectively. In each NPDCCH, a maximum of two DCIs can be transported, and eachUE can receive up to one DCI. The time interval between two successive NPDCCH opportunities isreferred to as an NPDCCH period (PP) [48].In the state of the art, different works have proposed solutions to the challenges that occur in PHYlayer features, such as initial cell acquisition and synchronization, random access, channel estimation,error correction, and co-channel interference, as summarized in Table 2.????????????????????????????????????????????????????????? ?????????????????????????????????????????????????????? ?????????? ?????????? ????????Figure 4. NB-IoT Uplink Frame Structure, (A) when 15 kHz spacing is used with differenttone-allocation possibilities with slot duration of 0.5 ms and (B) when 3.75 kHz is used only single-toneallocation is supported with 4 times longer slot duration (2 ms).Table 2. Articles on the proposed PHY layer enhancement techniques.Feature Article Technique Used Enhancement Criteria LimitationCellAcquisition[49] Maximum-Likelihood (ML)NPSS detectorAverage latency reductionfor timing synchronizationIt is a computationallycomplex detection method[50] Cell search and initialsynchronization algorithmTime and frequencysynchronization by usingNPSS and NSSS withtwo-stage time domainNPSS correlationmobility and new NB-IoTtransmit power are notconsidered which havea direct impact on inter-RATcamping and the detectedSNR, respectively[51]Non-orthogonal spectralefﬁcient frequency divisionmultiplexing (SEFDM)waveform and anoverlapped sphere decoding(OSD) detectorResource optimization bythe use of less bandwidthwith better data ratescompared to OFDM signalwaveformThe proposed methodwould lead to sampling ratemismatch, carrier frequencyoffset and also will need toraise the computationcomplexity to NB-IoT UE[52]New synchronization signalstructure with Zadoff-ChuconjugatesMinimization of timingerrors due tolow-complexity NB-IoTfrequency offsetIf the same model is usedfor uplink synchronization itmight lead to estimationerrors if mobility is involvedin NB-IoT[53]NPRACH detection andtime-of-arrival estimationfor NB-IoT systemEnhancement on cellacquisition and channelestimation accuracyThe algorithm might notwork for multi-toneallocation. Also, frequencyhopping may raise powerconsumption as well asdevice complexity[54]Receiver algorithm forNPRACH timing advanceestimation and detectionModeling the detectionthreshold to satisfy theNPRACH performance bylowering the probabilities offalse alarmThe paper did not explainhow receiver sensitivity canaffect the NPRACHdetection49Sensors 2019, 19, 2613Table 2. Cont.Feature Article Technique Used Enhancement Criteria LimitationCellAcquisition[55] Mathematical modeling ofNB-IoT performanceThroughput enhancementand NPRACH optimizationby the use of repetitionnumber, NPRACHpreamble transmission persecond and intersitedistanceThe work did not includesome parameters such as theimpact of mobility and howthe achieved MCL fordifferent coverage classescan impact the repetitionassignment[56] NPSS and NSSS frequencydiversity receptionTime and frequencysynchronization for cellsearch improvementAlternative switching ofNPSS and NSSS may requireadditional controlcommands which may leadto higher device complexityRandom Access[57] Conﬁgurable signalpropagation modelSystem performanceanalysis in terms of numberof supported devices, BERperformance, preambleretransmissions, etc.The impact of preambleretransmission on theoverall transmission latencyis not considered[58]Mathematical evaluation ofRACH preambletransmissionAnalysis of NB-IoTtransmission delay by usingperiodicity, start time,number of repetitions,number of preambleattempts and random accessresponse windowTheir model used minimum,intermediate, and maximumvalues for simulation whichis so deterministic.However, it could be betterto use random distributionto characterize NB-IoTrealistic channel variations[59]Random Access withdifferentiated barring(RADB) algorithmMinimization of randomaccess collisionNot resource efﬁcientmethod since it does notinclude the impact ofscheduling in different toneconﬁgurations[60]New frequency hoppingpattern of NPRACHpreambleTime-of-arrival estimationby the use of all the hoppingdistancesIt only used a small cellscenario, if applied in denseNB-IoT network, estimationby considering all hoppingdistances may lead tosystem overhead andpossible interferenceChannelestimation[61] Frequency trackingalgorithmFrequency synchronization,as well as channelestimation for NB-IoTsystemsMore pilot signals, are used.This increases the overheadand hence can degrade thespectral efﬁciency[62] Timing advance (TA)adjustmentPreamble sequencedecoding by means of roundtrip estimation for coverageenhancements (on the sea)It might not work forapplications that do notinvolve a direct line of sightsuch as in dense urbanenvironment[63] MCS and coverage leveloptimizationMobility effect on differentcoverage levels and howMCS affect pagingperformanceThe channel model does notinclude other factors such asthe effect of repetition,multipath, different Txpower for NB-IoT UEs aswell as carrier frequencyoffset and inter-RAToperability[64] New iterative algorithm forNB-IoT transmission schemeNB-IoT error correction byusing cryptographicredundancy and errorcorrecting codeThe channel estimationmodel to characterizeNB-IoT transmission is notgood, because some errorsmight be due to intersymbolinterference and others dueto intercarrier interferencehowever the model does notexplain50Sensors 2019, 19, 2613Table 2. Cont.Feature Article Technique Used Enhancement Criteria LimitationInterferencemitigation[65] Channel EqualizationalgorithmIntersymbol Interferencemitigation by thephase-shifted channelfrequency responses (CFR)to conquer the samplingmismatch between NB-IoTand base stationThe proposed model did notconsider the NPSS andNSSS impact ON time andfrequency synchronization[66]Mathematical model forsample duration in LTE andNB-IoT systemInterference and close-forminterference analysis due tosampling mismatch betweenNB-IoT and base stationThe model is computationalcomplex when implementedin NB-IoT systems3.1.1. Cell Acquisition and SynchronizationNB-IoT UE goes through the same process as LTE UE where to camp on a cell, it goes throughfrequency and timing synchronization to obtain the center carrier frequency as well as the allocatedslot and frame timing used for the cell acquisition. In general, if MIB and SIB are properly decoded, cellID, a subframe number, scheduling information, and system bandwidth can be detected successfully.In NB-IoT, the low complexity of devices may lead to poor synchronization and cell acquisitionperformance, especially due to carrier frequency offsets and poor channel estimation capacity.The following are the papers that have proposed different solutions to optimize the initial cellacquisition and initial synchronization procedure.In [49], the authors presented a Maximum-Likelihood (ML) NPSS detector which is based onfrequency domain cross-correlation metrics by using an overlap-save method. Their method achievesan average timing synchronization latency of 140 ms for the in-band deployed mode with SNR of−12.6 dB. Their proposed method showed a 34% reduction of the energy that is required for NPSSdetection. However, their work showed only how much energy could be reduced with respectto the autocorrelation NPSS detection methods. It could be better to show how much of the totaldevice’s energy is consumed by their proposed computationally complex detector, i.e., it could bemore realistic to include analysis in terms of reduction with respect to the energy consumed duringtime synchronization but also in terms of energy optimization over the total device consumption.In [50], the authors presented an algorithm for initial synchronization and cell search.The proposed algorithm uses NPSS for timing acquisition and initial Carrier Frequency Offset (CFO)estimation called the two-stage time domain NPSS correlation. They also used NSSS sequences forthe cell ID and frame timing. Their proposed algorithm showed that under extremely low SNR anddifferent fading conditions, NB-IoT could provide the required performance and could also quicklycamp on the cell, if any. However, practical experiments are still needed to prove the feasibility ofthese simulations especially on how the newly introduced NB-IoT power class and actual channelvariations could have an impact on the detected SNR at the base station.In [51], the authors presented an NB-IoT framework by using an advanced signal waveformcalled non-orthogonal spectral efﬁcient frequency division multiplexing (SEFDM). This waveformuses less bandwidth as compared to OFDM waveform. The designed signal could improve the datarate without the need for more bandwidth. At the base station, the minimum Euclidian norm searchdetector is used for better error correction. The simulation results reveal that the proposed advancedsignal waveform could achieve 25% improvement on data rate as compared to the OFDM signalwaveform. The work also proposed an overlapped sphere decoding (OSD) detector which reduces thecomputation complexity as compared to the single sphere decoding detector while guaranteeingthe needed performance. However, the model does not explain the impact of CFO due to thenon-orthogonality of the subcarriers on the received signal.51Sensors 2019, 19, 2613In [52], the work investigated the downlink synchronization signal design and proposedthe novel general synchronization signal structure with a couple of Zadoff-Chu (ZC) conjugatedsequences in order to remove the potential timing errors caused by large frequency offsets. Their newsynchronization signal structure demonstrated better functionality with the frequency offset toleranceof up to 40 kHz. However, the model does not explain the number of samples per symbol involved insynchronization operation and decision.In [53], the random access preamble is discussed based on the design of NPRACH for single-tonefrequency hopping only. It introduces the new single-tone frequency hopping random access signalused by NPRACH in NB-IoT systems. It further explains the design rationale and proposes somepossible receiver algorithms for NPRACH detection and ToA estimation. The simulation results showthat NPRACH performance is improved. However, the paper did not discuss the impact of massiveinterference which may result in lower received Signal Interference plus Noise Ratio (SINR) at the basestation such that the lower the SINR the lower the detection probability of NPRACH. In addition to thatthe higher the number of devices the higher probability of NPRACH preamble collision. So lower SINRdetected at base station and higher collision probability both affect NPRACH detection negatively.In [54], the authors described a NPRACH design as speciﬁed in 3GPP in standard in Release 13.They proposed a receiver algorithm for the NPRACH timing advance estimation as well as detection.The simulation results for the NPRACH detection shows that if one preamble sequence is transmitted,the detection threshold should be set between 55% to 70% of the average value to satisfy the desiredNPRACH performance at the lowest SNR. The results also showed that at 5 and 11 preamble sequencetransmission, the detection threshold should be 50% and 35% of the average value, respectively. It isnoted that increasing the detection threshold lowers the false alarm probability, which leads to anincreased likelihood of misdetection.In [55], the authors provided a mathematical model of an NB-IoT network in order to predictthe optimum performance with a speciﬁc conﬁguration of some design parameters (i.e., repetition,number of the preamble in NPRACH per second, coverage classes and intersite distance). The paperanalyzes the effects of parameter choice in outdoor, indoor, and deep indoor. The work ﬁnally proposeshow to choose the optimal conﬁguration i.e., by providing the highest throughput, as well as successprobability higher than minimal success probability with minimal one being of 90%. The work showedthat even though the success probability has a maximum limit, it can still be altered by modifying thenumber of repetitions to enhance the coverage or the system capacity in terms of throughput.In [56], the authors presented the NB-IoT frequency diversity (FD) reception for NPSS as wellas NSSS. In the reception mode, the NB-IoT UE alternatively receives the NPSS and NSSS in timedomain radio frame by switching the received signals transmitted in different resource blocks in thefrequency domain. Their simulation results show that using the proposed FD reception could improvethe detection probability by 16% more than without applying the frequency diversity. Additionally,using FD with precoding vector switching (PVC) transmit diversity, achieves 90% of physical cell IDdetection (PCID) probabilities at the average SNR of 0 dB with maximum carrier offset of 70 kHz.The method also achieves 97% of PCID detection probability without consideration of frequencycarrier offset.3.1.2. Random Access ProcedureLike in LTE, NB-IoT random access (RA) is intended for initial UE uplink synchronization throughwhich the UE acquires its unique UE ID used for communication with the base station. RA is alsoused to regain the lost UE access due to the long state of inactivity which has led to the loss inuplink synchronization. In NB-IoT, RA faces several challenges as seen on the research discussions;some solutions to improve the RA performance have been proposed as described in what follows.In [57], the authors presented the random access procedure (RAP) model and analyzed the systemperformance by taking into consideration the conﬁgurable signal propagation model, a number ofsupported users per cell, and the RAP conﬁguration parameters. The paper used the contention-based52Sensors 2019, 19, 2613random access with Msg3 collisions instead of Msg1 collision (as multipath transmission) for therandom access procedure. The proposed model results show the impact of the parameters (Msg3transmission mode, Msg3 modulation and coding scheme (MCS), power control schemes and powerramping step) in the single-tone and multi-tone transmissions Bit Error Rate (BER) performance.The results are presented in terms of the total number of preamble transmission success, preambleretransmissions and lost preamble attempts. The work concludes that Msg3 must be considered inthe random access procedure analysis, the transmission mode as well as MCS, and for better systemperformance and fairness distribution of UEs in the cell, it is better to conﬁgure power control correctly.In [58], the authors analyze NB-IoT transmission delay as well as mathematical evaluation of theprobability of success for the random access procedure preamble transmission. The analysis is based onthree scenarios; scenario one uses minimum values of parameters, scenario two uses the intermediatevalues, and scenario three uses maximum parameter values. The used parameters are NPRACHperiodicity, start time, number of repetitions, number of preamble attempts, and random accessresponse window size. The average delay analysis was performed such that k preamble sequences aremapped in n subcarriers. The preamble collision occurs when multiple UEs send preamble sequencesin the same subcarrier. A successful preamble attempt occurs when only one UE sends the preambleto a given subcarrier.In [59], the authors investigated a random access optimization algorithm and summarizedthe NPRACH feature and hence designed random access with differentiated barring (RADB) forNB-IoT system. It is observed that the RADB could solve the preamble request conﬂict caused bymassive NB-IoT UEs and hence provide reliable random access for latency-sensitive devices. However,the authors did not consider the problems of channel resource distribution and resource use rate.In [60], the authors designed a new frequency hopping pattern of NPRACH preamble which usesall feasible hopping distances for a given number of subcarriers. It is seen that their proposed patternwas compatible with standards that is keeping the same NPRACH structure with only very smallchanges (hopping in the standard is allowed only between the subcarriers of the same resource group).Their simulation where they adopted their ﬁrst trafﬁc model which deploys 3000 devices, 48 ms and40 ms of NPRACH preamble and periodicity, respectively, show that the proposed hopping patterncould improve the ToA estimation without additional system overhead.3.1.3. Channel Estimation and Error CorrectionLike in LTE systems, NB-IoT system performance depends to some extent on the quality of thechannel estimation. However, for NB-IoT systems massive deployment, the poor quality of channelestimates is highly inﬂuenced by the low complexity of the UEs that can lead to misdetection ofsome signals, frequency offset, phase noise, passive intermodulation (PIM) on the device level, etc.To address the challenges that affect the channel estimation as well as to improve the quality of errorcorrection to ensure the required performance with the low complexity, several works have proposedsome solutions, as summarized in the following paragraphs.In [61], the authors presented an NPSS detection method whose timing metric is composed ofsymbol-wise autocorrelation and a dedicated normalization factor in an in-band downlink NB-IoTsystem. The authors proposed a novel low-power algorithm for frequency tracking by the use ofmore pilot signals as compared to the LTE system. Their algorithm is implemented to compensatefor the accumulated frequency offset during the NB-IoT transmission of NB-IoT. Their proposedfrequency tracking algorithm delivers high estimation efﬁciency in terms of Minimum Mean SquareError (MMSE), the probability of correct cell acquisition, etc. However, their study did not elaborate onwhat could be the impact of mobility and inter-RACT support in the cell search procedure for NB-IoT.In [62], the authors presented a practical coverage test on the ocean; it is shown that the proposedsolution (where the base station decides whether the compensated round trip delay is short or longenough to decode the preamble sequence) was done by considering NPRACH design and hence theauthors proposed their solution which considers the TA adjustment. Their proposed solution proved53Sensors 2019, 19, 2613that NB-IoT coverage could reach as far as 35 km. However, the paper does not elaborate on thesolution feasibility in environments without line of sight.In [63], the work provided optimization cases for NB-IoT downlink in terms of MCS. The workalso provided the optimization cases of coverage level (CL) by taking into consideration the RACHsuccess rate with different driving speeds of NB-IoT devices in a commercially deployed network.Their results show that the base station paging success rate is decreased as the adjacent cell interferenceincreases. However, the decrease in MCS improves paging performance. Coverage level 0 is the bestchoice for NB-IoT use cases that involve mobility, whereas coverage level 1 and 2 are mostly for ﬁxedlocation NB-IoT use cases.In [64], the author presented an iterative algorithm for NB-IoT transmission procedure.The simulation results in terms of BER and blocks error rate (BLER) show that by use of concatenatederror correcting codes or cryptographic redundancy and error correcting code, the algorithm improvesthe NB-IoT coverage and reduces the overall NB-IoT power consumption. The modification of additionalcorrection of low reliable bits could demonstrate the error correction of the damaged messages by thenoisy transmission and hence can reduce the repetition number. However, this work did not discuss howeffective the algorithm is when taking into consideration different channel conditions, payload sizes, aswell as different repetition numbers with respect to device signal quality.In [67], the authors considered the presence of random phase noise of the received signals mainlycaused by oscillators impairments in both the transmitting and receiving sides and how to lowerthe mean square error (MSE) estimates. They presented the sequential MMSE channel estimationmethod that could be implemented in NB-IoT systems. Their model shows that if random phase noiseis considered during channel estimation, it is possible to improve the detected SNR by up to 1 dB.However, the model is assumed to be uniformly distributed hence does not present the real-timechannel which is randomly changing over time.3.1.4. Co-Channel InterferenceNB-IoT being deployed in the existing LTE spectrum, co-channel interference may occur betweenNB-IoT and LTE UEs. This is due to several reasons such as sampling rate mismatch, inter-PRBinterference due to power leaking between NB-IoT and LTE PRBs, etc. To mitigate the impact ofco-channel interference in the NB-IoT/LTE coexistence scenario, the following works have addressedthe problems and proposed potential solutions.In [65], the authors proposed the design guidance for channel equalization that can be usedfor 5G networks. The proposal set some assumptions such that the currently most used algorithmsin cyclic preﬁx—OFDM system for pilot design, channel estimation, equalization, synchronization,and system performance analysis may no longer be applicable to NB-IoT systems. Their mathematicalmodeling demonstrated that channel equalization coefﬁcients for NB-IoT UE are a set of phase-shiftedCFR combination and not a simple Fourier Transforms of the channel impulse responses. This is theconsequence of sampling rate mismatch between NB-IoT user and base station.In [66], the authors established a comprehensive system model for in-band and guard-bandNB-IoT by considering sample duration. They derived the mathematical expressions of received LTEand NB-IoT signals and analyzed the close-form interference power on LTE signal from adjacentNB-IoT signal. It is observed that the sample duration of NB-IoT signiﬁcantly impacts the desiredsignal as well as interference on LTE UE; this is due to mismatched sampling rate between NB-IoT UEand the base station. Their proposed system model and derivations match the simulations, hence canbe used for coexistence analysis for NB-IoT system.Summary: This subsection has addressed the state of the art of NB-IoT PHY layer protocol.The main focus was set on different approaches to improve cell acquisition process, random accessprocess, channel estimation, and interference mitigation. The next subsection focuses on MAC layerfeatures by addressing the corresponding challenges and potential solutions.54Sensors 2019, 19, 26133.2. Media Access Control LayerHandling retransmissions (HARQ), multiplexing, random access, timing advance, choice oftransport block formats, priority management, and scheduling are the tasks executed by the MAClayer. The discussion on this part focuses on features such as radio resource management, linkadaptation, coverage, and capacity improvement, power, and energy consumption reduction, assummarized in Table 3.Table 3. Articles on proposed MAC layer enhancement techniquesFeature Article Technique Used Enhancement LimitationResourceallocation[68] Resource blanking Interference cancellation byresource blankingThe proposed techniquemay lead to performancedegradation in terms ofspectral efﬁciency, especiallyfor NB-IoT massivedeployment.[69] Iterative algorithm bya cooperative approachRadio resource managementin terms of scheduling index,repetition number andinterferenceThe proposed solution issub-optimal hence it doesnot provide maximumachievable performance interms of maximum rate andcapacity[70] Scheduling algorithmEfﬁcient resource allocationby reducing the NPDCCHperiodsMobility is not consideredand reducing NPDCCHperiod could lower thechannel estimation qualityhence may degrade theperformance by unrealisticchannel estimation[71]Resource allocationtechnique by extendingthe speciﬁc PRB forpaging trafﬁc ofﬂoadpower consumptionreduction for NB-IoT UEduring paging loading andofﬂoadingThe use of speciﬁc PRB forpaging ofﬂoading is not anefﬁcient use of the existingresource blocks. Also,the model is not applicablein standalone mode.[72] NB-IoT schedulingalgorithmInterference analysis for 15kHz LTE coexistence with3.75 kHz guard-bandNB-IoTEmptying the LTE resourceis not efﬁcient resource use.Also, the model is notapplicable for thestandalone mode ofdeploymentLinkadaptation[73] NB-IoT basic scheduleralgorithmOptimal resource usage byconsidering an averagedevice delay and processingtimeThe scheduler did notconsider semi-persistentscheduling, especially forinter-RAT networks[31]Offset index selectionand UE speciﬁc andcommon search spacesfor NB-IoT densenetworksCell capacity enhancementby means of optimalschedulingDid not consider thenumber of sessions that eachdevice has to transmit withrespect to differentrequirements and use cases[74]Link adaptationalgorithm by using themathematicalexpression of ShannontheoremCoverage enhancement bycharacterizing SNR,repetition number andNB-IoT supportedbandwidthThe work did not considerthe impact of channel stateinformation on UE linkadaptation[75]Two-dimensionalNB-IoT dynamic linkadaptation algorithmOptimization of repetitionnumber by dynamicallyadjusting MCS to ensurebetter BLER and BERperformancethe model does notencompass the effect ofspeed and the deploymentof the optional HARQprocess to ensure betterchannel modeling55Sensors 2019, 19, 2613Table 3. Cont.Feature Article Technique Used Enhancement LimitationCoverageand capacity[11]NB-IoT coveragecomparisons in differentscenarios for 15 kHz and3.75 kHz spacingThe channel estimationimpairments, carrier offsetas well as mobility withrespect to differentconﬁgurations are notconsidered for the claimed170 dB of achieved MCL ofNB-IoT[76]Preconﬁgured accessscheme and the jointspatial and code domainschemecapacity and spectralefﬁciency improvementIt can only be applicable insmall cell conﬁgurationswhen NB-IoT is deployed inlarge scale, preconﬁguringaccess for different require[77] Control plane small datatransmission schemeEffective data transmissionenhancement bytransmitting small packetsin RRC connection set upThis scheme may results inNB-IoT signaling overheaddue to Radio ResourceControl (RRC) connectionsetup process encompassedwith small data[10]UE coverage andcapacity simulationmeasurement based onreal operators networkparametersNB-IoT enhanced coveragemeasurements by the use ofreal network conﬁgurationparametersOptimal repetition numberfor NB-IoT devices is notconsidered, with additionalpenetration loss, it does notexplain the additionalrepetition requirement toenhance the coverage whileguaranteeing the requiredperformance[78]Low Earth Orbit (LEO)satellite to extendNB-IoT coverageNB-IoT Coverage extensionbeyond LTE achieved linkbudgetThe work did not considerthe impact of repetitionnumber on extendedcoverage as well as time andfrequency synchronizationthat can lead to samplingrate mismatch as well ascarrier frequency offset forlow-end NB-IoT modulesPowermanagement[79] Practical powermeasurementPower consumptionanalysis for NB-IoT byvarying payloads andrepetition numbers, I-eDRxand PSMUsing two devices is notrepresentative massiveNB-IoT devices in thebecause different chips havedifferent powerconsumption depending onthe enabled features such asinter-RAT support that canaffect the overall deviceconsumption[80] Prediction-basedenergy-saving algorithmReduction of powerconsumption by reducingthe scheduling requestprocedureThe solution is not optimalbecause it reducesscheduling request withoutconsidering the devicerequirement with respect tochannel parameters[81] Semi-Markov chain forenergy evaluationEnergy consumption anddelay requirementevaluation for NB-IoTsystems by considering thefour states, namely powersaving mode, idle mode,RACH procedure,and transmission modeThe model does not includethe energy consumptionduring transition betweenthe four mentioned modesand it does not include theimpact of repetition on thedevice power consumption56Sensors 2019, 19, 26133.2.1. Radio Resource AllocationIn NB-IoT, resource allocation is the key feature to ensure the expected massive connections ina cell. Tone allocations, PRBs, repetition number options, power conﬁgurations, subframes, or timeslots, etc. must be optimized to maximize performance with minimum possible resources. SinceNB-IoT is intended for low rate, less frequent time insensitive applications but with the requiredperformance metrics, better radio resource management will ensure the optimal resource usage forexpected throughput, spectral efﬁciency, and coverage enhancement.In [68], the authors discussed the impact of interference for partial deployment of NB-IoT such thatif one PRB is used for NB-IoT in some of the cells, that same PRB could be used for LTE in other cellsfor the in-band mode of NB-IoT operation. In such a deployment, possible co-channel interference mayappear between NB-IoT UE and LTE UE. The authors modeled the partial deployment in percentilesuch that 100%, 75%, 50%, 25% represent the percentage of cells where NB-IoT enabled. Their resultswere analyzed by means of cumulative density functions (CDF) of respective SINR detected andmaximum coupling loss achieved. The work demonstrates possible NB-IoT interference betweenNB-IoT and another NB-IoT UEs from adjacent cells and between NB-IoT and LTE from the adjacentcell. The simulation is performed for the in-band mode of operation where both NB-IoT and LTE UEsshare the same PRB. They proposed the PRB blanking i.e., blanking the resources that are used byNB-IoT to not be used by LTE, not even being used for CRS. Blanking of these resources will omitthe interference from LTE UEs and will result in NB-IoT only access to this PRB. However,the paperdid not consider the performance degradation due to reduced available radio resources after whenresource blanking is applied.In [69], the authors formulated an analytical model to characterize the maximum achievable datarate, then investigated the impact of intercell interference in a multicell environment (for in-bandand standalone scenarios), and ﬁnally proposed an iterative algorithm which uses cooperativeapproach which takes into consideration the overhead of control channels, repetition number, intercellinterference as well as time offset. The proposed sub-optimal solution ensured better radio resourceallocation, which raised the data rate by 8% and reduced the overall device energy consumption by17% with respect to the non-cooperative approach.In [82], the authors presented preliminary results of RSSI and detected SNR by developinga DORM (integrateD cOmpact naRrowband platforM) node which was deployed on a universitycampus to test its practical feasibility in different indoor scenarios.Their SNR and RSSI values wereobserved to be in the range of 18 dB to 23 dB and −65 dBm to −70 dBm, respectively, which showsits suitability for indoor coverage. The RSSI and SNR values variations are considered to be due todifferent elevations that the nodes are, with respect to the serving base station. However, the paperdoes not explain the channel estimation and measurements quality and their impact on the achievablethroughput, moreover their paper does not cover the outdoor deployment and the impact of repetitionon the overall devices’ energy consumption when devices are located in different indoor environments.In [70], the authors introduced the NB-IoT radio access strategy in detail and studied the NB-IoTscheduling problem. Their primary objective is to lower the number of used radio resource while eachdevice’s data requirement can still be satisﬁed. They furthermore formulated the NB-IoT schedulingproblem and proposed an efﬁcient algorithm to overcome such a problem. Their simulation resultsshow that they could minimize the number of NPDCCH periods (NPs) used to satisfy each device’sdata requirement.However, the repetition number is given according to the distance between the base station andthe device. It could be better to use real-time channel parameters or MCS or BLER value to schedulethe respective downlink channels to the devices. This is because, within the same distances, devicesmay experience different signal attenuation due to different factors such as fading, non-line of sight,line of sight, indoor placement, outdoor placement, underground placement, etc. Therefore, there isstill an open space for practical deployment to analyze the effectiveness of the different downlink and57Sensors 2019, 19, 2613NB-IoT scheduling schemes which considers the device’s simplicity, modulation schemes, channelconditions, and delay requirement for speciﬁc use cases.In [71], the paper proposed a new resource allocation technique by extending the paging resourcethat will be speciﬁc for paging trafﬁc ofﬂoad. The authors noted that the new paging PRB couldlower power consumption which is mostly used to load and ofﬂoad the paging load. Also, the workproposed the selection scheme based on UE identity (ID) that is used to balance the load between thepaging resource blocks. The simulation results show that power consumption reduction and resourceoptimal usage are of 80% and 30.5%, respectively. This work considered adding other PRBs for pagingmonitoring; however, the authors do not demonstrate the trade-off between the newly introducedscheme and the UE complexity requirements.In [83], the authors proposed an enhanced access reservation protocol (ARP) that allowsthe device to transmit a fraction of a preamble sequence by providing an analytical model thatcaptures the performance of ARP in terms of the false alarm, misdetection, and collision probabilities.They mathematically analyze the trade-off between the misdetection and the collision probabilities.The drawback of this protocol is that with massive NB-IoT deployment, altering the conﬁguration ofthe protocol may result in detection performance degradation which can lead to huge packet loss.In [72], the authors analyzed the impact of interference when the 15 kHz LTE system coexistswith a guard-band NB-IoT system with 3.75 kHz subcarrier separation. Their simulation resultsdemonstrated that it is desirable that the scheduler of the LTE system empties the neighboring RBsof the NB-IoT system and allocates resources if possible. The authors then proposed an NB-IoTscheduling method for the LTE system to improve the performance of the studied NB-IoT system.Their results showed that if emptying is not done, at 103 BER there is 1 dB drop of SNR as compared towhen emptying of RBs is done.3.2.2. Link AdaptationLike in LTE, NB-IoT link adaptation involves adaptive modulation and coding schemes as wellas adaptive power allocation. However, the modulation schemes are limited to QPSK to enable lowcomplexity and hence reduce the overall power consumption. To extend the coverage and increase thelink reliability, a repetition number of up to 128 times is introduced. In the literature, it is seen thatNB-IoT link adaptation has several issues; potential solutions are also proposed, as summarized below.In [31], the author formulated the scheduling issue such that the resource assignment must be ina speciﬁc format taking into an account reserved signaling resources and capabilities of the NB-IoT UE.They proposed a solution that incorporates two parameters which are (i) offset index selection (k0) and(ii) UE speciﬁc and common search space conﬁguration. The offset index selection was chosen becausewith the limited k0 and varying size of payloads, it is critical to adapt the scheduling process for highresource use to accommodate more devices at the same time. Additionally, UE speciﬁc and commonsearch space conﬁguration were chosen because it decides the timing of NPDCCH and NPDSCH fordifferent UEs, hence it can consequently improve the overall scheduling efﬁciency.In [73], the authors presented a basic NB-IoT scheduler for NB-IoT system and analyzed theenhancements on average delay, optimal resource usage, and processing time. The proposed algorithmdemonstrated that shorter NPDCCH period selection may reduce the UE average delay and optimizethe overall system resource usage. Also, the model shows that the scheduling delay (k0) should bedetermined before the allocation of subcarriers. However, the model does not elaborate the type ofconﬁguration used since the choice of conﬁguration such as single tone or multi-tones have a directimpact on periodicity and transmission delay and hence can directly impact the system performance.In [74], the authors analyzed NB-IoT repetition number and bandwidth allocation and proposedanalytic expressions based on SNR, bandwidth, and energy per bit that can be derived from Shannontheorem in order to characterize the impact of the repetition number as well as bandwidth allocationto different UEs. Additionally, their work proposed an algorithm for link adaptation. The algorithmexploits resource unit number, repetition as well as bandwidth. Their results show that reducing58Sensors 2019, 19, 2613bandwidth and performing repetitions could enhance the coverage. However, the work did notconsider the actual impact of channel parameters as well as NB-IoT UE impairments such as CFOwhich may lead to transmission errors.In [75], the authors proposed a new NB-IoT link adaptation scheme with the consideration of therepetition factor. They claim that their proposed two-dimensional scheme is composed of Inner LoopLink Adaptation that copes with BLER by periodically adjusting the repetition number and outer looplink adaptation which coordinates the MCS and repetition number. This is because 20 dB coverageenhancement beyond LTE can be achieved by the repetition of transmitted data. So, in this work, theyproposed an algorithm that dynamically chooses MCS and repetition number based on estimatedreal-time channel state information (CSI). However, their algorithm does not elaborate on the differentNB-IoT power classes and to which range their respective coverage could be enhanced.3.2.3. Coverage and CapacityNB-IoT support for extended coverage of up to 164 dB of maximum coupling loss is to enable thetechnology to be used for cellular IoT services, especially for applications that are in hard-to-reach areas.Its narrow bandwidth and support for repetition are the key features to enable the enhanced coverage.In [10], the authors simulated and analyzed the NB-IoT wide-area rural deployment and deepindoor urban deployment by using the network parameters of one metropolitan operator. Their workshowed that NB-IoT devices could still transmit and receive data at an MCL of 167dB, which is 3 dBhigher than the 3GPP’s 164 dB of MCL limit set. Furthermore, in different indoor scenarios, evenwith an addition of 30 dB as penetration loss, NB-IoT had better outage probabilities as compare toanother LTE LPWAN technology (eMTC). For outdoor and light indoor conditions with an additional10 dB penetration loss and an average intersite distance of 2.8 km, NB-IoT had less than 0.1% of outageprobability. However, despite the varying additional penetration losses of 10 dB, 20 dB and 30 dB, theirsimulation does not consider the impact of features such as mobility, CFO, lower power class on theachieved MCL.In [11], the authors showed that for the maximum number of repetitions (128 times), with 15 kHzand 3.75 kHz subcarrier spacing, coverage of up to 170.2 dB MCL and 174.2 dB MCL could be achieved,respectively. The work concluded that the evaluations show that NB-IoT could provide up to 20 dBcoverage enhancement in various deployment scenarios as compared to legacy LTE. Similarly, the workdid not study the impact of mobility and weak channel estimation quality to the achieved MCL.In [76], the authors proposed two less complex scheduling schemes (compared to brute-force)that can be used NB-IoT. The ﬁrst proposed scheme is called the preconﬁgured access scheme and thesecond is the joint spatial and code domain scheme. Their simulation performance results (spectrumefﬁciency, number of active devices as well as low collision rate achieved) by the two low complexschemes were found better when compared to the ones that can be achieved by the brute-force scheme.In [78], the authors proposed a speciﬁc unidirectional system to study the coverage enhancementby using satellite network i.e., LEO constellation. Their proposed model with the mathematicalderivations shows that NB-IoT could achieve the 20 dB more than LTE achieved MCL and could stilloperate according to Release 13 standards. From their results, it is seen that the packet error rate (PER)of the transmitted signal is distorted by Doppler spread. However, the model does not consider theclock synchronization between NB-IoT device and satellite, which can lead to performance degradationespecially caused by the CFO or the sampling rate mismatch. Additionally, the work did not considerthe maximum achievable throughput when their system is employed to comment on the effectivenessof the techniques as compared to terrestrial NB-IoT deployment.3.2.4. Power and Energy ManagementThe NB-IoT reduced complexity is intended to reduce the power consumption in different modes.PSM and eDRX are the implemented features dedicated to foster the long-lasting battery life.59Sensors 2019, 19, 2613In [79], the authors presented the NB-IoT power measurement. Their measurements were set insuch a way that NB-IoT transmission consumes 716 mW when at 23 dBm with a power efﬁciency of37%. DL control and data signals consume 213 mW, idle-mode-eDRx and PSM consumes 21 mW and13 μW, respectively. In general, according to their empirical measurements, it is shown that the powerconsumption is 10% lower than the 3GPP estimates. During measurements, parameters such as timedomain repetition, I-eDRx, and PSM were taken into consideration. To characterize each componentin the proposed model, several test cases such as Tx power, UL, and DL data rates, I-eDRx, and PSMwere used and all parameters except one at a time were ﬁxed. Their results showed that the NB-IoTdevices power consumption is independent of the subcarrier spacing. However, the total ON time ofthe devices is in many cases deﬁning the overall battery life. As their remark, the data rates do notdirectly impact the power consumption, but it has a major direct impact because it deﬁnes the overalldevice ON time. If the transmitting interval is 1 h, the device achieves only 2.5 weeks of battery life.Increasing the duration to 24 h, the lifetime of the device increases to 12.8 years in PSM.In [80], the authors proposed a prediction-based energy-saving mechanism to reduce energyconsumption by decreasing the number of scheduling request procedures. Their proposed schemeshowed that it could reduce the NB-IoT active time from 5% to 16% for the medium and bad channelquality and achieve from 10% to 34% battery saving in different scenarios as compared to 3GPPconsumption simulation speciﬁcations in [43].In [81], the authors developed a semi-Markov chain with power saving mode, idle mode, randomaccess, and transmission mode to study the energy requirement and delay performance for NB-IoT.It is noted that for massive synchronous connections, extra power is drained in random access andtransmission states due to collisions. The paper further proposes an energy optimization model basedon a priori method that takes into consideration the PSM duration as well as power consumption.The results demonstrate that for optimal energy and delay requirement, it is important to set the higherRACH transmission number to accommodate more delay on the UE. However, their optimizationmodel did not consider the power consumption during the transition of different states, because whenthe UE is required to perform several sessions per day, it might go through several transitions that havea signiﬁcant effect on power consumption. Furthermore, the mode does not include the small datatransmission scheme during RRC connection as proposed in the updated standards. However, withthe introduction of the new power class in NB-IoT Release 14, there is a need for practical experimentsto evaluate the new coverage classes. With lower transmit power, the SNR detected at the base stationbecomes lower hence the device will need to perform more repetitions to enhance coverage.3.3. Upper LayersAlthough the focus of this paper is mainly on the features regarding PHY an MAC layers, it isstill imperative to address some enhancements, challenges, and potential solutions to the upper layers.Especially the changes that are implemented in Evolved Packet Core (EPC) by adding the ServiceCapability Exposure Function (SCEF) to manage both IP and non-IP data packets [84].Control and User Plane OptimizationTo support massive end-to-end device connectivity with extremely low complexity and reduce thetransmission signaling, NB-IoT implements new small data transmission procedures based on CellularIoT (CIoT) Evolved Packet System on both Control Plane (CP) and User Plane (UP). These transmissionprocedures support small bursts of data efﬁciently while guaranteeing the long-range coverage ascompared to legacy GPRS [85,86]. In this regard, NB-IoT is can support more than one data path in CPfor the transmission of user data which is carried by the signaling messages managed by the MobileMobility Entity (MME) as shown in Figure 5. The procedures are optimized to efﬁciently support thesmall data transfer as follows:• Mandatory CP CIoT EPS;• Optional UP CIoT EPS.60Sensors 2019, 19, 2613CP CIoT EPS optimization encapsulates the data packets in Non-Access Stratum (NAS) by usingcontrol plane signaling messages. In this regard, this procedure is mandatory. Compared to theconventional SR procedure, the NB-IoT UE skips some steps required for each data transfer hence thisoptimization procedure best ﬁts the short data transmission or reception.On the other hand, UP CIoT EPS optimization requires the RRC connected mode to get the scheduledradio resources as well as Access Stratum (AS) between the UE and the network. This mode uses thenewly introduced connection to Suspend and Resume procedures. Connection suspend procedure helpsto retain the network context so that the UE can resume the connection when traffic is available. Retainingthe context helps the UE and the network to skip the AS and RRC reconfiguration in each data transfer.Since it uses user plane, the UP CIoT EPS is suitable for both small and large transactions.Mobile Mobility Entity (MME)Serving Gateway (SGW)Packet Data Network GateWay (PDN GW)Service Capability Exposure Function (SCEF)NB-IoT UEEvolved Packet Core (EPC)Application servereNodeBFigure 5. Representation of NB-IoT IP and Non-IP data path: Blue line displays the IP data path in UPmode (as Legacy LTE), Red line displays the non-IP data path in CP mode, and dashed-line displaysthe IP data path in CP mode.Furthermore, the UE in Service Request procedure (an LTE procedure used by the UE and basestation to transmit or receive data in RRC idle state) is required to be in a connected state in orderfor base station to allocate the radio resources. For NB-IoT this SR is optional; however, NB-IoT UEthat supports UP optimization needs also to support SR. For example; if the NB-IoT UE wants totransmit the uplink data in idle state, it will send the random access preamble through which the basestation and UE will establish RRC connection and UE will be allocated with the radio resources fordata transfer. After a certain period of inactivity, the base station initiates the release procedure.Similarly, for UE downlink data reception, if the UE is in DRX mode, the UE regularly listensto downlink signaling and if the UE notices the paging message, it will perform the SR procedure asdescribed in uplink data transmission. Additionally, if the UE is in PSM mode, it will be completelyinaccessible until it initiates the same SR procedure for the uplink grant or by using Tracking AreaUpdate (TAU).There are works that are addressing the upper layers such as [77], where the authors proposed anefﬁcient small data transmission scheme by using CP procedure. The proposed scheme enables thedevices to transmit data packets through the RRC connection setup procedure when the device is inidle mode. This process reduces the signaling overhead caused by the security setup process and dataradio bearer setup process. However, a suggestion could be to analyze the power consumption duringthis small data transmission and compare its effectiveness to when the same data is transmitted duringthe UP procedure.Summary: This section discussed PHY layer features, highlighting the corresponding enhancementson cell acquisition procedure, random access channel estimation, and interference mitigation. It thenaddressed the MAC layer enhancements regarding resource allocation, link adaptation, coverage and61Sensors 2019, 19, 2613capacity, and power management. It further addressed the upper layers changes related to cellular IoTevolved packet system optimization through user and control planes to enhance the small data packetstransmissions for end-to-end massive connectivity.4. Narrowband-IoT Possible Deployment StrategiesThis section proposes potential deployment strategies for NB-IoT massive deployments byconsidering the NB-IoT support for small cells in heterogeneous network scenarios.HetNets are effective network deployment strategies in which small cells are incorporated inmacrocells with the objective of improving performance in terms of capacity, coverage, and spectralefﬁciency. In general, the macrocells are characterized by higher transmit power and broader range ascompared to small cells. When smalls cells are overlaid in macrocells, interference becomes a concern,especially to small cell edge users. Several techniques for interference cancellation, estimation andcoordination that involve frequency hopping, frequency reuse, power control etc. have been proposed;however, the performance trade-offs for the proposed techniques for macrocells and small cells arestill challenging [87–90].Similarly, NB-IoT is expected to coexist with the currently deployed legacy LTE as well as theforthcoming 5G networks. This questions the existing interference management techniques i.e., are theyapplicable to the newly deployed technology since NB-IoT is expected to support different powerclasses while maintaining the low complexity which can severely affect the channel estimation qualityand hence interference estimation quality. In NB-IoT coexistence with the legacy cellular networks,the possible deployment scenarios are as follows:• Synchronous NB-IoT deployment in all small cells;• Asynchronous NB-IoT deployment in all small cells;• Synchronous NB-IoT deployment in small cells and Macrocells;• Asynchronous NB-IoT deployment in small cells and LTE in macrocells.These scenarios, as shown in Figure 6, are detailed in what follows.??????????? ??????????????????? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????Figure 6. Summary of NB-IoT deployment strategies. For example, when NB-IoT is deployed inmacrocell and LTE in small cell, when LTE is in macrocell and NB-IoT is in small cells, when NB-IoTis in macrocell and small cells support both NB-IoT and LTE, and when LTE is in macrocell andLTE/NB-IoT is in small cells62Sensors 2019, 19, 26134.1. Synchronous NB-IoT Deployment in All Small CellsThis is the NB-IoT deployment strategy which is enabled in all the small cells by using the samephysical resource blocks. All the small cells are synchronized in such a way that with the samePRBs, all the NB-IoT UEs are using the transmit power that is conﬁgured regardless of its maximumtransmitting power capacity. This means that even though NB-IoT devices might support differentpower classes such as 14 dBm, 20 dBm, or 23 dBm, the NB-IoT devices will only be conﬁgured to usethe minimum allowed transmit power in order to avoid causing the co-channel interference to otherUEs using the same radio resources.In this strategy, power control may be the key feature to ensurethe required performance. However, cell edge UEs may still suffer from the interference problem.This interference may highly be increased due to the low channel estimation quality of NB-IoT UEsassociated by its reduced computational complexity.4.2. Asynchronous NB-IoT Deployment in All Small CellsThis deployment strategy is employed in such a way that NB-IoT is enabled in all small cells byusing different physical resource blocks. This implementation may avoid the interference betweenNB-IoT UEs from different small cells; however, this may result in co-channel interference betweenNB-IoT and LTE UEs that are using the same radio resources. When deploying under this strategy,it is imperative to implement proper frequency planning as well as proper power conﬁguration forNB-IoT devices. As seen from the state of the art, some works have proposed blanking of the radioresources to the adjacent cells for the resources that are already occupied by NB-IoT even in the cellsthat NB-IoT is not enabled. However, blanking of the resources is a wastage of resources, so, thereshould be some other means such as frequency hopping to avoid wastage of resources (blanking) aswell as to mitigate interference.4.3. Synchronous NB-IoT Deployment in Small Cells and Macro CellsIn this strategy, NB-IoT is enabled in the small cells as well as in macro cell on the same PRBs.Macrocell UEs are conﬁgured to use higher transmit power as compared to small cell UEs whilekeeping the same PRBs for NB-IoT while others left for legacy LTE. Possible co-channel interferencemay occur in small cell edge UEs if the UEs are scheduled on the same resource units. The impact mayfurther increase for UEs under mobility which might require the use of handover for smoothing the UEstransition from one serving cell to another. From our review, no work has addressed the interferencecancellation mechanism for such a case. It is imperative to employ the existing geographical planning,frequency reuse, frequency hopping, and power control while considering the low complexity buthigh coverage range NB-IoT.4.4. Asynchronous NB-IoT Deployment in Small Cells and LTE in MacrocellsIn this strategy, NB-IoT uses separate PRBs between small cells and macro cells. This means thatone or more PRBs are used for small cells and different PRB(s) for the macrocells. If the PRBs arenot well planned, NB-IoT users from adjacent cells (using the same resource units) may suffer frominterference. Also, LTE users that are using the same resource elements may interfere with small cell ormacro cell UEs. Different transmit power control conﬁgurations may be used to control interference.The choice of the deployment strategy depends on several factors such as use-case requirements,environmental conditions, equipment quality, etc. It is imperative to implement better interferenceestimation, mitigation or management techniques that will ensure better performance and spectralefﬁciency for the massive NB-IoT deployment in coexistence with other technologies.Summary: This section has presented the possible NB-IoT deployment strategies by consideringthe NB-IoT support for small cells in coexistence with legacy LTE in HetNet scenario.The following section presents the open research challenges to motivate future research directions.63Sensors 2019, 19, 26135. Open Research Questions and Discussion5.1. Battery LifePSM and eDRx were introduced in NB-IoT Release 12 and 13 to lengthen the NB-IoT devices’battery life. Moreover, the most recent updates require the UE to be able to transmit during RRC-idlemode which will reduce the required ON time for data transmission. However, devices experiencingbad channel conditions due to hard-to-reach areas will require to perform several retransmissions persession, which will drain the device’s energy and hence shortens the battery life. Similarly, devices thatrequire a relatively large number of reporting sessions per day will consume more energy, which makesenergy management a concern. As seen in Section 3, most of the proposed algorithms are powerhungry because most of the power is consumed during transmission and reception. Therefore, energyharvesting alternatives such as solar, biogas, vibrations, etc. that will lengthen the NB-IoT devicebattery life should be introduced to complement or replace frequent battery charging.5.2. Radio Resource Management5.2.1. Tones Allocation.As seen in the literature, most of the articles consider single-tone allocation for the simplicityin the simulation, thus, multi-tone allocation is not well studied. This causes a knowledge gap inthe effectiveness of different tone-allocation possibilities. Moreover, for guard-band, in-band andstandalone it is still not clear about the respective performance metrics that could be achieved interms of throughput, coverage range, interference robustness etc. This restricts to a certain extentthe optimal choice of deployment for a large number of devices with the required performance.Furthermore, different frame structures, especially for TDD conﬁgurations, are not discussed eventhough NB-IoT is required to support TDD. Therefore, optimal resource use techniques must beproposed that incorporate repetition, mobility, tones allocation, etc. for efﬁcient spectrum usage.5.2.2. Interference MitigationInterference prediction, estimation, cancellation, and coordination techniques for NB-IoT becomea challenge. This is because of the sharing of spectrum resources between NB-IoT and legacy LTE.Similarly, with NB-IoT being deployed in a small cell or macrocell scenarios in heterogeneous networks,interference becomes a concern. Several works have tried to address this by means of resourceblanking, power control, or better uplink and downlink scheduling schemes and frequency andtiming synchronization, etc. However, it is still challenging to incorporate the NB-IoT features such asrepetition, low complexity (which affects channel estimation quality), and mobility in deploying thealready existing LTE interference management techniques. As seen in the possible NB-IoT deploymentscenarios above, there is still a need for deploying effective schemes that will ensure better NB-IoTperformance without degrading the LTE performance [91,92].5.3. Mobility ManagementAs seen in Section 2, most of the simulation works have ignored the mobility impact of NB-IoTchannel modeling. However, for use cases that involve movement, Doppler shift has to be takeninto consideration during channel estimation, which might slightly increase the device complexity tosupport handover and other mobility features such as the support for inter-RAT mobility during idlemode [93,94]. The increase in NB-IoT UEs mobility makes the channel suffer from fast varying channelconditions, due to which adaptive transmission schemes that might involve channel estimation, errorcorrection, etc. must be implemented.Therefore, applying intelligent/adaptive algorithms that are low power and optimal for repetitionnumber, yet mobility-aware, is of great importance. The algorithms could involve low-power frequentCSI reporting, early data transmission by using both user and control plane in either Msg 3 or Msg 4.64Sensors 2019, 19, 26135.4. LatencyNB-IoT latency tolerance is set to 10 ms. This is due to its support for use cases of UEs that arein environments with bad channel conditions [95–97]. Initial cell acquisition, frequency, and timingrequirements, RACH transmission, half duplex mode of transmission and several repetitions that areperformed during transmission are some of the features that play part in the overall data transmissiondelay. Several works are trying to reduce the timing requirement so as to reduce transmission latencyof devices; however, most of the works have not addressed delay by taking into consideration themassive congestion that is expected for the IoT networks, processing delays due to low complexdevices, queuing delays, propagation delays especially with long-range feature, as well as errors anderror recovery.However, early data transmission schemes and the second NB-IoT HARQ process for devicesthat have good channel conditions are among the features that can be used to reduce the transmissionlatency and improving the transmission link performance. However, only a handful of research articleshave discussed the effectiveness of these processes when applied in NB-IoT.5.5. Semi-Persistent vs. Dynamic SchedulingMost of the NB-IoT literature addresses dynamic uplink and downlink scheduling by studyingthe scheduling of logical channels and signals. There are still very few NB-IoT studies about theeffectiveness of Semi-Persistent Scheduling schemes (SPS) even though SPS helps to reduce theNPDCCH overhead as compared to dynamic scheduling. It provides the NB-IoT UEs with longerallocated resources (more than one subframe) so that the NB-IoT device will not need the frequentdownlink assignment as well as an uplink grant which is delivered by NPDDCH for each subframe.However, for applications that involve mobility or fast varying channel conditions, how is thisscheduling scheme going to be effective knowing that NB-IoT has poor channel estimation capabilitiesas compared to LTE?5.6. Random AccessMassive NB-IoT modules that try to request the radio channel resources at the same time foruplink data transmission may suffer from random access preamble collision. This is caused by severalfactors such as detection inaccuracy that may not satisfy the detection threshold, the high probabilityof false alarm, etc. Several works have proposed random access preamble detection algorithms(i.e., random access with differential barring etc.) and others have developed mathematical modelsto characterize the preamble transmissions in order to improve the NPRACH success rate andbetter time-of-arrival estimation and other NPRACH performance improvements. However, it isstill unclear which scheme is effective for massive deployment, since most of the proposed schemesdo not consider the heterogeneous network architecture, channel estimation impairments, or realisticchannel conditions [98,99].5.7. Timing Advance (TA)When the base station responds to NB-IoT UEs about RRC connection request, it incorporatesthe TA command to be used for NB-IoT UE terminal data uplink transmission timing (i.e., totime-synchronize the UEs to the base station and help to compensate the propagation delays). However,for NB-IoT UE, the TA adjustment accuracy of the signaled timing advance with respect to the prioruplink transmission may highly be affected by the massive number of NB-IoT devices contendingfor the access. This is because the base station may need to correct some UE timing while forother NB-IoT UEs that had already transmitted NPRACH could receive the random access responsewhich is not intended for them. Some works have addressed the receiver algorithms for NPRACHTA estimation as well as detection timing advance adjustment decoding schemes to improve the65Sensors 2019, 19, 2613estimation but the NB-IoT receiver sensitivity and weak channel estimation quality still negativelyaffect the TA adjustment.5.8. Cell Search and Initial SynchronizationNPSS and NSSS are two signals based on frequency domain Zadoff-Chu sequence that areused for NB-IoT time and frequency synchronization to the base station. According to NB-IoTstandard, NPSS and NSSS may not be transmitted on the same antenna port hence NB-IoT initialsynchronization may rely on NPSS only. The challenge is that the imperfect channel conditions mayseverely affect the cell camping procedure as a small CFO may result in a phase shift to a receivedfrequency domain sequence which as a consequence may degrade the cell search and synchronizationperformance. To improve this, frequency diversity techniques should also be used for NPSS and NSSSreception improvement.5.9. Uniﬁed NB-IoT Testing ToolSince NB-IoT is a promising technology, there should be a uniﬁed testing tool used as a reference toverify if the produced products comply with the standards. Taking Bluetooth as an example, for bettercompatibility towards different available products from handsets to car kits, Proﬁle Tuning Suite (PTS)software is used to automate the compliance testing to speciﬁc Bluetooth function. So, to supportcompliance with standards and hence backward compatibility and interoperability, what is the testingtool to validate if different available products will ﬁt standards? Similarly, for simulation purposes,most of the works choose the parameters that can generate results easily. If there is a concrete simulationmodel that takes into account the major NB-IoT features and incorporating all the possibilities fromrepetition number allocation, mobility selection, modulation and coding scheme, real-time channelvariations, etc. it would be easier to get realistic modeling for different scenarios.5.10. Backward Compatibility and InteroperabilityA ten-year telecommunication generation is characterized by different changes in releases andupdates. In order to reach their lifespan as compared to what the standards stipulate, NB-IoT devicesshould operate for around ten years with a single battery charge. Whenever new releases or updatesare introduced, backward compatibility and interoperability should be possible. Apparently, the devicecomplexity is set as low as possible; will these simple devices (hardware) support hard and robustalgorithms that will be implemented by over-the-air upgrades/updates to satisfy the demands offuture NB-IoT use cases?Summary: This section has presented the open research questions regarding battery life, radioresource allocation, cell search, and initial acquisition procedures, mobility management, latency,random access, etc., as summarized in Table 4, in order to motivate future research directions. The nextsection concludes the paper.Table 4. Open Research Questions related to the physical layer, MAC layer, and standard.Physical Layer MAC Layer StandardRadio resource management Timing advance adjustment Support for small cellFrequency and timesynchronizationDynamic scheduling andsemi-persistent scheduling TDD supportRandom access Latency Antenna diversityChannel estimation Power management Mobility and handover supportError correction Network throughput More efﬁcient group messagesLink adaptation Control packet overhead Multicarrier operationInterference mitigation Control plane small datatransmissionNetwork management tool for UEdifferentiation66Sensors 2019, 19, 26136. ConclusionsDue to the fact that most of the existing works are segmented and only consider one or tworeleases in their corresponding studies or simulations, this paper has presented a comprehensiveoverview of NB-IoT standard from Release 13 to Release 16 prospects to enhance and enable morerealistic research. It further presented the detailed current state of the art of NB-IoT based on theongoing discussion on NB-IoT protocol stack along with the related contributions and analyzed theknowledge gaps by using NB-IoT standard as a benchmark. It is observed that most of the articles focuson improving one or few features while neglecting others,it could be better to display the trade-offsbetween the improvement feature and the neglected ones, i.e., performance trade-off between PHYand MAC layer when one feature is changed in either of the layers, the impact of repetition on overallenergy consumption, CFO on channel estimation quality etc. This paper also presented the NB-IoTdeployment strategies to highlight the coexistence possibilities with other legacy technologies i.e., LTE,by considering the NB-IoT support for small cells in HetNet scenarios. Lastly, it discussed the openresearch challenges and the future common research focus on NB-IoT i.e., battery life, optimal resourceusage, handover support during mobility, transmission latency, scheduling, etc. To the best of theauthor’s knowledge, this is the ﬁrst survey that covers broadly these mentioned contributions andhence this work will help the researchers get most of the needed information to accelerate their researchby ﬁnding the relevant information and sources for deeper exploration of the research concepts as wellas ﬁnding possible solutions.Author Contributions: Conceptualization, C.B.M., H.M., M.M.A., Y.L.M., S.P. and S.M.; investigation, C.B.M.,H.M., M.M.A., Y.L.M., S.P. and S.M.; resources, C.B.M., H.M., M.M.A., Y.L.M., S.P. and S.M.; writing—originaldraft preparation, C.B.M..; writing—review and editing, C.B.M., H.M.; formal analysis, C.B.M., H.M., M.M.A.,Y.L.M., S.P. and S.M.; supervision, M.M.A., H.M., Y.L.M.Funding: This research received no external funding.Acknowledgments: This project has received funding partly from European Union’s Horizon 2020 Research andInnovation Program under Grant 668995 and European Union Regional Development Fund in the framework ofthe Tallinn University of Technology Development Program 2016–2022. This material reﬂects only the authors’view and the EC Research Executive Agency is not responsible for any use that may be made of the informationit contains.Conﬂicts of Interest: The authors declare no conﬂict of interest.References1. Lucero, S. IoT Platforms: Enabling the Internet of Things. 2019. Available online: https://cdn.ihs.com/www/pdf/enabling-IOT.pdf (accessed on 12 January 2019).2. Salman, L.; Salman, S.; Jahangirian, S.; Abraham, M.; German, F.; Blair, C.; Krenz, P. Energy efﬁcientIoT-based smart home. In Proceedings of the IEEE 3rd World Forum on Internet of Things (WF-IoT),Reston, VA, USA, 12–14 December 2016.3. Bardyn, J.; Melly, T.; Seller, O.; Sornin, N. IoT: The era of LPWAN is starting now. In Proceedings ofthe ESSCIRC Conference 2016: 42nd European Solid-State Circuits Conference, Lausanne, Switzerland,12–15 September 2016; pp. 25–30. [CrossRef]4. Devalal, S.; Karthikeyan, A. LoRa Technology—An Overview. In Proceedings of the Second InternationalConference on Electronics, Communication and Aerospace Technology (ICECA), Tamilnadu, India,29–30 March 2018.5. Mekki, K.; Bajic, E.; Chaxel, F.; Meyer, F. A comparative study of LPWAN technologies for large-scale IoTdeployment. ICT Express 2019, 5, 1–7. [CrossRef]6. Lauridsen, M.; Nguyen, H.; Vejlgaard, B.; Kovacs, I.Z.; Mogensen, P.; Sorensen, M. Coverage Comparisonof GPRS, NB-IoT, LoRa, and SigFox in a 7800 square km Area. In Proceedings of the IEEE 85th VehicularTechnology Conference (VTC Spring), Sydney, Australia, 4–7 June 2017.7. Sinha, R.S.; Wei, Y.; Hwang, S.H. A survey on LPWA technology: LoRa and NB-IoT. ICT Express 2017,3, 14–21. [CrossRef]67Sensors 2019, 19, 26138. Rico-Alvarino, A.; Vajapeyam, M.; Xu, H.; Wang, X.; Blankenship, Y.; Bergman, J.; Tirronen, T.; Yavuz, E.An overview of 3GPP enhancements on machine to machine communications. IEEE Commun. Mag. 2016,54, 14–21. [CrossRef]9. Gozalvez, J. New 3GPP Standard for IoT [Mobile Radio]. IEEE Veh. Technol. Mag. 2016, 11, 14–20. [CrossRef]10. Lauridsen, M.; Kovacs, I.Z.; Mogensen, P.; Sorensen, M.; Holst, S. Coverage and Capacity Analysis ofLTE-M and NB-IoT in a Rural Area. In Proceedings of the 2016 IEEE 84th Vehicular Technology Conference(VTC-Fall), Montreal, QC, Canada, 18–21 September 2016.11. Adhikary, A.; Lin, X.; Wang, Y..E. Performance Evaluation of NB-IoT Coverage. In Proceedings of the IEEE84th Vehicular Technology Conference (VTC-Fall), Montreal, QC, Canada, 18–21 September 2016.12. Telia. Verdens Storste Iot Pilot. 2018. Available online: https://www.telia.no/magasinet/verdens-storste-iot-pilot/ (accessed on 23 December 2018).13. U Blox. Press Releases. 2019. Available online: https://www.u-blox.com/en/press-releases/ (accessed on26 January 2019).14. Norway, T. Sub-Pump. 2018. Available online: https://www.teliacompany.com/en/news/news-articles/2017/sub-pump/ (accessed on 26 December 2018).15. T Mobile. Smart Cities. 2019. Available online: https://iot.t-mobile.com/solutions/smart-cities/ (accessedon 10 February 2019).16. Skyworks. Cellular IoT. 2019. Available online: http://www.skyworksinc.com/Products/1152/Cellular_IoT (accessed on 15 February 2019).17. Mediatek. NB-IoT. 2019. Available online: https://www.mediatek.com/products/nbIot/mt2625 (accessedon 19 February 2019).18. Huawei. 2019. Available online: https://e.huawei.com/my/solutions/technical/iot/nb-iot (accessed on20 February 2019).19. Quactel. LPWA IoT Module. 2019. Available online: https://www.quectel.com/product/list/LPWAIoTModule.htm (accessed on 15 March 2019).20. Nordic Semiconductors. 2019. Available online: https://www.nordicsemi.com/News/2018/12/ (accessedon 10 February 2019).21. Intel. Modem Solutions. 2019. Available online: https://www.intel.com/content/www/us/en/mobile/modem-solutions.html (accessed on 10 February 2019).22. Sequans. 2019. Available online: http://www.sequans.com/products-solutions/ (accessed on 10February 2019).23. Qualcomm. IoT Modem. 2019. Available online: https://www.qualcomm.com/products/mdm9206-iot-modem (accessed on 10 February 2019).24. Sierrawireless. Products and Solutions. 2019. Available online: https://www.sierrawireless.com/products-and-solutions/embedded-solutions/ (accessed on 10 February 2019).25. Samsung. Samsung Exynos. 2019. Available online: https://news.samsung.com/global/samsungs-exynos-i-s111 (accessed on 10 February 2019).26. Altair Semiconductors. 2018. Available online: https://altair-semi.com/ (accessed on 10 February 2019).27. U blox. Cellular Modules. 2019. Available online: https://www.u-blox.com/en/cellular-modules (accessedon 10 February 2019).28. Chen, M.; Miao, Y.; Hao, Y.; Hwang, K. Narrow Band Internet of Things. IEEE Access 2017, 5, 20557–20577.[CrossRef]29. Hoglund, A.; Lin, X.; Liberg, O.; Behravan, A.; Yavuz, E.A.; Van Der Zee, M.; Sui, Y.; Tirronen, T.;Ratilainen, A.; Eriksson, D. Overview of 3GPP Release 14 Enhanced NB-IoT. IEEE Netw. 2017, 31, 16–22.[CrossRef]30. Ratasuk, R.; Mangalvedhe, N.; Xiong, Z.; Robert, M.; Bhatoolaul, D. Enhancements of narrowband IoT in3GPP Rel-14 and Rel-15. In Proceedings of the IEEE Conference on Standards for Communications andNetworking (CSCN), Helsinki, Finland, 18–20 September 2017.31. Boisguene, R.; Tseng, S.; Huang, C.; Lin, P. A survey on NB-IoT downlink scheduling: Issues and potentialsolutions. In Proceedings of the 13th International Wireless Communications and Mobile ComputingConference (IWCMC), Valencia, Spain, 26–30 June 2017.32. Xu, J.; Yao, J.; Wang, L.; Ming, Z.; Wu, K.; Chen, L. Narrowband Internet of Things: Evolutions, Technologies,and Open Issues. IEEE Internet Things J. 2018, 5, 1449–1462. [CrossRef]68Sensors 2019, 19, 261333. Feltrin, L.; Tsoukaneri, G.; Condoluci, M.; Buratti, C.; Mahmoodi, T.; Dohler, M.; Verdone, R. Narrowband IoT:A Survey on Downlink and Uplink Perspectives. IEEE Wirel. Commun. 2019, 26, 78–86. [CrossRef]34. Zayas, A.D.; Merino, P. The 3GPP NB-IoT system architecture for the Internet of Things. In Proceedingsof the IEEE International Conference on Communications Workshops (ICC Workshops), Paris, France,21–25 May 2017.35. ERICSSON. Scope of Release 16-NB-IoT, RP-181187. 2018. Available online: http://www.3gpp.org/ftp/TSG_RAN/TSG_RAN/TSGR_80/Report/ (accessed on 23 December 2018).36. Huawei. On NB-IoT Evolution in Rel-16, RP-180877. 2018. Available online: http://www.3gpp.org/ftp/TSG_RAN/TSG_RAN/TSGR_80/Report/ (accessed on 23 December 2018).37. 3GPP. Revision of WI on Enhancements of NB-IoT, RP-161901. 2018. Available online: https://portal.3gpp.org/ngppapp/CreateTdoc.aspx?mode=view&contributionId=730352 (accessed on 23 December 2018).38. 3GPP. TS 36.300 V13.8.0 (2017-06). 2018. Available online: https://portal.3gpp.org/desktopmodules/Speciﬁcations/SpeciﬁcationDetails.aspx?speciﬁcationId=2430 (accessed on 23 December 2018).39. Xu, T.; Darwazeh, I. Non-Orthogonal Narrowband Internet of Things: A Design for Saving Bandwidth andDoubling the Number of Connected Devices. IEEE Internet Things J. 2018, 5, 2120–2129. [CrossRef]40. Beyene, Y.D.; Jantti, R.; Tirkkonen, O.; Ruttik, K.; Iraji, S.; Larmo, A.; Tirronen, T.; Torsner, J. NB-IoTTechnology Overview and Experience from Cloud-RAN Implementation. IEEE Wirel. Commun. 2017,24, 26–32. [CrossRef]41. Ratasuk, R.; Mangalvedhe, N.; Kaikkonen, J.; Robert, M. Data Channel Design and Performance forLTE Narrowband IoT. In Proceedings of the IEEE 84th Vehicular Technology Conference (VTC-Fall),Montreal, QC, Canada, 18–21 September 2016 .42. Ratasuk, R.; Mangalvedhe, N.; Zhang, Y.; Robert, M.; Koskinen, J. Overview of narrowband IoT in LTERel-13. In Proceedings of the IEEE Conference on Standards for Communications and Networking (CSCN),Berlin, Germany, 31 October–2 November 2016.43. 3GPP. Specs. 2019. Available online: http://www.3gpp.org/ftp/Specs/archive/45_series/45.820/45820-d10.zip (accessed on 24 April 2019).44. 3GPP. TS 36.213. 2018. Available online: https://portal.3gpp.org/desktopmodules/Speciﬁcations/SpeciﬁcationDetails.aspx?speciﬁcationId=2427 (accessed on 29 December 2018).45. Ericsson. Ground-Breaking Long-Range NB-IoT Connection. 2018. Available online:https://www.ericsson.com/en/press-releases/2018/9/ericsson-and-telstra-complete-ground-breaking-long-range-nb-iot-connection, (accessed on 25 December 2018).46. Ericsson. Key Technology Choices for Optimal Massive IoT Devices. 2019. Availableonline: https://www.ericsson.com/en/ericsson-technology-review/archive/2019/key-technology-choices-for-optimal-massive-iot-devices (accessed on 20 March 2019).47. Ericsson. Technology Overview. 2019. Available online: https://www.ericsson.com/en/ericsson-technology-review/archive/2019/ (accessed on 10 April 2019).48. Wang, Y.E.; Lin, X.; Adhikary, A.; Grovlen, A.; Sui, Y.; Blankenship, Y.; Bergman, J.; Razaghi, H.S. A Primeron 3GPP Narrowband Internet of Things. arXiv 2016, arXiv:1606.04171.49. Kroll, H.; Korb, M.; Weber, B.; Willi, S.; Huang, Q. Maximum-Likelihood Detection for Energy-EfﬁcientTiming Acquisition in NB-IoT. In Proceedings of the IEEE Wireless Communications and NetworkingConference Workshops (WCNCW), San Francisco, CA, USA, 19–22 March 2017.50. Ali, A.; Hamouda, W. On the Cell Search and Initial Synchronization for NB-IoT LTE Systems.IEEE Commun. Lett. 2017, 21, 1843–1846. [CrossRef]51. Xu, T.; Darwazeh, I. Uplink Narrowband IoT Data Rate Improvement: Dense Modulation Formats orNon-Orthogonal Signal Waveforms? In Proceedings of the 2018 IEEE 29th Annual International Symposiumon Personal, Indoor and Mobile Radio Communications (PIMRC), Bologna, Italy, 9–12 September 2018;pp. 142–146. [CrossRef]52. Zou, J.; Xu, C. Frequency Offset Tolerant Synchronization Signal Design in NB-IoT. Sensors 2018, 18, 4077.[CrossRef]53. Lin, X.; Adhikary, A.; Eric Wang, Y. Random Access Preamble Design and Detection for 3GPP NarrowbandIoT Systems. IEEE Wirel. Commun. Lett. 2016, 5, 640–643. [CrossRef]69Sensors 2019, 19, 261354. Cho, S.; Kim, H.; Jo, G. Determination of Optimum Threshold Values for NPRACH Preamble Detectionin NB-IoT Systems. In Proceedings of the 2018 Tenth International Conference on Ubiquitous and FutureNetworks (ICUFN), Prague, Czech Republic, 3–6 July 2018; pp. 616–618.55. Feltrin, L.; Condoluci, M.; Mahmoodi, T.; Dohler, M.; Verdone, R. NB-IoT: Performance Estimation andOptimal Conﬁguration. In Proceedings of the European Wireless 24th European Wireless Conference,Catania, Italy, 2–4 May 2018.56. Shimura, A.; Sawahashi, M.; Nagata, S.; Kishiyama, Y. Physical Cell ID Detection Performance ApplyingFrequency Diversity Reception to NPSS and NSSS for NB-IoT. In Proceedings of the 24th Asia-PaciﬁcConference on Communications (APCC), Ningbo, China, 12–14 November 2018.57. Martín, A.G.; Leal, R.P.; Armada, A.G.; Durán, A.F. NBIoT Random Access Procedure: System Simulationand Performance. In Proceedings of the Global Information Infrastructure and Networking Symposium(GIIS), Thessaloniki, Greece, 23–25 October 2018.58. Baracat, G.H.; Brito, J.M.C. NB-IoT Random Access Procedure Analysis. In Proceedings of the 2018 IEEE 10thLatin-American Conference on Communications (LATINCOM), Guadalajara, Mexico, 14–16 November 2018;pp. 1–6. [CrossRef]59. M, Y.; Cheng, Y.; Hossain, J.; Ahmed, M.S.G. RADB: Random Access with Differentiated Barring forLatency-Constrained Applications in NB-IoT Network. Wirel. Commun. Mob. Comput. 2018, 2018, 6210408.60. Jeon, W.S.; Seo, S.B.; Jeong, D.G. Effective Frequency Hopping Pattern for ToA Estimation in NB-IoT RandomAccess. IEEE Trans. Veh. Technol. 2018, 67, 10150–10154. [CrossRef]61. Li, Y.; Chen, S.; Ye, W.; Lin, F. A Joint Low-Power Cell Search and Frequency Tracking Scheme in NB-IoTSystems for Green Internet of Things. Sensors 2018, 18, 3274. [CrossRef] [PubMed]62. Ha, S.; Seo, H.; Moon, Y.; Lee, D.; Jeong, J. A Novel Solution for NB-IoT Cell Coverage Expansion.In Proceedings of the Global Internet of Things Summit (GIoTS), Bilbao, Spain, 4–7 June 2018.63. Chung, H.; Lee, S.; Jeong, J. NB-IoT Optimization on Paging MCS and Coverage Level. In Proceedingsof the 15th International Symposium on Wireless Communication Systems (ISWCS), Lisbon, Portugal,28–31 August 2018; pp. 1–5.64. Živic, N. Improved Up-Link Repetition Procedure for Narrow Band Internet of Things. In Proceedings ofthe International Conference on Computational Science and Computational Intelligence (CSCI), Las Vegas,NV, USA, 13–15 December 2018.65. Zhang, L.; Ijaz, A.; Xiao, P.; Tafazolli, R. Channel Equalization and Interference Analysis for UplinkNarrowband Internet of Things (NB-IoT). IEEE Commun. Lett. 2017, 21, 2206–2209. [CrossRef]66. Yang, B.; Zhang, L.; Qiao, D.; zhao, G.; Imran, M. Narrowband Internet of Things (NB-IoT) and LTE SystemsCo-existence Analysis. In Proceedings of the 2018 IEEE Global Communications Conference (GLOBECOM),Abu Dhabi, UAE, 9–13 December 2018.67. Rusek, F.; Hu, S. Sequential channel estimation in the presence of random phase noise in NB-IoT systems.In Proceedings of the IEEE 28th Annual International Symposium on Personal, Indoor, and Mobile RadioCommunications (PIMRC), Montreal, QC, Canada, 8–13 October 2017.68. Mangalvedhe, N.; Ratasuk, R.; Ghosh, A. NB-IoT deployment study for low power wide area cellular IoT.In Proceedings of the IEEE 27th Annual International Symposium on Personal, Indoor, and Mobile RadioCommunications (PIMRC), Valencia, Spain, 4–8 September 2016.69. Malik, H.; Pervaiz, H.; Mahtab Alam, M.; Le Moullec, Y.; Kuusik, A.; Ali Imran, M. Radio ResourceManagement Scheme in NB-IoT Systems. IEEE Access 2018, 6, 15051–15064. [CrossRef]70. Yu, Y.; Tseng, S. Downlink Scheduling for Narrowband Internet of Things (NB-IoT) Systems. In Proceedingsof the IEEE 87th Vehicular Technology Conference (VTC Spring), Porto, Portugal, 3–6 June 2018.71. Liu, J.; Mu, Q.; Liu, L.; Chen, L. Investigation about the paging resource allocation in NB-IoT.In Proceedings of the 20th International Symposium on Wireless Personal Multimedia Communications(WPMC), Bali, Indonesia, 17–20 December 2017.72. Kim, H.; Cho, S.C.; Lee, Y. Interference Analysis of Guardband NB-IoT System. In Proceedingsof the International Conference on Information and Communication Technology Convergence (ICTC),Sydney, NSW, Australia, 4–7 June 2017.73. Hsieh, B.; Chao, Y.; Cheng, R.; Nikaein, N. Design of a UE-speciﬁc uplink scheduler for narrowbandInternet-of-Things (NB-IoT) systems. In Proceedings of the 3rd International Conference on IntelligentGreen Building and Smart Grid (IGBSG), Yi-Lan, Taiwan, 22–25 April 2018.70Sensors 2019, 19, 261374. Andres-Maldonado, P.; Ameigeiras, P.; Prados, J.; Ramos, J.; Navarro-Ortiz, J.M.; Lopez-Soler, J. AnalyticAnalysis of Narrowband IoT Coverage Enhancement Approaches. In Proceedings of the 2018 Global Internetof Things Summit (GIoTS), Bilbao, Spain, 4–7 June 2018; pp. 1–6.75. Yu, C.; Yu, L.; Wu, Y.; He, Y.; Lu, Q. Uplink Scheduling and Link Adaptation for Narrowband Internet ofThings Systems. IEEE Access 2017, 5, 1724–1734. [CrossRef]76. Jiang, Z.; Han, B.; Chen, P.; Yang, F.; Bi, Q. OnNovel Access and Scheduling Schemes for IoT Communications.Mob. Inf. Syst. 2016, 2016, 1–9. [CrossRef]77. Oh, S.; Shin, J. An Efﬁcient Small Data Transmission Scheme in the 3GPP NB-IoT System. IEEE Commun. Lett.2017, 21, 660–663. [CrossRef]78. Cluzel, S.; Franck, L.; Radzik, J.; Cazalens, S.; Dervin, M.; Baudoin, C.; Dragomirescu, D. 3GPP NB-IOTCoverage Extension Using LEO Satellites. In Proceedings of the IEEE 87th Vehicular Technology Conference(VTC Spring), Porto, Portugal, 3–6 June 2018.79. Lauridsen, M.; Krigslund, R.; Rohr, M.; Madueno, G. An Empirical NB-IoT Power Consumption Model forBattery Lifetime Estimation. In Proceedings of the IEEE 87th Vehicular Technology Conference (VTC Spring),Porto, Portugal, 3–6 June 2018.80. Lee, J.; Lee, J. Prediction-Based Energy Saving Mechanism in 3GPP NB-IoT Networks. Sensors 2017, 17, 2008.[CrossRef] [PubMed]81. Bello, H.; Jian, X.; Wei, Y.; Chen, M. Energy-Delay Evaluation and Optimization for NB-IoT PSM withPeriodic Uplink Reporting. IEEE Access 2019, 7, 3074–3081. [CrossRef]82. Khan, S.; Malik, H.; Alam, M.; Le Moullec, Y. DORM: Narrowband IoT Development Platform and IndoorDeployment Coverage Analysis. In Proceedings of the 2nd International Workshop on Recent Advances inCellular Technologies and 5G for IoT Environments (RACT-5G-IoT 2019), Leuven, Belgium, 29 April–2May 2019.83. Kim, T.; Kim, D.M.; Pratas, N.; Popovski, P.; Sung, D.K. An Enhanced Access Reservation Protocol Witha Partial Preamble Transmission Mechanism in NB-IoT Systems. IEEE Commun. Lett. 2017, 21, 2270–2273.[CrossRef]84. 3GPP. Architecture Enhancements to Facilitate Communications with Packet Data Networks andApplications. 2017. Available online: https://portal.3gpp.org/desktopmodules/Speciﬁcations/SpeciﬁcationDetails.aspx?speciﬁcationId=862 (accessed on 20 April 2019).85. 3GPP. TS 23.401 V14.3.0. 2017. Available online: https://portal.3gpp.org/desktopmodules/Speciﬁcations/SpeciﬁcationDetails.aspx?speciﬁcationId=849 (accessed on 25 April 2019).86. Andres-Maldonado, P.; Ameigeiras, P.; Prados-Garzon, J.; Navarro-Ortiz, J.; Lopez-Soler, J.M. NarrowbandIoT Data Transmission Procedures for Massive Machine-Type Communications. IEEE Netw. 2017, 31, 8–15.[CrossRef]87. Mohamed, M.O.; Abdelhamid, B.; El Ramly, S. Interference mitigation in heterogeneous networks usingFractional Frequency Reuse. In Proceedings of the International Conference on Wireless Networks andMobile Communications (WINCOM), Fez, Morocco, 26–29 October 2016.88. Monteiro, N.; Mihovska, A.; Rodrigues, A.; Prasad, N.; Prasad, R. Interference analysis in a LTE-A HetNetscenario: Coordination vs. uncoordination. In Proceedings of the Wireless VITAE 2013, Atlantic City, NJ, USA,24–27 June 2013.89. Palanisamy, P.; Nirmala, S. Downlink interference management in femtocell networks—A comprehensivestudy and survey. In Proceedings of the International Conference on Information Communication andEmbedded Systems (ICICES), Chennai, India, 21–22 Feburary 2013.90. Song, S.; Li, H.; Fan, Y.; Kong, W.; Zhang, W. Downlink Interference Rejection in Ultra Dense Network.In Proceedings of the 13th APCA International Conference on Control and Soft Computing (CONTROLO),Ponta Delgada, Portugal, 4–6 June 2018; pp. 361–364.91. Chiumento, A.; Pollin, S.; Desset, C.; der Perre, L.V.; Lauwereins, R. Scalable HetNet interferencemanagement and the impact of limited channel state information. EURASIP J. Wirel. Commun. Netw.2015, 2015, 74. [CrossRef]92. Oo, T.; Tran, N.H.; Saad, W.; Niyato, D.; Han, Z.; Hong, C. Ofﬂoading in HetNet: A Coordination ofInterference Mitigation, User Association, and Resource Allocation. IEEE Trans. Mob. Comput. 2017,16, 2276–2291. [CrossRef]93. Liou, R.H.; Lin, Y.B.; Tsai, S.C. An Investigation on LTE Mobility Management. IEEE Trans. Mob. Comput.2013, 12, 166–176. [CrossRef]71Sensors 2019, 19, 261394. Karandikar, A.; Akhtar, N.; Mehta, M. Mobility Management in LTE Networks. In Mobility Management inLTE Heterogeneous Networks; Springer: Berlin, Germany, 2017.95. Francois, F.; Abdelrahman, O.H.; Gelenbe, E. Impact of Signaling Storms on Energy Consumption andLatency of LTE User Equipment. In Proceedings of the 2015 IEEE 17th International Conference on HighPerformance Computing and Communications, New York, NY, USA, 24–26 August 2015.96. Koc, A.T.; Jha, S.C.; Vannithamby, R.; Torlak, M. Device Power Saving and Latency Optimization in LTE-ANetworks Through DRX Conﬁguration. IEEE Trans. Wirel. Comm. 2014, 13, 2614–2625.97. Nikaein, N.; Krea, S. Latency for Real-Time Machine-to-Machine Communication in LTE-Based SystemArchitecture. In Proceedings of the 17th European Wireless 2011—Sustainable Wireless Technologies,Vienna, Austria, 27–29 April 2011.98. Laya, A.; Alonso, L.; Alonso-Zarate, J. Is the Random Access Channel of LTE and LTE-A Suitable for M2MCommunications? A Survey of Alternatives. IEEE Commun. Surv. Tutor. 2013, 16, 4–16. [CrossRef]99. Cheng, J.; Lee, C.; Lin, T. Prioritized Random Access with dynamic access barring for RAN overload in3GPP LTE-A networks. In Proceedings of the 2011 IEEE GLOBECOM Workshops (GC Wkshps), Houston,TX, USA, 5–9 December 2011.c© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open accessarticle distributed under the terms and conditions of the Creative Commons Attribution(CC BY) license (http://creativecommons.org/licenses/by/4.0/).72sensorsArticleAggregated Throughput Prediction for CollatedMassive Machine-Type Communications in 5GWireless NetworksAhmed Adel Aly 1,*, Hussein M. ELAttar 2,*, Hesham ElBadawy 3,* and Wael Abbas 11 Department of Basic and Applied Sciences. Arab Academy for Science, Technology and MaritimeTransport (AASTMT), Cairo P.O. Box 2033, Egypt; wael_abass@aast.edu2 Department of Electronics and Communications Engineering. Arab Academy for Science, Technology andMaritime Transport (AASTMT), Cairo P.O. Box 2033, Egypt3 Network Planning Department, National Telecommunication Institute (NTI), Cairo 11432, Egypt* Correspondence: a.adel1992@aast.edu (A.A.A.); hattar@aast.edu (H.M.E.); heshamelbadawy@ieee.org (H.E.)Received: 3 July 2019; Accepted: 17 August 2019; Published: 22 August 2019Abstract: The demand for extensive data rates in dense-traﬃc wireless networks has expandedand needs proper controlling schemes. The ﬁfth generation of mobile communications (5G) willaccommodate these massive communications, such as massive Machine Type Communications(mMTC), which is considered to be one of its top services. To achieve optimal throughput, which isconsidered a mandatory quality of service (QoS) metric, the carrier sense multiple access (CSMA)transmission attempt rate needs optimization. As the gradient descent algorithms consume a longtime to converge, an approximation technique that distributes a dense global network into localneighborhoods that are less complex than the global ones is presented in this paper. Newton’s methodof optimization was used to achieve fast convergence rates, thus, obtaining optimal throughput.The convergence rate depended only on the size of the local networks instead of global denseones. Additionally, polynomial interpolation was used to estimate the average throughput of thenetwork as a function of the number of nodes and target service rates. Three-dimensional planes ofthe average throughput were presented to give a profound description to network’s performance.The fast convergence time of the proposed model and its lower complexity are more practical thanthe previous gradient descent algorithm.Keywords: 5G; mMTC; IoT; CSMA; SINR; throughput; polynomial interpolation1. IntroductionThe evolution of the ﬁfth generation of cellular mobile systems (5G) has become one of the mostsigniﬁcant ﬁelds for commercial applications. The 5G system is promising to increase data rates by10 times that of the traditional Long-Term Evolution (LTE) networks, to an average of 10 Gbps witha 1 ms round-trip latency. This high bandwidth is to accommodate an enormous number of connecteddevices per unit area under the Internet of Things (IoT) framework [1]. In fact, the 5G requirementcovers a wide range of core services, speciﬁcally the massive Machine-Type Communications (mMTC)is one of the top three services. The other core services being the ultra-reliable low latency (URLLC) andthe extreme mobile broadband (eMBB) communications [2]. The services in mMTC are deﬁned by largenumbers of linked devices that are generally transmit data traﬃc. It includes algorithms, mechanisms,and techniques that permit the exchange of information or data without explicit human involvement.Recent research studies have shown that most of the existing machine-type communications suﬀerfrom limited coverage and access reservation. Currently, the procedure for reserving access is limitedto a low number of devices and each device requires high data rates [3]. The main challenge in mMTCSensors 2019, 19, 3651; doi:10.3390/s19173651 www.mdpi.com/journal/sensors73Sensors 2019, 19, 3651is the need for eﬃcient connectivity for this massive number of devices that share packets of data.Additionally, mMTC suﬀers from losses in data packets due to heavy traﬃc and congestions. a properway to overcome these data losses is to provide suitable quality of service (QoS) requirements such ashigh network throughput with low latency.In [4], an overview of key radio resource management techniques for 5G dense small cellswas studied. Preliminary system-level simulation results indicated that a mean throughput gain ofaround 63% and up to 84% in latency reduction can be achieved by utilizing resource managementtechniques. In [5], an eﬃcient online scheme was proposed for predicting channel state informationfrom historical data, in 5G wireless communication systems. The experiment results showedthat the scheme not only obtained the predicted channel state information values very quicklybut also achieved highly accurate predictions with up to 2.650%–3.457% average diﬀerence ratiobetween the prediction and measurements. In [6], a new Machine-to-Machine (M2M) communicationparadigm based on cognitive radio technology was studied, namely the cognitive M2M communication.The cognitive M2M network architecture and cognitive machine model is presented and the coexistenceof cognitive M2M devices in TV white spaces was discussed. Additionally, a spectrum explorationscheme motivated by energy-eﬃciency was introduced. Numerical results show important energysavings and eﬃciency in providing smart grid data transmission. In [7], device-to-device (D2D)energy-eﬃcient resource allocation algorithm was introduced. To enhance QoS eﬃciency, a distributedinterference mitigation mechanism consisting of a method for canceling interference and a methodfor optimizing transmission power constraint was discussed. Simulations analyze the achievableperformance of the proposed algorithm and discuss implementation and complexity. Additionally,intelligent energy management based on the safe transfer of information between millions of sensorsand actuators installed with little or no human involvement was developed in [8]. By investigating theinclusion of software-deﬁned networking with machine-to-machine communication, this motivatesthe study of a coherent communication structure for intelligent energy management. The proposedsoftware-deﬁned machine-to-machine system was described, with a focus on its price reduction,resource allocation, and end-to-end service quality assurance. In [9], two-stage access control andresource allocation algorithm were developed. In the ﬁrst phase, a contract-based incentive systemwas introduced to motivate some delay-tolerant machine-type communication equipment. a long-termcross-layer online resource allocation method was suggested in the second phase, which optimizedrate control, energy allocation, and channel choice, without previous channel state information. Finally,under diﬀerent simulation situations, the performance of the suggested algorithm was veriﬁed.On the other hand, optimizations regarding carrier sense multiple access with collision avoidance(CSMA/CA) have met with great success in diﬀerent applications. Research on CSMA/CA hasa long tradition for years, on which a node senses the channel before transmitting on a sharedtransmission medium to avoid collision of data in wireless networks [10]. Recent papers [11–14]proposed various CSMA/CA scheduling algorithms that are able to optimize network QoS metrics,particularly the network throughput. In [11], a CSMA scheme was formulated in which throughputand power consumption of each node were optimized by controlling back-oﬀ and sleeping timers,while ensuring throughput optimality. In [12], link throughput was analyzed by taking back-oﬀcollisions into account; a model was formed to characterize the collision eﬀect among the network’snodes. Results showed that their model was robust against diﬀerent network topologies. In [13],the performance of CSMA network’s throughput was studied under the signal to interference and noiseratio (SINR) model, where a packet was received as long as a certain SINR threshold was exceeded.In [14], they provided eﬀective carrier sensing threshold adjustment algorithms for large wirelessCSMA networks. Simulation for evaluating consistency and goodput guaranteed safe interference.They also introduced dynamic signal detection thresholds depending on neighboring transmissionfeedback. In [15], a distributed iterative algorithm was studied, which produced approximate solutionsmotivated by an approximation that allowed the expression of approximate solutions via a certainnon-linear system with a polynomial size. Numerical results showed that the algorithm produced74Sensors 2019, 19, 3651highly accurate solutions and converged much faster than previous studies. In [16], a distributedscheduling algorithm for the SINR model was studied and proved to be throughput optimal. Further,the algorithm was augmented by using a parallel update technique and the numerical results showeda good performance in terms of a supportable throughput and the convergence rate to steady-state.Moreover, a random access channel evaluation and load estimation of a large number of MTC deviceswere developed in [17]. a closed-form expression and an eﬀective approach for achieving the JointProbability Distribution Function (PDF) were extracted from the amount of eﬀective and collided accessrequests within a random access opportunity. Numerical results justiﬁed their formulation’s eﬃciencyand demonstrated that the computational cost was smaller than that of other similar techniques.There has been extensive consideration to the issue of connection scheduling for peak throughputperformance with a focus on the maximum weight scheduling model established in [18,19]. Despite itsoptimization characteristic, a central controller is needed. In addition, solving for each schedulechoice is a non-deterministic polynomial-time hard (NP-hard) problem. Various studies have triedto modify the algorithm of maximal weight to make it easier to deploy [20–22]. Such methods aregreedy and might not, however, attain an optimized performance. a series of articles [23–25] optimizedthe throughput calculations for a group of connection scheduling schemes called adaptive CSMA,which can, thus, maintain any attainable rate. In particular, the transmission attempt rate was tunedby each connection to guarantee adequate average desired rates of service.In [26], the transmission attempt rate was adjusted in the CSMA algorithm to support the requiredtarget service rates. This technique poses a problem in adjusting the transmission attempt rateparameter, as it is an NP-hard problem, which is diﬃcult to handle. In addition, most of the research inthis ﬁeld aims at solving this problem using a stochastic gradient descent algorithm which is an iterativeoptimization method for diﬀerentiable objective functions [23]. The drawback of this method is that itconsumes millions of iterations to converge. Unfortunately, this approach results in an impracticaltime of convergence depending on the size of the network. Few studies focus on using a proper SINRmodel for interference, as most of the studies just settled with using interference model based onconﬂict graph [23–25].The work in this paper aimed to enhance the performance of global dense networks with a CSMAscheduling algorithm under a more practical and realistic SINR model, to adequately capture thecomplexity of wireless network interferences. In addition, by utilizing an approximation techniqueto overcome large network sizes, the optimization problem could be solved as the approximationtechnique distributes large networks into smaller ones. The size of the network and node distancewith its neighbors is independent of the size of the whole large network. Thus, the solution tosuch optimization is achievable due to the scale of the small networks. This means that the globaloptimization function of the transmission attempt rate parameter in CSMA is distributed into localoptimization functions for each node and its neighbors. The local optimization function is thensolved using Newton’s method of optimization instead of stochastic gradient descent, as it has a fasterconvergence rate [26]. The achievable service rate of each link is then calculated and its percentageerror with the target service rate is formulated. In addition, the average throughput is ﬁnally calculated.The whole process is repeated under diﬀerent network operational parameters as SINR thresholds,target service rates, and the number of nodes, to emphasize their eﬀect on the average throughput.The main contribution of the presented work is giving a full description of the eﬀect of changingnetwork operational parameters on the average throughput and proving asymptotic relations usingpolynomial interpolation that describes the following.• Throughput as a function of both target service rates and SINR threshold for a given numberof nodes.• Throughput as a function of both the number of nodes and the target service rates for a givenSINR threshold.• Maximum throughput for a diﬀerent number of nodes at diﬀerent SINR thresholds.75Sensors 2019, 19, 3651The proposedmodel proves its robustness against the increasing number of nodes due to its dependenceon the local network size instead of the global one. The model also provides eﬃcient asymptoticthroughput relations to be used for estimating the performances of such wireless networks.The rest of the paper is organized as follows. In Section 2, the system model is described,Section 2.1 describes the used CSMA scheduling algorithm and the research problem is explained.In Section 2.2, the global optimization function is introduced while the distributed local networksalgorithm and its model are described in Section 2.3. In Section 2.4, the computational complexityis explained, Section 2.5 discusses the polynomial interpolation technique, and Section 2.6 analyzesthe delay performance. In Section 3, the numerical analysis of the proposed model is obtained anddescribed. Results are discussed in Sections 4 and 5 concludes the work and results.The used parameters and variables in the following sections and their descriptions are summarizedand given in Table 1.Table 1. System parameters and their descriptions.Parameter/Variable Descriptiondii Distance between transmitter and receiver of same node i.dij Distance between two nodes i and j.N Total number of nodes.Nj Number of nodes at node j neighborhood.x(t) Schedule of the network where xi(t) = 1 means that link i is active andtransmitting data and xi(t) = 0 means that link i is not active.R Close-in-Radius distance where interference is neglected if distances betweennodes exceeded it.SINR Signal to Interference and Noise Ratio.pi Transmit power of link i.α Path loss exponent.ω The variance of the Gaussian thermal noise present at all receivers.T SINR threshold that has to be exceeded to ensure successful data reception.I List of all feasible schedules.λ Transmission attempt ratesiThe long-term service rate of node i which is the marginal probability thatnode i is active.1(x ∈ I) The indicator for the feasibility of the schedule.z Normalizing constant.r Transmission aggressiveness.ykThe feasible schedule such that yk = 0 means that node k is inactive whileyk = 1 indicates that node k is active and meets the required SINR threshold.Th Average Normalized Throughput.Th_max Maximum Normalized Throughpute(st)The approximate error between the achieved and the target service rates.P(x) the stationary distribution of the CSMA Markov chain2. System ModelThe model used in this article is based on a single-hop wireless network. Each node is formed froma pair of transmitter and receiver similar to the bipole model in [27], the distance between transmitterand receiver is dii for node i. N is deﬁned as the total number of nodes in the network model. Let dij bethe distance between two nodes i and j.For scheduling data transmissions between nodes, x(t) is deﬁned as the schedule of the network;it can also be referred to as x. In other words, xi(t) = 1 indicates that node i is active at time slot t andduring data transmission. Two nodes are considered to be neighbors and interfere with each other ifthe distance between them is less than or equals to R (Close-in-Radius). Interference between nodes isneglected if they are not neighbors or, in other words, the distance between them is higher than R asshown in Figure 1.76Sensors 2019, 19, 3651Figure 1. Local Neighborhood network as a part of the global network, the maximum distance betweena node (a pair of transmitter and receiver) and its neighbors is the Close-in-Radius distance RThe SINR model at node i as a function of x(t) is given by:SINR =Pidii−α∑{ j∈ Nj , jI, xj(t)=1} Pj dji−α +ω, (1)where Pj is the transmitting power of node j, α is the path loss exponent of the standard path lossmodel |d|−α. ω is the variance of the Gaussian thermal noise present at all receivers. Nj is a set of nodej and its neighbors.The condition for xi(t) = 1 (Node i is active and transmitting data) is SINR ≥ T; T is the SINRpresumed threshold constraint. If all active nodes in a schedule satisfy this condition, their schedule iscalled a feasible schedule. The list of all feasible schedules is deﬁned as I, where receivers are able toreceive data successfully.2.1. CSMA Scheduling AlgorithmThe main aim of the CSMA scheduling algorithm is to ensure proper data packets reception.The procedure objective is obtaining the transmission attempt rate λ, hence supporting the requiredtarget service rates of each node for acquiring the optimal throughput. The scheduling algorithmis based on the SINR readings of the previous time slot, to ensure successful transmission of data ifa certain threshold T is exceeded. The process is described in the following CSMA SINR thresholdscheduling Algorithm 1.Algorithm 1. CSMA SINR threshold scheduling algorithm-Each node is assigned with transmission attempt rate λ > 0.-In each time slot, a randomly selected node i is allowed to update its schedule xi(t) based on the informationin the previous time slot.if SINRi(x(t− 1)) < T, thenxi(t) = 0 and node i waits for another time slot to update its schedule again.else if SINRi(x(t− 1)) ≥ T, thenNode i exchanges messages with neighbors, to ﬁnd if they can meet their SINR requirements if link i getsactivated.if any of its neighbors can’t meet its requirement, thenxi(t) = 0 and node i waits for another time slot to update its schedule again.else if all neighbors can meet their SINR requirements if link i gets activated, thenxi(t) = 1 with probabilityλi1+λiand xi(t) = 0 with probability 11+λi .end ifend if77Sensors 2019, 19, 3651It can be shown in [25] that the adaptive CSMA algorithm induces a Markov chain on the statespace of the schedules {0, 1}N. Further, the stationary distribution of the Markov chain, parametrized bythe transmission attempt rate vector λ = [λi]Ni=1, is given by:p(x) =1z∏j:xj=1λ j 1(x ∈ I ),∀ x ∈ {0, 1}N , (2)where 1(x ∈ I) is an indicator of x being a feasible schedule, and z is the normalizing constant. Then,due to the ergodicity of the Markov chain, the long-term service rate of a node i denoted by si isequal to the marginal probability that node i is active, i.e., pi(xi = 1). Thus, the service rates and thetransmission attempt rates are related as follows:si = pi(1) =∑x: xi=11z∏j:xj=1λ j,∀ i ∈ N , (3)where pi(1) denotes pi(xi = 1). The adaptive CSMA algorithm can support any service provided thatappropriate transmission attempt rates are used for the underlying distribution [28]. If the desiredservice rates are known, these transmission attempt rates can be obtained by solving the system ofequations in Equation (3).Assume that each node i has a capacity of 1. If node i transmits data all the time (without aﬀectingother nodes), then its service rate is 1 (unit of data per unit time). Then, si(r) is also the normalizedservice rate with respect to the node capacity.The following concave function G(r) can be maximized by choosing a suitable value ofthe transmission aggressiveness r, this maximization is equivalent to the minimization of theKullback–Leibler divergence between the arrival rate and the service rate distribution functionsas established in [29].2.2. The Global Optimization FunctionThe global optimization function of the proposed network model is given by:r = argmax r∈RN G(r)G(r) =∑k∈Nskrk − ln⎛⎜⎜⎜⎜⎜⎜⎝∑y∈Iexp⎛⎜⎜⎜⎜⎜⎝∑k∈Nykrk⎞⎟⎟⎟⎟⎟⎠⎞⎟⎟⎟⎟⎟⎟⎠ , (4)where r is the transmission aggressiveness vector of dimension N and it is a function of λ as r = ln(λ).let y = [yk]k∈N ∈ {0, 1}N be the global feasible schedule such that yk = 0 means that node k is inactivewhile yk = 1 indicates that node k is active and meets the required SINR threshold. The desired servicerates vector is denoted as {si}i∈N.For a proper understanding of the global optimization Equation (4), let ∂G(r)∂ri = 0 to show that itsolves Equation (3) and results in:si =∑y∈I:yi=1 exp(∑Nk=1 ykrk)∑y∈I exp(∑Nk=1 ykrk) ∀ i ∈ N , (5)To solve Equation (4), the distributed stochastic gradient descent algorithm was used [23].However, the gradient of (4) estimation was calculated in a distributed manner and took an impracticaltime of convergence in order to reach steady state. In order to rectify this problem, the proposedglobal optimization function was divided into separate and scalable approximated local optimizationfunctions and, ﬁnally, these local functions were appropriately combined for estimating the solution tothe global problem. The target service rates were assumed as predeﬁned for each node.78Sensors 2019, 19, 36512.3. The Local Optimization FunctionThe local optimization function was similarly structured as the global one with some parametersreplaced by ones with a j index to represent its local attachment to node j and its neighbors. The localoptimization function of node j was deﬁned as:rj = argmax r∈RNj F(r)F(r) =∑k∈Njskrk − ln⎛⎜⎜⎜⎜⎜⎜⎜⎝∑y∈Ijexp⎛⎜⎜⎜⎜⎜⎜⎜⎝∑k∈Njykrk⎞⎟⎟⎟⎟⎟⎟⎟⎠⎞⎟⎟⎟⎟⎟⎟⎟⎠ , (6)where rj = [rjk]k∈ Nj is the local transmission aggressiveness of node j, y = [yk]k∈Nj ∈ {0, 1}Nj is thelocal feasible schedule at node j such that yj = 0 means that node j is inactive and yj = 1 indicates thatnode j is active and exceeds the SINR threshold. s( j) = {sk∣∣∣k ∈ Nj} is the local service rate vector ofnode j.Due to the downscaling of the global network dimensions, the solution might be simpliﬁedby solving local optimization functions. The local solutions of the transmission attempt rate arethen combined to produce a global transmission attempt rate that can be directly used in the CSMAalgorithm. This process is chosen over the adaptation of transmission attempt rates using a stochasticgradient descent that requires extensive time to converge on the global function. Each node in thenetwork executes (Algorithm 2) in parallel to get the average normalized throughput.Algorithm 2. Proposed Algorithm to obtain Average Normalized ThroughputInput:(sk, k ∈ Nj); Output: Average Normalized Throughput Th(st)(1) Neighbors of node j provide their target service rates.(2) Local optimization function (6) is solved using Newton’s method of optimization for node j with itssurrounding neighbors and the maximum achievable service rates is locally obtained for all feasibleschedules of the neighborhood.(3) Node j and each neighbor from k ∈ Nj provide their locally maximum achievable service rates.(4) The average achievable service rate can be calculated for each node separately by averaging the servicerates in each neighborhood contained in that node.(5) Calculate the approximate error between the achieved and the target service rates:e(st)=∑Ni=1∣∣∣sti − sai ∣∣∣N, (7)where sa = [sai ]Ni=1 are the service rates that can be achieved and st = [sti ]Ni=1 is the given target servicerate vector.(6) Compute the average normalized throughput:Th(st)=∑Ni=1 sti(1− e(st))N, (8)(7) Use Polynomial interpolation to form the average normalized throughput equations as a function ofeither—number of nodes and target service rates or SINR threshold and target service rates. Polynomialinterpolation is also used to get the maximum normalized throughput as a function of the numberof nodes.79Sensors 2019, 19, 3651The approximate global transmission attempt rate λ˜ that can be directly applied to the CSMASINR threshold scheduling algorithm (Algorithm 1) is given by:λ˜ =(1− sjsj)|Nj |−1 ∏k∈Njerkj , (9)where sj is the target service rate of link j,∣∣∣Nj∣∣∣ is the number of nodes in the local neighborhood ofnode j. The transmission aggressiveness is rkj, k ∈ Nj is the optimized parameter in Step (2) of node jand its presence in every local neighborhood.Newton’s method of optimization used in Step (2) is used to optimize the local optimizationfunction (6) [30]. It can be computed without complexity due to its relatively smaller size as comparedto the global function and is given by:r(t) = r(t−1) − ([∇2F(r(t−1))]ik)−1 . [∇F(r(t−1))]k , (10)where r(t) is the transmission aggressiveness of the current iteration and r(t−1) is of the previous iteration.[∇2F(r(t−1))]ikand [∇F(r(t−1))]kare the Hessian matrix and the gradient vector of Newton’s method,respectively. The gradient and Hessian of node j computations are done through the distribution:bˆ j(x( j))=1zjexp⎛⎜⎜⎜⎜⎜⎜⎜⎝∑k∈Njxkrk⎞⎟⎟⎟⎟⎟⎟⎟⎠,∀ x( j) ∈ Ij , (11)where zj is the local normalization constant of node j and its neighbors. Ij is a list of all feasibleschedules of node j and its neighbors. The gradient of (6) is given by:[∇F(r)]k = sk −mk(r), k ∈ Nj , (12)where mk(r) = p(xk = 1) under bˆ j distribution for k ∈ Nj.The Hessian of (6) is given by:[∇2F(r)]ik ={mi(r)mk(r) −mik(r), i, k ∈ Nj, i jmk(r)2 −mk(r), i = k. , (13)where mik(r) = p(xi = 1, xk = 1) of the distribution bˆ j.2.4. Computational ComplexityIt is possible to implement the Newton method in (Algorithm 2). This is because it is possible toanalytically calculate the gradient and the Hessian of the local objective function F(r) since the problemdimension is reduced from the global one.The gradient and the Hessian calculations need information about the feasible local schedulesat a link. This information is speciﬁcally needed to calculate the normalization constant z.These computations are feasible because of the complexity of O(2|Nj |) associated with a computationscale of only a particular size of the local neighborhood that is independent of the network’s globalsize, which could be considerably large.2.5. Polynomial InterpolationThe polynomial cubic interpolation used in Step (7) is explained in [31–33] and has a commonform of:f (θ) = a0 + a1θ+ a2θ2 + . . .+ anθn, an 0 , (14)where θ is the variable of the function f and n is the degree of the polynomial.80Sensors 2019, 19, 3651The solution of Equation (14) is computed using the Vandermonde matrix [33] formed inEquation (15), in order to calculate the coeﬃcient an given both θ and f :⎡⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣1 θ1 θ21 · · · θm11 θ2 θ22 · · · θm21 θ3 θ23 · · · θm3.......... . ....1 θn θ2n · · · θmn⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦⎡⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣a0a1a2...am⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦=⎡⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣f0f1f2...fn⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦, (15)Another form of the polynomial interpolation used in this work is the bicubic interpolation fora function of two variables, such as:f (θ,ϕ) = a0 + a1θ+ a2ϕ+ a3θ2 + a4θϕ+ a5ϕ2 + a6θ3 + a7θ2ϕ+ a8θϕ2 + a9ϕ3, (16)where θ and ϕ are the two variables of the function f .The solution of Equation (16) is similar to Equation (14) and is computed using the Vandermondematrix formed in Equation (17) to calculate the coeﬃcient an given θ, ϕ, and f :⎡⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣1 θ1 ϕ1 θ21 θ1ϕ1 · · · ϕm11 θ2 ϕ2 θ22 θ2ϕ2 · · · ϕm21 θ3 ϕ3 θ23 θ3ϕ3 · · · ϕm3................ . ....1 θn ϕn θ2n θnϕn · · · ϕmn⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦⎡⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣a0a1a2...am⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦=⎡⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣f0f1f2...fn⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦, (17)Equations (15) and (17) are solved using the Gaussian elimination method [34] until a reducedechelon form is reached; hence, the values of the coeﬃcients are computed. In Step (7),Polynomial interpolation is used to obtain the average normalized throughput as a function ofeither number of nodes and target service rates or SINR threshold and target service rates. Polynomialinterpolation is also used to get the maximum normalized throughput as a function of the numberof nodes.2.6. Delay PerformanceConsidering the delay performance, it is known that the CSMA Markov chain’s impractically slowmixing time leads to a bad delay performance [28,35]. Recent work such as [36,37] have enhanced delayperformance by using several parallel CSMA Markov chain cases. These results [36] are demonstratedon the assumption that the ideal transmission attempt rate is pre-computed and is easily accessible tothe algorithm. The proposed local algorithm can be used in combination with the methods in [36,37]to achieve a practical CSMA algorithm with an excellent throughput and a low delay, to eﬀectivelyestimate these transmission attempt rates.3. ResultsIn this section numerical analysis are used to estimate the performance of the proposed algorithm.Random topology graphs are considered with a diﬀerent number of nodes to test the robustness ofthe model. Random networks are generated by placing nodes on a two-dimensional area of length of12 unit distance. The system parameters are summarized in Table 2.Random networks are generated with varying densities from 10 up to 100 nodes, and vertices aredrawn when nodes are neighbors and interfere with each other. The total number of links is higherthan the total number of nodes, but it does not reach the mesh topology where the number of linksequals to N(N − 1)/2. The interference graph of a 100-node random topology is shown in Figure 2.81Sensors 2019, 19, 3651Table 2. The system parameters used in numerical analysis.System Parameter ValueDimensions of the interference graph (unit area) 12 × 12Distance between transmitter and its corresponding receiver(unit distance) 0.5Path loss exponent 3Close in Radius (unit distance) 2.5SINR threshold (dB) 9 up to 15Target service rate (Unit of data per unit time) 0 up to 0.9Transmit power (unit power) 1Figure 2. Interference graphs of 100-node random topology.In order to prove the fast convergence rate of the proposed model, Figure 3 shows that the normof the gradient in Newton’s method of a random local network sample converges in 4 to 5 iterations.Figure 3. Sample of Convergence Rate of the proposed algorithm.In Figure 4, the average normalized throughput as a function of the target service rate is shownfor diﬀerent network topologies of 10, 30, 50, and 100 randomly distributed nodes at 9, 12 and 15 dBSINR thresholds.82Sensors 2019, 19, 3651(a) (b) (c) Figure 4. Average normalized throughput for a diﬀerent number of nodes—(a) 9 dB SINR threshold,(b) 12 dB SINR threshold, and (c) 15 dB SINR threshold.83Sensors 2019, 19, 3651As shown in Figure 4, increasing the SINR threshold showsdegradation in throughput, especially athigh-target service rates. This is because the nodes will be unable to transmit until a higher SINRthreshold is met. Meanwhile, it is observed that initially, as target service rate increases, the averagenormalized throughput increases until it reaches its maximum. This is where the network works under“stable operating conditions”. After that point, the network enters “unstable operating conditions”in which collisions become more likely and the number of backlogged frames increases. This meansthat the arrival rate of new frames to the system will be larger than the capability of successful framestransmission, thus, leading to a decrease of the average normalized throughput.In order to acquire the obtained results in Figure 4 as a system of asymptotic relations,polynomial bicubic interpolation algorithm is used. Therefore, the average normalized throughput canbe estimated via the following proposed equation in the general form of:Th(N,S) = a0 + a1N + a2S+ a3N2 + a4NS+ a5S2 + a6N3 + a7N2S+ a8NS2 + a9 S3, (18)where Th is the average normalized throughput as a function of the number of nodes N and targetservice rate is denoted as S. The constants a0, . . . , a9 are the constant coeﬃcients in Equation (18) andare given in Table 3.Table 3. Coeﬃcients of Equation (18) for diﬀerent SINR thresholds.SINRThreshold a0 a1 a2 a3 a4 a5 a6 a7 a8 a9T = 9 0.3476 −3× 10−4 0.559 3.5× 10−6 0.0029 1.096 10−9 −3.4× 10−5 −0.00466 −1.016T = 10 0.1716 −4.6× 10−4 −1.2147 3.4× 10−6 0.004496 5.442 10−9 −3.26× 10−5 −0.0064 −3.865T = 11 −0.314 3.65× 10−4 4.92 3.3× 10−6 −0.00486 −8.14 10−9 −3× 10−5 0.00256 4.28T = 12 −0.119 8.45× 10−5 2.49 3.18× 10−6 −0.00189 −2.97 10−9 2.8× 10−5 7.5× 10−4 1.239T = 13 −0.305 3.65× 10−4 4.896 4× 10−6 −0.00614 −8.66 10−9 −3× 10−5 0.0038 4.81T = 14 −0.3 1.9× 10−4 4.898 5.3× 10−6 −0.0059 −8.73 10−9 −3.4× 10−5 0.004 4.847T = 15 −0.0515 −4.25× 10−4 1.718 6.6× 10−6 −7.35× 10−4 −1.597 10−9 −3.76× 10−4 −5.79× 10−4 0.447The interpolation equations are used to describe the missing values between the calculatedthroughput results and also give more details in three-dimensional planes of the average normalizedthroughput as a function of both target service rates and the number of nodes. In Figure 5, the planeis represented at diﬀerent SINR thresholds of 9 up to 15 dB. It can be observed that increasing thenumber of transmitting nodes in the network will cause more collisions, thus, leading to a decrease inthe network throughput.Another way to make use of the polynomial bicubic interpolation is to generate another form ofthe previous equation with diﬀerent parameters, such as:Th(T,S) = c0 + c1T + c2S+ c3T2 + c4 TS+ c5S2 + c6T3 + c7T2S+ c8TS2 + c9 S3, (19)where Th is the average normalized throughput as a function of target service rate denoted as S andthe SINR threshold T. The constants c0, . . . , c9 are the constant coeﬃcients in Equation (19) and aregiven in Table 4.Moreover, the previous step of interpolation is repeated to ﬁnd an asymptotic relation of theaverage normalized throughput as a function of both target service rates and the SINR threshold.In Figure 6, the plane is shown with a number of nodes from 10 up to 100 nodes.84Sensors 2019, 19, 3651 (a) (b) (c) Figure 5. Plane of average normalized throughput generated from polynomial interpolation ata diﬀerent number of nodes for—(a) 9 dB SINR threshold, (b) 12 dB SINR threshold, and (c) 15 dBSINR threshold.85Sensors 2019, 19, 3651Table 4. Coeﬃcients of Equation (19) for diﬀerent number of nodes.No. ofNodes c0 c1 c2 c3 c4 c5 c6 c7 c8 c9N = 10 −0.028 −0.00138 1.32 8.95× 10−5 0.0166 −0.315 10−6 8.95× 10−4 −0.028 −0.128N = 20 −0.0367 0.0025 1.25 −1× 10−4 0.0015 −0.112 10−6 −1.66× 10−4 −0.03 −0.219N = 30 −0.069 0.0063 1.5 −2.89× 10−4 −0.0135 −0.566 10−7 −5.66× 10−4 −0.0324 0.0755N = 40 −0.077 0.01069 1.43 −4.8× 10−4 −0.0328 −0.313 10−7 0.00129 −0.03 −0.0665N = 50 −0.449 0.01425 0.8649 −6.74× 10−4 −0.0437 0.9377 10−8 0.002 −0.0365 −0.752N = 60 −0.0276 0.0143 0.64 −6.76× 10−4 −0.0428 1.339 10−8 0.00197 −0.036 −1.025N = 70 −0.0265 0.0143 0.623 −6.79× 10−4 −0.0419 1.29 10−8 0.0019 −0.036 −1.04N = 80 −0.134 0.0165 1.942 −6.8× 10−4 −0.0647 −1.38 10−8 0.00185 −0.012 0.366N = 90 −0.0599 0.0156 1.006 −6.8× 10−4 −0.052 0.516 10−9 0.0018 −0.023 −0.77N = 100 −0.061 0.0156 1.019 −6.86× 10−4 0.0517 0.409 10−9 0.00175 −0.0229 −0.749After obtaining the average normalized throughput from the numerical analysis, the maximumnormalized throughput could be extracted. Polynomial cubic interpolation algorithm was used toconcatenate the above ﬁgures and deﬁne a relation for acquiring the maximum normalized throughputin the general form of:Th_max(N) = k0 + k1N + k2N2, (20)where Th_max is the maximum normalized throughput as a function of the number of nodes N.The constants k0, k1, and k2 are the constant coeﬃcients in Equation (20) and are given in Table 5.Table 5. Coeﬃcients of Equation (20) for diﬀerent SINR thresholds.SINR Threshold k0 k1 k2S = 9 0.6989 −0.003 −4× 10−7S = 12 0.6282 0.0033 −7.48× 10−8S = 15 0.551 −0.00278 −1.2× 10−7The interpolation equations graphs of the maximum normalized throughput are shown in Figure 7as a function of the number of nodes at the diﬀerent SINR thresholds from 9 up to 15 dB.Another way to describe the maximum normalized throughput more generally, using thepolynomial bicubic interpolation is to formulate a plane that is a function of, both, the number of nodesand the SINR threshold. Therefore, the maximum normalized throughput was estimated using thefollowing equation as the general form:Th_max(N,T) = e0 + e1N + e2T + e3N2 + e4NT + e5T2, (21)where Th_max is the maximum normalized throughput as a function of the number of nodes N andthe SINR threshold denoted as T. The constants e0, . . . , e5 are the constant coeﬃcients in Equation (21)and are given in Table 6.86Sensors 2019, 19, 3651(a) (b) (c) Figure 6. Plane of average normalized throughput generated from the polynomial interpolation fordiﬀerent SINR thresholds of—(a) 10-node topology, (b) 50-node topology, and (c) 100-node topology.87Sensors 2019, 19, 3651Figure 7. Maximum normalized throughput at diﬀerent number of nodes for SINR threshold of 9 dB,12 dB, and 15 dB.Table 6. Coeﬃcients of Equation (21).Coeﬃcient e0 e1 e2 e3 e4 e5Value 0.9213 −0.00345 −0.025 −8.47× 10−6 9.81× 10−5 10−5As a result to Equation (21), the maximum normalized throughput as a function of both thenumber of nodes and the SINR threshold can be generated approximately. Diﬀerent from Equation (20),the new equation adds the dimension of the SINR threshold to give a more general description andeases the prediction to be based on two parameters instead of one. Equation (21) is shown in Figure 8for diﬀerent number of nodes and SINR thresholds.Figure 8. Maximum normalized throughput as a function of both the number of nodes and theSINR threshold.A sample of error using Equation (7) between the proposed approximation technique andthe stochastic gradient descent algorithm is shown in Figure 9. Where the local approximationalgorithm that calculates the approximate transmission attempt rates uses their static values in theCSMA algorithm (i.e., they are not adapted during the algorithm). On the other hand, the stochasticgradient descent algorithm begins with some initial transmission attempt rates, and by observing thecorresponding service rates, it adapts the transmission attempt rates [23].88Sensors 2019, 19, 3651Figure 9. The approximate error of the achievable and the target service rates due to stochastic gradientdescent and the proposed approximation algorithm with 50 nodes distributed with the same randomtopology used in this article.For 5G networks, the error has to be minimum at all target service rates. Additionally, determiningthe transmission attempt rate is an NP-hard problem and the stochastic gradient descent is unpracticalas it adapts those transmission attempt rates in several iterations, as shown in the Figure 9. This is whythe proposed approach of pre-calculating those transmission attempt rates and using them directly inthe network helps in reducing the error at most target service rates. The results would be comparableeven at higher rates but there still exists an advantage in the early stages where the network worksunder “stable operating conditions”. The error of the proposed model did not exceed 0.06, while thegradient descent started with an error exceeding 0.2 for a target service rate up to 0.4 unit data perunit time.4. DiscussionThe proposed model aims at providing possible ways of predicting random network performancesunder diﬀerent circumstances. One thing to mention is that the proposed model is scalable, as it studieda 10- to 100-node random topology in an area of 12 × 12 square units of distance. This means that ifa 400-node random topology is studied at 24 × 24 square units of distance, the performance will besimilar to the proposed 100-node one. This is due to the dependence of the proposed model on the localneighborhood size, not the global one. Therefore, what matters is the performance degradation thatwould arise from increasing the number of nodes in the same limited area. Additionally, increasing thegiven area can allow for ﬁtting a higher number of nodes, as the limited area considered in this workaﬀects the interference to a great extent, due to the high-density neighborhoods interfering with eachother. Additionally, increasing the number of nodes in the given area to a value higher than the studiedones might lead to a density (number of nodes/unit area) > 1, which is rarely found and should mostof the time be ≤ 1. Additionally, the resulting model might be used for scalable networks. In otherwords, the resulted network could be used to estimate the performance of both small as well as largenetworks. The presented analysis is based on the normalized unit area and the normalized throughput,so it might provide a good performance whatever be the network size.The proposed polynomial interpolation’s throughput asymptotic relations here are based on thestudied topologies and might lead to other performance prediction if diﬀerent parameters were used.To sum up the used parameters here, the SINR threshold was set to 9, 12, and 15 dB, up to 0.9 unittarget service rates, and up to 100 nodes randomly distributed in the unit area were considered.89Sensors 2019, 19, 3651Three-dimensional topologies were applicable and would not have diﬀered much from thetwo-dimensional topology used in the proposed model, as the distance was the only matter.We previously mentioned the close-in-radius R distance as the distance where interference wasneglected if two nodes were at distance > R apart from each other. This radius could either be of a circleif a two-dimensional topology was considered or a radius of a sphere in case of the three-dimensionaltopology. Therefore, at the end what matters is distances at any directions and the three-dimensionaltopology could be equivalent to a dense network that is already studied in the proposed model.Additionally, the main scope of this paper was to calculate the eﬀect of the increasing the number ofnodes on the throughput in dense networks, which is suitable for 5G applications. The energy and thedelay could be considered in details in future work.5. ConclusionsA CSMA algorithm of a single-hop wireless network was considered under a realistic SINR model.An approximation algorithm of distributing the global network into downscaled local neighborhoodswas used to calculate the transmission attempt rate to optimize the throughput of the global network.This was done by achieving target service rates of nodes while varying the number of nodes up toa random 100-node topology. The proposed model converged fast and proved its robustness againstthe increasing number of nodes as it depended only on the size of the local network instead of theglobal dense one. Three-dimensional planes of the average normalized throughput were obtained bypolynomial interpolation that produced a complete description of the performance of the network.Maximum normalized throughput was obtained too as a function of the number of nodes usingpolynomial interpolation. The used approach of pre-calculating the transmission attempt rates andusing them directly in the network helped in reducing the error at most values of the target servicerates. Even at higher rates, results would be comparable with a gradient descent algorithm but therestill would exist an advantage in the early stages where the network would work under “stableoperating conditions”. The proposed model is also scalable, so it might provide a network performance,whatever be its size. Additionally, the proposed model has a faster convergence time and is consideredto be less complex and more practical than the previously used gradient descent algorithm.Author Contributions: Conceptualization, A.A.A. and H.E.; Investigation, A.A.A. and H.M.E.; Methodology,A.A.A. and H.M.E.; Project administration, H.E.; Resources, A.A.A.; Software, A.A.A.; Supervision, H.E., H.M.E.and W.A.; Validation, A.A.A. and H.E.; Visualization, H.E.; Writing—original draft, A.A.A.; Writing—review andediting, A.A.A., H.M.E., H.E., and W.A.Funding: This research received no external funding.Conﬂicts of Interest: The authors declare no conﬂict of interest.References1. Agiwal,M.; Roy,A.; Saxena,N.Next generation 5Gwireless networks: a comprehensive survey. IEEECommun.Surv. Tutor. 2016, 18, 1617–1655. [CrossRef]2. Marsch, P.; Bulakci, Ö.; Queseth, O.; Boldi, M. 5G System Design: Architectural and Functional Considerationsand Long Term Research; John Wiley & Sons: Hoboken, NJ, USA, 2018.3. Bockelmann, C.; Pratas, N.K.; Wunder, G.; Saur, S.; Navarro, M.; Gregoratti, D.; Vivier, G.; De Carvalho, E.;Ji, Y.; Stefanovic´, Cˇ. Towards massive connectivity support for scalable mMTC communications in 5Gnetworks. IEEE Access 2018, 6, 28969–28992. [CrossRef]4. Mahmood, N.H.; Lauridsen, M.; Berardinelli, G.; Catania, D.; Mogensen, P. Radio resource managementtechniques for eMBB and mMTC services in 5G dense small cell scenarios. In Proceedings of the IEEE 84thVehicular Technology Conference, Montréal, QC, Canada, 18–21 September 2016; pp. 1–5.5. Luo, C.; Ji, J.; Wang, Q.; Chen, X.; Li, P. Channel state information prediction for 5G wireless communications:a deep learning approach. IEEE Trans. Network Sci. Eng. 2018. [CrossRef]6. Zhang, Y.; Yu, R.; Nekovee, M.; Liu, Y.; Xie, S.; Gjessing, S. Cognitive machine-to-machine communications:Visions and potentials for the smart grid. IEEE Netw. 2012, 26, 6–13. [CrossRef]90Sensors 2019, 19, 36517. Zhou, Z.; Dong,M.; Ota, K.; Wang, G.; Yang, L.T. Energy-eﬃcient resource allocation for D2D communicationsunderlaying cloud-RAN-based LTE-A networks. IEEE Int. Things J. 2015, 3, 428–438. [CrossRef]8. Zhou, Z.; Gong, J.; He, Y.; Zhang, Y. Software deﬁned machine-to-machine communication for smart energymanagement. IEEE Commun. Mag. 2017, 55, 52–60. [CrossRef]9. Zhou, Z.; Guo, Y.; He, Y.; Zhao, X.; Bazzi, W.M. Access Control and Resource Allocation for M2MCommunications in Industrial Automation. IEEE Trans. Ind. Inform. 2019, 15, 3093–3103. [CrossRef]10. Wyglinski, A.M.; Pu, D. Digital Communication Systems Engineering with Software-Deﬁned Radio; Artech House:Norwood, MA, USA, 2013.11. Maatouk,A.; Assaad,M.; Ephrmmides,A. Energy eﬃcient and throughput optimalCSMAscheme. IEEEACMTrans. Netw. 2019, 27, 316–329. [CrossRef]12. Kai, C.; Zhang, S.; Wang, L. Impacts of packet collisions on link throughput in CSMA wireless networks.China Commun. 2018, 15, 1–14. [CrossRef]13. Sun, X. Maximum throughput of CSMA networks with capture. IEEE Wirel. Commun. Lett. 2016, 6, 86–89.[CrossRef]14. Chau, C.; Ho, I.W.; Situ, Z.; Liew, S.C.; Zhang, J. Eﬀective static and adaptive carrier sensing for densewireless CSMA networks. IEEE Trans. Mobile Comput. 2016, 16, 355–366. [CrossRef]15. Yun, S.Y.; Shin, J.; Yi, Y. CSMA using the Bethe approximation for utility maximization. In Proceedings of the2013 IEEE International Symposium on Information Theory, Istanbul, Turkey, 7–12 July 2013; pp. 206–210.16. Swamy, P.S.; Ganti, R.K.; Jagannathan, K. Spatial CSMA: a distributed scheduling algorithm for the SIRmodel with time-varying channels. In Proceedings of the IEEE Twenty First National Conference onCommunications, Mumbai, India, 27 February–1 March 2015; pp. 1–6.17. Tello-Oquendo, L.; Pla, V.; Leyva-Mayorga, I.; Martinez-Bauset, J.; Casares-Giner, V.; Guijarro, L.Eﬃcient random access channel evaluation and load estimation in LTE-A with massive MTC. IEEE Trans.Veh. Technol. 2018, 68, 1998–2002. [CrossRef]18. TassiulAs, L.; Ephremides, A. Stability properties of constrained queueing systems and scheduling policiesfor maximum throughput in multihop radio networks. In Proceedings of the 29th IEEE Conference onDecision and Control, Honolulu, HI, USA, 5–7 December 1990; pp. 2130–2132.19. Tassiulas, L.; Ephremides,A.Dynamic server allocation to parallel queueswith randomly varying connectivity.IEEE Trans. Inform. Theory 1993, 39, 466–478. [CrossRef]20. Chaporkar, P.; Kar, K.; Luo, X.; Sarkar, S. Throughput and fairness guarantees through maximal schedulingin wireless networks. IEEE Trans. Inform. Theory 2008, 54, 572–594. [CrossRef]21. Dimakis, A.; Walrand, J. Suﬃcient conditions for stability of longest-queue-ﬁrst scheduling: Second-orderproperties using ﬂuid limits. Adv. Appl. Probab. 2006, 38, 505–521. [CrossRef]22. Wu, X.; Srikant, R.; Perkins, J.R. Queue-Length Stability of Maximal Greedy Schedules in Wireless Networks.Available online: http://ita.ucsd.edu/workshop/06/papers/262.pdf (accessed on 19 August 2019).23. Jiang, L.; Walrand, J. a distributed CSMA algorithm for throughput and utility maximization in wirelessnetworks. IEEE/ACM Trans. Netw. 2010, 18, 960–972. [CrossRef]24. Rajagopalan, S.; Shah, D.; Shin, J. Network adiabatic theorem: An eﬃcient randomized protocol for contentionresolution. ACM SIGMETRICS Perform. Eval. Rev. 2009, 37, 133–144.25. Ni, J.; Tan, B.; Srikant, R. Q-CSMA: Queue-length-based CSMA/CA algorithms for achieving maximumthroughput and low delay in wireless networks. IEEE/ACM Trans. Netw. 2012, 20, 825–836.26. Swamy, P.S.; Ganti, R.K.; Jagannathan, K. Adaptive CSMA under the SINR model: Eﬃcient approximationalgorithms for throughput and utility maximization. IEEE/ACM Trans. Netw. 2017, 25, 1968–1981. [CrossRef]27. Baccelli, F.; Singh, C. Adaptive spatial Aloha, fairness and stochastic geometry. In Proceedings of the 201311th International Symposium and Workshops on Modeling and Optimization in Mobile, IEEE Ad Hoc andWireless Networks, Ibaraki, Japan, 13–17 May 2013; pp. 7–14.28. Jiang, L.; Leconte, M.; Ni, J.; Srikant, R.; Walrand, J. Fast mixing of parallel Glauber dynamics and low-delayCSMA scheduling. IEEE Trans. Inform. Theory 2012, 58, 6541–6555. [CrossRef]29. Jiang, L.; Walrand, J. Scheduling and congestion control for wireless and processing networks. Synth. Lect.Commun. Netw. 2010, 3, 1–156. [CrossRef]30. Boyd, S.; Vandenberghe, L. Convex Optimization; Cambridge University Press: Cambridge, UK, 2004.31. Chapra, S.C.; Canale, R.P. Numerical Methods for Engineers, 7th ed.; McGraw-Hill Education: New York, NY,USA, 2015.91Sensors 2019, 19, 365132. Rao, G.S. Numerical Analysis, 3rd ed.; New Age International (P) Ltd.: New Delhi, India, 2006.33. Cheney, E.W.; Kincaid, D.R. Numerical Mathematics and Computing, 6th ed.; Cengage Learning: Mason, OH,USA, 2008.34. Lay, D.C.; Lay, S.R.; McDonald, J. Linear Algebra and Its Applications, 5th ed.; Pearson: London, UK, 2016.35. Shah, D.; David, N.C.; Tsitsiklis, J.N. Hardness of low delay network scheduling. IEEE Trans. Inform. Theory2011, 57, 7810–7817. [CrossRef]36. Lee, D.; Yun, D.; Shin, J.; Yi, Y.; Yun, S.Y. Provable per-link delay-optimal CSMA for general wireless networktopology. In Proceedings of the IEEE Conference on Computer Communications (INFOCOM 2014), Toronto,ON, Canada, 27 April–2 May 2014; pp. 2535–2543.37. Kwak, J.; Lee, C.H.; Eun, D.Y. Exploiting the past to reduce delay in CSMA scheduling: a high-order Markovchain approach. ACM SIGMETRICS Perform. Eval. Rev. 2013, 41, 353–354. [CrossRef]© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open accessarticle distributed under the terms and conditions of the Creative Commons Attribution(CC BY) license (http://creativecommons.org/licenses/by/4.0/).92sensorsArticleInterference-Aware Subcarrier Allocation for MassiveMachine-Type Communication in 5G-EnabledInternet of ThingsWenjun Hou 1, Song Li 1, Yanjing Sun 1,2,*, Jiasi Zhou 1, Xiao Yun 1 and Nannan Lu 11 School of Information and Control Engineering, China University of Mining and Technology,Xuzhou 221000, China; hauomenguanu@cumt.edu.cn (W.H.); lisong@cumt.edu.cn (S.L.);jiasi_zhou@cumt.edu.cn (J.Z.); yx.tong@163.com (X.Y.); lunannan@cumt.edu.cn (N.L.)2 school of Communication and Information Engineering, Xi’an University of Science and Technology,Xi’an 710054, China* Correspondence: yjsun@cumt.edu.cnReceived: 3 August 2019; Accepted: 14 October 2019; Published: 18 October 2019Abstract: Massive machine-type communication (mMTC) is investigated as one of three typical scenesof the 5th-generation (5G) network. In this paper, we propose a 5G-enabled internet of things (IoT) inwhich some enhanced mobile broadband devices transmit video stream to a centralized controller andsome mMTC devices exchange short packet data with adjacent devices via D2D communication topromote inter-device cooperation. Since massive MTC devices have data transmission requirementsin 5G-enabled IoT with limited spectrum resources, the subcarrier allocation problem is investigatedto maximize the connectivity of mMTC devices subject to the quality of service (QoS) requirementof enhanced Mobile Broadband (eMBB) devices and mMTC devices. To solve the formulatedmixed-integer non-linear programming (MINLP) problem, which is NP-hard, an interference-awaresubcarrier allocation algorithm for mMTC communication (IASA) is developed to maximize thenumber of active mMTC devices. Finally, the performance of the proposed algorithm is evaluatedby simulation. Numerical results demonstrate that the proposed algorithm outperforms the threetraditional benchmark methods, which signiﬁcantly improves the utilization of the uplink spectrum.This indicates that the proposed IASA algorithm provides a better solution for IoT application.Keywords: 5G; internet of things; mMTC; eMBB1. IntroductionIn the future industrial internet of things (IIoT), a large number of devices including monitoringsensors and execution control units will be deployed to support factory automation and industrycontrol system [1]. Massive periodic/non-periodic data will be transferred to a centralized control unitor adjacent devices via an industry wireless network, including video monitoring information, sensingdata, operation instructions. However, due to the limited capacity and throughput of the currentcellular system, it is insufﬁcient in supporting future IoT applications with a tremendous number ofdevices and heterogeneous information trafﬁc [2].Massive machine-type communication (mMTC), as one of three typical application scenariosin the 5th-generation (5G) network, is investigated to support communication among a massivenumber of devices, which provides a feasible solution for future industrial IoT (IIoT) [3]. Due tolimited spectrum resources in the cellular system, massive devices access the wireless network in aspectrum-sharing manner in which multiple devices are allocated in the same spectrum at the sametime. Thus, the co-channel interference among devices restricts the number of devices connected tothe cellular system. Effective interference management plays a vital role in mMTC to support theSensors 2019, 19, 4530; doi:10.3390/s19204530 www.mdpi.com/journal/sensors93Sensors 2019, 19, 4530simultaneous access of more devices. The features and challenges of mMTC in IoT are as follows. First,devices in IoT need to exchange information with their neighbor devices frequently. In other words,the communication is performed between adjacent devices [4]. Second, the coding blocklengths forIoT are usually short, to reduce the transmission delay. The transmission rate cannot be estimatedby the conventional Shannon’s capacity, which assumes an inﬁnite blocklength [5]. Third, a massivenumber of devices in IoT need to be supported. Thus, efﬁcient resource allocation in mMTC needs tobe investigated to address these challenges.A wide range of works have contributed to the resource allocation problem in mMTC. In [6],the authors establish an interference model and a formulate resource allocation problem betweenusers and machine-type communication (MTC) gateways in mMTC burst scenarios. In [7], the authorsinvestigated the access management issues for MTC devices with heterogeneous quality of service(QoS) in the same cellular network. This work does not consider bandwidth utilization becausetransmission opportunities are reserved for a group of MTC devices at the same time. The authorsin [8] propose two relay schemes and transmission protocols to speciﬁcally stimulate system capacity.In a multi-cell MTC system, Kwon et al. [9] establishes the interference model and analyzes thesignal-to-interference-plus-noise-ratio (SINR) distributions and drives efﬁcient resource allocationschemes. In [10], the authors propose a dynamic resource allocation algorithm based on theestimation of the number of MTC devices to handle massive and dynamic MTC devices whilesatisfying the random access delay requirement of MTC devices. To achieve effective resourceutilization, a resource allocation metric based on statistical priority is proposed in [11]. In this way,effective resource utilization is achieved by letting MTC devices send a reduced set of their data. In [12],the authors consider a connectivity maximization problem for narrowband IoT with non-orthogonalmultiple-access (NOMA). However, articles [6–12] assume that all devices communicate with thebase station or centralized controller directly and do not consider the communications betweenadjacent devices.D2D technology, as another promising technology in 5G, can establish communication betweenadjacent nodes, which can improve the spectrum efﬁciency and ofﬂoad the load of Base Station (BS).D2D communication is introduced to mMTC system to stimulate spectrum efﬁciency and supportmore mMTC devices accessed with limited spectrum resources. However, designing better resourceallocation algorithms to manage the inter-user interference between D2D users and cellulars is thekey challenge for improving system performance. Resource allocation and interference problems ofD2D communication have been investigated in many works [13–15]. The authors in [16] propose acell sectorization scheme to alleviate the interference between cellular users and D2D users. In [17],the authors investigate interference coordination for downlink full-dimension multiple-output systemswith underlying D2D communications.Adopting D2D technology, the number of supported devices can be improved in mMTCscenarios [18]. By allowing unauthorized devices to reuse the frequency bands of authorized cellularusers, bandwidth utilization can be improved [19–21]. The literature [22] proposes a heuristicsubcarrier allocation method to set the user’s signal-to-noise ratio (SNR) threshold to meet the QoS ofthe system. The authors in [23] propose a mobile trafﬁc ofﬂoading scheme that combines small basestations with D2D ofﬂoading. The goal is to accommodate a large number of MTC connections bymaximizing the throughput of the network system. The impact of radio frequency energy harvestingon the spectral efﬁciency of the D2D-assisted MTC system is analyzed ﬁrst in [24]. In [25], the authorspropose two solutions to manage the communication between D2D devices and the BS to lighten theoverhead of MTC devices on the 5G network. However, resource allocation [19–25] mainly focuses onthroughput maximization or interference minimization. In an IIoT enabled by D2D communications,massive devices demand access to the network via D2D mode. Thus, the connectivity maximizationproblem becomes a challenging issue to tackle. To support a system in which the number of users ishigher than the number of subcarriers, a range of fair subcarrier allocation algorithms is proposed that94Sensors 2019, 19, 4530always improves the reliability [26]. However, the author only considers the scenario of mobile usersin the downlink and does not consider the inﬂuence of interference.In this paper, we investigate a D2D-enabled internet of things in which some devices (enhancedMobile Broadband (eMBB) devices) connect to the centralized controller while other devices (mMTCdevices) communicate with their adjacent devices via D2D communication to promote inter-devicecooperation in industry automation. Speciﬁcally, mMTC devices reuse the spectrum resourcewith eMBB devices. We establish the connectivity maximization problem of mMTC deviceswhile guaranteeing the QoS of eMBB devices and mMTC devices. Furthermore, we propose aninterference-aware subcarrier allocation algorithm to tackle the problem. The main contributions ofthis paper are as follows:• We establish a problem of maximizing the number of accessed mMTC pairs subject to theconstraints of QoS in a system with both eMBB and mMTC devices, which is proven to bea mixed-integer non-linear programming (MINLP) problem.• We propose an interference-aware subcarrier allocation algorithm for mMTC (IASA) consideringthe interference range of each mMTC device.• In order to evaluate the proposed algorithm, a simulation is conducted. The results demonstratethat the proposed algorithm outperforms two benchmark algorithms signiﬁcantly in terms of thenumber of mMTC pairs accessed under the same constraints.The remainder of the paper is organized as follows. The system model and assumptions iselaborated in the “System Model” section. The optimization problem and constraints are introducedin the “Problem Formulation” section. The proposed subcarrier allocation algorithm is presentedin the “Interference-Aware Subcarrier Allocation for mMTC Communication Algorithm” section.Comprehensive simulation results are provided in the “Simulation result analysis” section. Finally,we conclude the paper in the ”Conclusion” section.2. System ModelIn this paper, we investigate an industrial wireless network in which some devices, referred to aseMBB devices (such as monitoring cameras), transmit video information to a centralized controller,while devices referred to as mMTC devices (such as sensors and actuators), transmit short blocklengthpackets to their adjacent devices to promote inter-device cooperation and industrial automation.The system model is illustrated in Figure 1, whereN eMBB devices andM mMTC devices are randomlydistributed, represented by sets N = {CU1,...,CUi,...,CUN} and M ={MU1,...,MUj,...,MUM},respectively. The mMTC devices transmit information from the transmitters to receivers by theD2D communication mode, and the mMTC pairs reuse the eMBB devices’ uplink resources inorder to improve the spectrum efﬁciency. The mMTC pairs and eMBB devices are representedby MUj and CUi, respectively. Each mMTC pair is composed of one mMTC transmitter and onemMTC receiver represented by MUtj and MUrj , respectively. All available spectrum resources aredivided into sub-carriers with the same bandwidth. Each eMBB device occupies mutually orthogonalsub-carriers. Therefore, there is no co-channel interference between the eMBB devices. We assume thatthe eMBB device CUi occupies the subcarrier i. Considering the impact of devices on each other, eachmMTC pair is allowed to access no more than one subcarrier, and each subcarrier can be accessed bymultiple mMTC devices. All of the channels in the system are assumed to be quasi-static Rayleighfading channels. The channel gain remains constant for each symbol transmission period but variesindependently between different symbol periods. The parameters in the article are shown in Table 1.95Sensors 2019, 19, 4530Figure 1. System model.Table 1. System parameters.Notation Descriptiongj Gain between MUtj and MUrjgi,B Gain between CUi and the centralized controllerhi,j Interference gain between CUi and MUrjhj,B Interference gain between MUtj and the centralized controllerf j′ ,j Interference gain between MUtj′ and MUrjPci The transmit power of CUiPdj The transmit power of MUtjWe deﬁne a binary subcarrier allocation matrix A ∈ {0, 1}M×N , where αij = 1 indicates that MUjoccupies subcarrier i, otherwise αij = 0, i ∈ N , j ∈ M. Thus, the received signal of the base station onsubcarrier i isyi =√Pci gi,Bxi + ∑j∈Mαij√Pdj hj,Bxj + n0, (1)where xi and xj are the transmitted signals of the eMBB devices CUi and the mMTC transmitter MUtj ,respectively. n0 represents normalized additive white Gaussian noise, n0 ∼ CN(0, σ20). When thecentralized controller receives the uplink signals of the eMBB device CUi, the SINR at centralizedcontroller can be calculated asγicu =|gi,B|2 PciI + σ20B, (2)where B is the subcarrier bandwidth and I represents the interference caused by the mMTC pairswhich access the subcarrier i:I =M∑j=1αijPdj∣∣hj,B∣∣2 . (3)Since the eMBB devices transmit long packet data, the achievable transmission rate of the eMBBdevice CUi can be obtained by Shannon’s theorem:Ricu = B log(1+ γicu). (4)96Sensors 2019, 19, 4530The signal received by the jth mMTC pair MUj iszj =√Pdj gjxj + ∑i∈Nαij√Pci hi,jxi +M∑j′=1j′ =j∑i∈Nαij′√Pdj′ f j′ ,jxj′ + n0, (5)where the ﬁrst term is the signal receiver MUrj received from mMTC transmitter MUtj . The secondterm is the interference from the eMBB device CUi. The third term is the interference signal fromthe transmitter of the mMTC pair MUj′ that occupies the same subcarrier with mMTC pair MUj.According to (5), the SINR of the mMTC receiver MUrj can be derived asγj =Pdj∣∣gj∣∣2∑i∈N αijPci∣∣hi,j∣∣2 +∑Mj=1j′ =j∑i∈N αij′Pdj′∣∣∣ f j′ ,j∣∣∣2 + σ20 . (6)Since mMTC communication is mostly aimed at periodic monitoring data in IoT applications suchas smart cities, the length of data packets transmitted is usually very short. According to informationtheory, the rate of short packets cannot achieve the Shannon limit. Therefore, the transmission rate ofthe mMTC devices is represented by the short packet rate [27], as shown in (7):Rj = log2(1+ γj)−√1mlog2(e)Q−1(ε)√1− 1(1+ γj)2(7)where m is the block length, ε is the transmission error probability, and Q−1(x) is the inverse of theGaussian Q function.3. Problem FormulationThe optimization goal of this paper is to maximize the total number of mMTC devices accessedunder the QoS of each eMBB device. All of the mMTC devices access the network adhere to thefollowing criteria:(i) Each mMTC device is allowed to access no more than one subcarrier.(ii) To ensure the transmission quality of the eMBB devices, the interference each subcarrier cansuffer should be below a threshold.(iii) Both eMBB devices and mMTC devices should satisfy their own transmission rate requirements.When the jth mMTC pair is allowed to access the subcarrier i, the SINR of the receiver of MUj isγj =Pdj∣∣gj∣∣2Pci∣∣hi,j∣∣2 +∑Mj′=1j =jαij′Pdj′∣∣∣ f j′ ,j∣∣∣2 + σ20 . (8)Taking γj in (8) into (7), we can get the transmission rate Rj after mMTC pair MUj is allowed toaccess the subcarrier i. In order to ensure the QoS of the mMTC pair MUj, the achievable rate Rj of themMTC pair MUj should not be less than the minimum required rate Rj−min, i.e.,Rj ≥ Rj−min. (9)When the mMTC pair MUj is allowed to access the subcarrier i, the minimum transmission rate ofthe eMBB device CUi is Ricu−min. Under the QoS constraint of the eMBB devices and the mMTC pairs,97Sensors 2019, 19, 4530we maximize the number of mMTC pairs accessed in the network. Mathematically, the optimizationproblem is formulated asmaxαijN∑i=1M∑j=1αij , (10a)s.t Ricu ≥ Ricu−min , (10b)Rj ≥ Rj_min , (10c)αij ∈ {0, 1}, ∀i ∈ N j ∈ M , (10d)N∑i=1αij ≤ 1. (10e)The constraints described in (10b) and (10c) indicate that the respective minimum transmissionrate requirements of the eMBB devices and the mMTC pairs should be satisﬁed. Equation (10e) revealsthat each mMTC pair is allowed to occupy at most one subcarrier. The optimization problem is abinary optimization problem, and the traversal complexity of the problem is 2MN . In the followingsection, we propose a lower complexity algorithm named interference-aware subcarrier allocation formMTC communication.4. Interference-Aware Subcarrier Allocation for mMTC Communication AlgorithmAccording to the constraint condition (10b), the accumulated interference allowed by the eMBBdevice CUi can be derived from the minimum transmission rate Ricu−minIicu−max =Pci |gi,B|22Ricu−min − 1− σ20B. (11)When the jth mMTC pair MUj occupies the subcarrier i, the interference caused by MUj to thebase station isIj,B = Pdj∣∣hj,B∣∣2 . (12)To ensure the rate requirement of the eMBB device, the interference caused by the mMTC pairMUj should not exceed the maximum interference allowed by the subcarrier i, i.e.,Ij,B ≤ Iicu−max. (13)Therefore, a set of mMTC pairs allowed to occupy subcarrier i can be selected according toEquation (13). For all mMTC pairs that can be accessed, we ﬁrst deﬁne the normalized interferencecaused by each pair as Ij,i = Ij,B/Iicu−max. Deﬁne the interference matrix asΩ =⎡⎢⎢⎢⎢⎣I11 I12 · · · I1MI21 I22 · · · I2M...... · · · ...IN1 IN2 · · · INM⎤⎥⎥⎥⎥⎦ .When themMTCpair MUj′ , j′ = j, attempts to access the subcarrier i, the following two conditionsshould be satisﬁed.(i) The interference caused by MUj′ and the total interference of mMTC pairs cannot exceed themaximum interference allowed by subcarrier i, i.e., Iicu−max.(ii) All of the accessed mMTC pairs should satisfy their own QoS.98Sensors 2019, 19, 4530When both conditions are satisﬁed, the mMTC pair is allowed to access the subcarrier. In orderto facilitate the calculation, we convert the rate constraint of the eMBB device in (11) into aninterference limit:Iicu ≤ Iicu−max, (14)whereIicu = ∑j∈MIj,B ∀i ∈ N . (15)Substituting (14) for (11), then the optimization problem is converted intomaxαijN∑i=1M∑j=1αij , (16a)s.t Iicu ≤ Iicu−max , (16b)Rj ≥ Rj−min , (16c)αij ∈ {0, 1}, ∀i ∈ N j ∈ M , (16d)N∑i=1αij ≤ 1 ∀j ∈ M. (16e)The optimization variables of this problem are all binary variables, and the optimizationproblem (16a) is an NP-hard problem that cannot be solved by the convex optimization method.The traditional way to solve an NP-hard problem is to carry out an exhaustive search, which involvesunaccepted computational complexity. Thus, it is difﬁcult to obtain the optimal result by direct solution.This paper proposes a heuristic algorithm with lower complexity to tackle the problem, referred to asinterference-aware subcarrier allocation for mMTC.Firstly, the mMTC pair with minimal interference to the base station is selected in thetwo-dimensional matrix Ω, and the access conditions (16b) and (16c) are updated according to the QoSof the eMBB devices and other mMTC pairs. Once an mMTC pair occupies a subcarrier, the mMTCpair is prohibited from accessing other subcarriers. When the sum of the interference ratio accumulatedon subcarrier i is not less than 1, other mMTC pairs will be no longer allowed to access the subcarrier i.For the constraint condition (16c), we can estimate whether the QoS requirements are still satisﬁedafter each mMTC pair occupies the subcarrier according to (7) and (9). Due to the interference betweenmMTC pairs accessing the same channel, the interference range of the mMTC pair is deﬁned in thispaper to suppress interference between mMTC pairs. The interference range of an mMTC pair MUj isdeﬁned as the range in which the mMTC pairs suffer from the interference of MUj. Thus, the mMTCpairs in the interference range of MUj cannot access the same subcarrier with MUj to avoid interference.In other words, when mMTC pair MUj accesses subcarrier i, the other pairs within the interferencerange of MUj cannot access subcarrier i to reduce the interference between the mMTC pairs and toensure the QoS of each mMTC pair.The IASA algorithm is summarized in Algorithm 1. Lines 1–5 of the algorithm calculate themaximum interference that all subcarriers can support (line 3) and the proportion of the interferencefrom each mMTC pair (line 5). Then a two-dimensional matrix is formed. Lines 6–12 of the algorithmsort the data in the two-dimensional matrix in ascending order one-dimensionally. Firstly, we ﬁndthe mMTC pair and subcarrier corresponding to the minimum interference ratio. Then we estimatewhether the accumulated interference caused by the mMTC pair exceeds the maximum interferenceallowed by the subcarrier so as to determine whether the mMTC pair can access the sub-carrier (line 7).After the mMTC pair MUj accesses subcarrier i, the mMTC pair within the interference range of MUjis prohibited from accessing subcarrier i (line 9).99Sensors 2019, 19, 4530Algorithm 1 Interference-Aware Spectrum Allocation for mMTC Communication Algorithm.1: Initialize: B, Pci , Pdj , Ricu−min, Rj, αij = 0 ∀i ∈ N , j ∈ M.2: Calculate Iicu−max, i ∈ N in accordance with (11).3: Calculate the interference Ij,B caused by all mMTC pairs accessing subcarriers according to (10c).4: Calculate the proportion of all mMTC pairs to the maximum interference that each subcarrier canwithstand Iicu−max and obtain the interference matrix Ω.5: Select the smallest element (i∗, j∗) in matrix Ω. For CU∗i , MU∗j , judge whether (9) and (14) isestablished.6: If Iicu ≤ Iicu−max, the mMTC pair is allowed to access subcarriers, αij = 1.If Iicu > Iicu−max, the mMTC pair is not allowed to access the subcarrier, αij = 0.7: αi′ j = 0, ∀i′ ∈ N , i′ = i.8: Filter out the mMTC pair set J within the pair MUj interference range. Assign all the αij j ∈ Jcorresponding to subcarrier i to 0.9: Assign the ratio Ω(:, j) of the mMTC pair to the corresponding subcarrier in this cycle to s(s ≥ 1).10: Repeat 6–9 until all of the mMTC pairs have been assigned.11: Output matrix A.The IASA algorithm is a centralized algorithm that can be implemented by a centralized controller.First, the centralized controller collects the information from mMTC devices who want to transmitpackets with their neighborhood devices, including the channel state information and the requiredtransmission rate. Then the centralized controller obtains the spectrum allocation results accordingto the IASA algorithm and broadcasts the allocation results to each mMTC pair. Then each mMTCtransmitter completes the packets transmission on its allocated subcarrier.5. Simulation Result AnalysisIn this section, we present numerical results to verify the performance of the proposed IASAalgorithm. We compare the number of mMTC pairs accessing the network successfully according tothe proposed subcarrier allocation algorithm, the random access algorithm, and the sequential accessalgorithm. The two benchmark algorithms are described as follows.Random access algorithm (RAA): Firstly, an interference ratio is randomly selected in thetwo-dimensional interference matrix. Then, we ﬁnd out the corresponding mMTC devices and theaccess to subcarrier i. According to (9) and (14), it can be judged whether the QoS of the eMBB devicesand the mMTC devices is satisﬁed, that is, whether the mMTC devices can access the subcarrier i.In the end, we repeat the above selection and access process until all subcarriers achieve the limit ofinterference they can sustain.Sequential access algorithm (SAA): All of the mMTC devices sequentially judge whether the QoSof the eMBB devices and the mMTC devices are satisﬁed. When the sum of the interference ratio ofsubcarrier i is more than 1, other mMTC devices are prohibited from accessing the subcarrier i.Greedy algorithm (GA): In the greedy algorithm, each subcarrier gives priority to its own accessnumber. Speciﬁcally, the mMTC pair with minimum interference to a certain subcarrier is ﬁrstlyaccessed, when the QoS of the eMBB device and mMTC pair can be satisﬁed. When the cumulativeinterference of subcarrier i exceeds 1, other mMTC pairs are forbidden from accessing the subcarrier.It is assumed that the eMBB and mMTC devices are evenly distributed in a circular region wherethe radius is 200 m. All of the devices are served by the same base station that controls the allocationof subcarriers. This paper considers a ﬂat Rayleigh fading channel. The distance-dependent path lossPL(D) is [12]PL(D) = 120.9+ 37.6 log(D/1000) + L + AG, (17)100Sensors 2019, 19, 4530where D is the communication distance and AG is the antenna gain. The value of AG is 0.4 dB. L is theindoor penetration loss. we assume that 80% of mMTC equipment is indoor equipment, where L takes20 dB; 20% is outdoor mMTC equipment, where L takes 0 dB.Figure 2 is a location distribution diagram of eMBB devices and mMTC devices. We consider anarea with a radius of 200 m, in which the eMBB devices and mMTC devices are evenly and randomlydistributed in the area, and the base station is set at the origin.Figure 3 shows the number of mMTC pairs that successfully access subcarriers for differenteMBB devices with Pci = 10 dB, Pdj = 7 dB, Ricu−min = 10 bps, Rj = 5 bps. The total number ofmMTC pairs is 150, and the interference range of the mMTC pair is 80 m. Among the four algorithms,the number of mMTC pairs accessed increases as the number of subcarriers increases. The reasonis that when the number of subcarriers increases, the mMTC pairs will be more likely to accessthe subcarriers. Some mMTC pairs with large interference also have the opportunity to access thesubcarriers. Compared to the three contrastive algorithms, the IASA algorithm can realize more mMTCdevices accessed in the system. The reason is that under the premise of guaranteeing the QoS of theeMBB devices and the mMTC pairs, the mMTC pair with the least interference to the subcarriers isselected ﬁrst according to the IASA algorithm. The distance limitation is established to reduce theinterference between the adjacent mMTC pairs. Meanwhile, the complexity of the algorithm is reduced.For GA, each subcarrier gives priority to the access number optimization of itself rather than the accessperformance of the whole system, so the access number of the system cannot be maximized. However,since GA considers the mMTC priority access with less interference in each subcarrier, its performanceis better than RAA and SAA. For RAA and SAA, there may be an mMTC pair with large interferenceaccessing the subcarrier at any time, which occupies a large proportion of the interference space thatthe subcarrier can sustain. Under these circumstances, some mMTC pairs with small interferencecannot access the subcarrier because the space for interference is ﬁnite. Therefore, the IASA algorithmenables the system to accommodate more mMTC pairs.-200 -150 -100 -50 0 50 100 150 200-200-150-100-50050100150200Base StationeMBB DevicemMTC NodeFigure 2. Distribution of eMBB devices and mMTC devices in cellular systems.101Sensors 2019, 19, 45305 6 7 8 9 10 11 12 13 14 15number of CU0102030405060708090number of D2D accessedGreedy Algorithm(GA)Interference-Aware Subcarrier Allocation Algorithm(IASA)Sequential Access Algorithm(SAA)Random Access Algorithm(RAA)Figure 3. Number of mMTC pairs accessed versus different eMBB devices using different algorithms.Figure 4 depicts the number of mMTC pairs that access subcarriers for different eMBB deviceswith Pci = 10 dB, Pdj = 7 dB, Rj = 5 bps. The interference range of the mMTC pairs is 80 m. The numberof mMTC pairs that access the subcarriers decreases gradually when the minimum transmission rateof the eMBB devices gradually increases. The reason is that when the minimum transmission rate ofthe eMBB devices increases, the maximum interference that each subcarrier can sustain is reduced.In the case where the interference caused by the mMTC pair is unchanged, the number of mMTCpairs that can access the subcarriers is reduced. In the low-rate phase, the RAA exhibits much loweraccess performance than the IASA algorithm. And the SAA exhibits a comparable access performancewith IASA algorithm. However, as the rate of eMBB devices increase, the performance of SAA havedropped signiﬁcantly compared to the IASA algorithm.??????????????????????? ? ? ? ? ? ? ? ??????????????????????????????????????????????????????Figure 4. Number of mMTC pairs accessed versus different rate requirements of eMBB devices.Figure 5 presents the number of mMTC pairs accessed when the power of the mMTC pairstransmitter change with Pci = 10 dB, Rcu−min = 10 bps, Rj = 5 bps. The interference range of themMTC pairs is 80 m. The number of accessed mMTC devices gradually decreases as the power of themMTC transmitter increases. This is due to the fact that the interference to subcarriers increases while102Sensors 2019, 19, 4530the power of the mMTC transmitting increases. Therefore, the number of mMTC devices accessed isrelatively reduced. Compared to the variables such as the number of eMBB devices and the minimumtransmission rate of eMBB devices, the power of the mMTC transmitter has a relatively small impacton the number of mMTC pairs accessed in the system.??????????????????????? ?? ?? ??????????????????????????????????????????????????????Figure 5. Number of mMTC pairs accessed versus different mMTC device transmit powers.Figure 6 investigates the number of mMTC pairs accessed when there are different numbers ofmMTC pairs in the system. The parameter settings are the same as in Figure 3. The number of mMTCdevices accessed increases as the number of mMTC pairs in the system increases. Before the number ofsubcarriers is saturated, the more mMTC devices in the system, the greater the opportunity to accessdevices that satisfy the QoS requirements of the eMBB devices. Thus, the total number of mMTC pairsaccessed will increase. However, when the power and the minimum transmission rate of the eMBBdevices are ﬁxed, the number of mMTC devices accessed by all subcarriers is constant, so the numberof mMTC pairs accessed will gradually become saturated. Among the three algorithms shown inFigure 6, the RAA achieves the access saturation state ﬁrst.?????????????????????????????????? ??? ??? ??? ??? ?????????????????????????????????????????????????????Figure 6. Number of mMTC pairs accessed versus different number of mMTC devices.6. ConclusionsIn this paper, an interference-aware subcarrier allocation algorithm for mMTC is proposed forthe subcarrier allocation problem of D2D communication in mMTC scenarios. Initially, we establisha model maximizing the number of mMTC pairs accessed. When carrying out subcarrier allocation,we calculate the maximum interference that each subcarrier can sustain. Then we determine the mMTCpair with the least interference to the subcarrier and estimate the access property according to whether103Sensors 2019, 19, 4530the QoS of the mMTC pair is satisﬁed. When the accumulated interference caused by the mMTC pairis greater than the maximum interference limit that the subcarrier can sustain, the subcarrier will benot be accessible to other mMTC pairs anymore. Simulation results demonstrated the effectiveness ofthe proposed algorithm.Author Contributions: W.H. designed the algorithm, performed the theoretical analysis, and wrote themanuscript. S.L. and Y.S. implemented the simulation and contributed to themanuscript preparation. J.Z. analyzedthe data. X.Y. and N.L. contributed to polishing the revised manuscript and provided suggestions on simulationevaluation.Acknowledgments: This work was supported by the Science and Technology Project of Xuzhou (KC18105KC18068), the National Natural Science Foundation of China (61771417 51734009 51804304), the National KeyResearch and Development Program (2016YFC0801403), and the Fundamental Research and DevelopmentFoundation of Jiangsu Province.Conﬂicts of Interest: The authors declare no potential conﬂicts of interest with respect to the research, authorship,and/or publication of this article.References1. Meng, Z.; Wu, Z.; Gray, J. A Collaboration-Oriented M2M Messaging Mechanism for the CollaborativeAutomation between Machines in Future Industrial Networks. Sensors 2017, 17, 2694. [CrossRef] [PubMed]2. Park, J.H. Advances in Future Internet and the Industrial Internet of Things. Symmetry 2019, 11, 244.[CrossRef]3. Al-Sakran, A.; Qutqut, M.H.; Almasalha, F.; Hassanein, H.S.; Hijjawi, M. An Overview of the Internetof Things Closed Source Operating Systems. In Proceedings of the 2018 14th International WirelessCommunications Mobile Computing Conference (IWCMC), Limassol, Cyprus, 25–29 June 2018; pp. 291–297.[CrossRef]4. Laya, A.; Alonso, L.; Alonso-Zarate, J. Is the random access channel of LTE and LTE-A suitable for M2Mcommunications? A survey of alternatives. IEEE Commun. Surv. Tutor. 2013, 16, 4–16. [CrossRef]5. Kawabata, H.; Ishibashi, K.; Vuppala, S.; de Abreu, G.T. Robust relay selection for large-scaleenergy-harvesting IoT networks. IEEE Internet Things J. 2016, 4, 384–392. [CrossRef]6. Hu, X.; Sun, J. Interference Analysis and Resource Allocation of Burst Scenario in Massive Machine-TypeCommunications. In Proceedings of the 2018 IEEE 18th International Conference on CommunicationTechnology (ICCT), Chongqing, China, 8–11 October 2018; pp. 822–826. [CrossRef]7. Lien, S.; Chen, K. Massive Access Management for QoS Guarantees in 3GPP Machine-to-MachineCommunications. IEEE Commun. Lett. 2011, 15, 311–313. [CrossRef]8. Tefek, U.; Lim, T.J. Relaying and Radio Resource Partitioning for Machine-Type Communications in CellularNetworks. IEEE Trans. Wirel. Commun. 2017, 16, 1344–1356. [CrossRef]9. Kwon, T.; Choi, J. Multi-Group Random Access Resource Allocation for M2M Devices in Multicell Systems.IEEE Commun. Lett. 2012, 16, 834–837. [CrossRef]10. Oh, C.; Hwang, D.; Lee, T. Joint Access Control and Resource Allocation for Concurrent and Massive Accessof M2M Devices. IEEE Trans. Wirel. Commun. 2015, 14, 4182–4192. [CrossRef]11. Mostafa, A.E.; Gadallah, Y. Uniqueness-Based Resource Allocation for M2M Communications inNarrowband IoT Networks. In Proceedings of the 2017 IEEE 86th Vehicular Technology Conference(VTC-Fall), Toronto, ON, Canada, 24–27 September 2017; pp. 1–5. [CrossRef]12. Mostafa, A.E.; Zhou, Y.; Wong, V.W. Connectivity maximization for narrowband IoT systems withNOMA. In Proceedings of the 2017 IEEE International Conference on Communications (ICC), Paris, France,21–25 May 2017; pp. 1–6.13. Han, S.; Xu, X.; Zhao, L.; Tao, X. Joint time and power allocation for uplink cooperative non-orthogonalmultiple access based massive machine-type communication Network. Int. J. Distrib. Sens. Netw. 2018,14, 1550147718778215. [CrossRef]14. Li, J.; Zhang, X.; Feng, Y.; Li, K.C. A Resource Allocation Mechanism Based on Weighted EfﬁciencyInterference-Aware for D2D Underlaid Communication. Sensors 2019, 19, 3194. [CrossRef] [PubMed]104Sensors 2019, 19, 453015. Li, Z.; Gui, J.; Xiong, N.; Zeng, Z. Energy-Efﬁcient Resource Sharing Scheme With Out-Band D2DRelay-Aided Communications in C-RAN-Based Underlay Cellular Networks. IEEE Access 2019, 7,19125–19142. [CrossRef]16. Ningombam, D.D.; Lee, C.G.; Shin, S. Interference Mitigation for Multicast D2D Communications UnderlayCellular Networks. In Proceedings of the 2019 International Conference on Artiﬁcial Intelligence inInformation and Communication (ICAIIC), Okinawa, Japan, 11–13 February 2019; pp. 1–4. [CrossRef]17. Li, X.; Qin, N.; Sun, T. Interference coordination for FD-MIMO cellular network with D2D communicationsunderlaying. China Commun. 2018, 15, 75–88.18. Asadi, A.; Wang, Q.; Mancuso, V. A survey on device-to-device communication in cellular networks.IEEE Commun. Surv. Tutor. 2014, 16, 1801–1819. [CrossRef]19. Kai, Y.; Wang, J.; Zhu, H.; Wang, J. Resource Allocation and Performance Analysis of Cellular-AssistedOFDMA Device-to-Device Communications. IEEE Trans. Wirel. Commun. 2019, 18, 416–431. [CrossRef]20. Lu, B.; Lin, S.; Shi, J.; Wang, Y. Resource Allocation for D2D Communications Underlaying Cellular NetworksOver Nakagami- m Fading Channel. IEEE Access 2019, 7, 21816–21825. [CrossRef]21. Chour, H.; Jorswieck, E.A.; Bader, F.; Nasser, Y.; Bazzi, O. Global Optimal Resource Allocation for EfﬁcientFD-D2D Enabled Cellular Network. IEEE Access 2019, 7, 59690–59707. [CrossRef]22. Huang, Y.F.; Tan, T.H.; Liu, S.P.; Liu, T.Y.; Chen, C.M. Performance of subcarrier allocation of D2Dmulticasting for wireless communication systems. In Proceedings of the 2018 Tenth International Conferenceon Advanced Computational Intelligence (ICACI), Xiamen, China, 29–31 March 2018; pp. 193–196.23. Cao, W.; Feng, G.; Qin, S.; Liang, Z. D2D Communication Assisted Trafﬁc Ofﬂoading for MassiveConnections in HetNets. In Proceedings of the 2016 IEEE Global Communications Conference (GLOBECOM),Washington, DC, USA, 4–8 December 2016; pp. 1–6. [CrossRef]24. Atat, R.; Liu, L.; Mastronarde, N.; Yi, Y. Energy Harvesting-Based D2D-Assisted Machine-TypeCommunications. IEEE Trans. Commun. 2017, 65, 1289–1302. [CrossRef]25. Bagaa, M.; Ksentini, A.; Taleb, T.; Jantti, R.; Chelli, A.; Balasingham, I. An efﬁcient D2D-based strategiesfor machine type communications in 5G mobile systems. In Proceedings of the 2016 IEEE WirelessCommunications and Networking Conference, Doha, Qatar, 3–6 April 2016; pp. 1–6. [CrossRef]26. Shi, J.; Yang, L.L. Novel subcarrier-allocation schemes for downlink MC DS-CDMA systems. IEEE Trans.Wirel. Commun. 2014, 13, 5716–5728. [CrossRef]27. Xu, S.; Chang, T.; Lin, S.; Shen, C.; Zhu, G. Energy-Efﬁcient Packet Scheduling With Finite Blocklength Codes:Convexity Analysis and Efﬁcient Algorithms. IEEE Trans. Wirel. Commun. 2016, 15, 5527–5540. [CrossRef]c© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open accessarticle distributed under the terms and conditions of the Creative Commons Attribution(CC BY) license (http://creativecommons.org/licenses/by/4.0/).105sensorsArticleK-Means Spreading Factor Allocation for Large-ScaleLoRa NetworksMuhammad Asad Ullah 1, Junnaid Iqbal 1, Arliones Hoeller 1,2,3, Richard Demo Souza 2 andHirley Alves 1,*1 Centre for Wireless Communications, University of Oulu, 90014 Oulu, Finland;muhammad.asadullah@oulu.ﬁ (M.A.U.); junnaid.iqbal@oulu.ﬁ (J.I.)2 Department of Electrical and Electronics Engineering, Federal University of Santa Catarina,Florianópolis 88040-900, Brazil; richard.demo@ufsc.br3 Department of Telecommunications Engineering, Federal Institute for Education, Science,and Technology of Santa Catarina, São José 88103-310, Brazil; arliones.hoeller@ifsc.edu.br* Correspondence: Hirley.Alves@oulu.ﬁReceived: 30 September 2019; Accepted: 28 October 2019; Published: 30 October 2019Abstract: Low-power wide-area networks (LPWANs) are emerging rapidly as a fundamental Internetof Things (IoT) technology because of their low-power consumption, long-range connectivity, andability to support massive numbers of users. With its high growth rate, Long-Range (LoRa) isbecoming the most adopted LPWAN technology. This research work contributes to the problemof LoRa spreading factor (SF) allocation by proposing an algorithm on the basis of K-meansclustering. We assess the network performance considering the outage probabilities of a large-scaleunconﬁrmed-mode class-A LoRa Wide Area Network (LoRaWAN) model, without retransmissions.The proposed algorithm allows for different user distribution over SFs, thus rendering SF allocationﬂexible. Such distribution translates into network parameters that are application dependent.Simulation results consider different network scenarios and realistic parameters to illustrate howthe distance from the gateway and the number of nodes in each SF affects transmission reliability.Theoretical and simulation results show that our SF allocation approach improves the network’saverage coverage probability up to 5 percentage points when compared to the baseline model.Moreover, our results show a fairer network operation where the performance difference betweenthe best- and worst-case nodes is signiﬁcantly reduced. This happens because our method seeks toequalize the usage of each SF. We show that the worst-case performance in one deployment scenariocan be enhanced by 1.53 times.Keywords: stochastic geometry; resource allocation; Internet of Things1. IntroductionThe Internet of Things (IoT) is the integration of modern electronic devices, smart sensors, internetprotocols, and wireless communications technologies. IoT applications are rapidly gaining popularityin many domains such as industrial operations, smart parking, augmented maps, healthcare, smartcars, and smart homes [1–5]. According to a Gartner Inc. report, there will be around 26 billion IoTdevices deployed worldwide by 2020 [6]. In the Statista report, it is predicted that there will be over75 billion IoT devices worldwide by 2025 [7].In the modern era, the spectacular growth and transformation of wireless connectivity are drivenby the IoT paradigm, with technologies having attributes of large-scale network infrastructure withlow-cost sensors connected to the Internet. In this context, low-power wide-area networks (LPWANs)are quite popular in terms of prototypes, standards, and on the commercial level because of theirSensors 2019, 19, 4723; doi:10.3390/s19214723 www.mdpi.com/journal/sensors106Sensors 2019, 19, 4723signiﬁcance with respect to power efﬁciency along with long range [8,9]. Within this context, LoRA,SigFox, NB-IoT, Weightless, RPMA and DASH7 [10,11] are the most distinguished technologies.This paper focuses on LoRa, which provides good performance in terms of reliability and energyconsumption. The network architecture contains end-devices, gateways, and a network server (NS),forming a star topology. It operates at unlicensed frequency ISM (Industrial, Scientiﬁc, Medical) bandsof 863–870 MHz and 915 MHz in Europe and the U.S., respectively [12,13]. In Europe, the duty cyclelimitations range from 0.1% to 10%, following European Telecommunications Standards Institute(ETSI) standards. In addition, LoRa works on variable and adaptive data rates by using differentspreading factors. This is achieved by the NS controlling the spreading factors (SFs) and bandwidth(BW) of the end-devices. Higher SFs allow larger coverage areas; however, as a drawback, they reducethe data rate and increase the time-on-air (ToA) of LoRa packets [14].Notably, the gateway has the ability to receive data from multiple nodes at the same time becauseof the orthogonality of sub-bands and the quasi-orthogonality of different SFs. The LoRa MAC layer,known as LoRaWAN [15], is a type of ALOHA protocol controlled by the NS. LoRaWAN deﬁnes threeclasses of devices depending upon the application. Class A devices may wait for acknowledgments(ACK) only in their receiving windows during downlink transmission and consume the least power.Class B devices are able to open extra receiving windows at scheduled times, thus reducing downlinklatency. Class C nodes consume the most energy because they leave the receiver enabled all the time,allowing for the lowest latency time [16].For instance, extensive measurement campaigns show that the communication range of LoRareaches up to 30 km over the water and more than 15 km on the ground [11]. LoRa is suitable for a widerange of telemetry applications (e.g., sensing and monitoring), which can be used in several industryverticals, such as smart grids and cities, and smart agriculture up to industrial IoT applications [17,18].During the past few years, many studies have contributed by proposing new algorithms, systemsmodels, analyses, and by designing new approaches for performance enhancement of LoRa networks.However, only a few considered resource allocation.The major contribution of this work is the modeling of an approach for SF allocation for a largescale LoRa network based on K-means clustering and the analysis of connection, capture, and coverageprobabilities. Instead of using constant steps of distance from the gateway to deﬁne SF areas [19,20],the proposed algorithm assigns a maximum range of individual SF regions, which allows for distinctuser distribution. Then, we evaluate the performance of the proposed algorithm over the uplink of alarge-scale LoRa network with a single gateway based on the model introduced in [19].The remainder of this article is structured as follows. Section 2 discusses related work and a shortoverview of LPWAN technologies. Section 3 introduces the system model, and Section 3.1 details theoutage probabilities of the baseline model, used to examine the performance of proposed SF allocationapproach. The proposed algorithm is presented in Section 4. Simulation results are discussed inSection 5. Finally, Section 6 concludes the paper and proposes future work.2. Related WorkOverviews of LoRa and LPWAN technologies are provided in [21,22]. Usually, LoRa operates witha bandwidth of 125 kHz, but it also allows for bandwidths of 250 kHz and 500 kHz. The wider bandspromote resistance to fading, channel noise, Doppler effects, and long-term relative frequency [23].Chirp spread spectrum (CSS) modulation, which enables high receiver sensitivity, makes LoRa morerobust against the interference when compared to Sigfox, which employs ultra-narrowband (UNB)communication [24]. As a tradeoff, the use of wider bands for the transmission of narrowband signalsmakes less efﬁcient use of the spectrum. A realistic SigFox communication model is implementedand tested in [25]; it evaluates the performance of a high-density large-scale wireless sensor network(WSN). From the obtained results, one can observe that the performance of the SigFox networksigniﬁcantly degrades by increasing the number of sensors, and some solutions are presented toimprove the performance.107Sensors 2019, 19, 4723Unlike Sigfox, LoRa can be deployed locally, i.e., without the need for a cellular infrastructure,and has higher bit rates. By contrast, NB-IoT is an expensive technology having the pros of low latencyand high quality of service (QoS) [26]. In [27], the authors compare different LPWAN technologies(Bluetooth, ZigBee, SigFox, and LoRa) and discuss LoRa with respect to code rate (CR), bandwidth(BW), and SF but without considering the inﬂuence of Rayleigh fading and path loss attenuation.Theoretical and simulation results show that SF, BW, and CR inﬂuence the ToA of a packet. Larger SFsand CRs result in higher ToA of LoRa packets. Conversely, ToA reduces with larger bandwidths.The work in [28] proposes two different algorithms named EXPLoRa-SF and EXPLoRa-AT andshows in simulation results that these algorithms perform considerably better than the LoRaWANadaptive rate strategy (ADR). EXPLoRa-AT delivers higher bit rates in the event of higher trafﬁc loads,while EXPLoRa-SF allocates SFs at the different subgroups of end-devices depending on the receivedsignal strength indicator (RSSI). The results demonstrate that the data extraction rate (DER) dropsdramatically for higher SFs and larger numbers of end-devices. The authors, however, assume a shortrange and dense network in their analysis.The EXPLoRa approach is further extended in [29], K-means is applied to identify the non-circularcrowded region, and all the nodes inside that area are assumed to have same SF. On the other hand,in the proposed work the geometry of network is circular, with six annuluses representing the rangeof individual SFs. We have analyzed the scalability and the performance of the uplink LoRa modelconsidering Rayleigh fading, connection H1, capture Q1, and coverage probabilities H1Q1 in thepresence of interfering signals using the same SF. The considerations of H1 and H1Q1 are missingin [28,29]. Moreover, in our model, we consider a dense and wide network (radius of several kilometers)and analyze the performance by considering the maximum distance of individual SF boundaries fromthe gateway.Another scientiﬁc study used K-means for the classiﬁcation of end-devices into three groupsbased on trafﬁc characteristics with different priorities. The grouping of end-devices was computed interms of priority-based transmission instead of SF allocation [30].In [31,32], SF distribution is mainly based on the power level of the signals that the gatewayreceives from the end-devices and gateway sensitivity, without considering the location of end-devices.As a drawback, SF allocation was disturbed because of high-density buildings, and 53.2% of theend-devices were forced to use SF12. Furthermore, in [28–31], only network-level simulators suchas ns-3 and LoRaSim are used, which abstracts some characteristics of the physical layer that areincorporated in our analysis. Conversely, our study evaluated the performance of the proposed SFallocation algorithm considering the analytical model, realistic parameters, and averaging over 105random deployment of the Poisson point process (PPP) by Monte Carlo computer simulations, whichmatch with the theoretical results.The tree-based spreading factor clustering algorithm (TSCA) for SF allocation in multihop LoRAnetworks is introduced in [33]. This approach ofﬂoads the data trafﬁc in many sub-networks, whichare linked to a sink node assigning a speciﬁc SF according to network clustering, thus enabling parallelframe transmission with multiple SFs. The authors show that TSCA increases the network performancein a network with rectangular geometry.A single gateway uplink model considering path loss attenuation and Rayleigh fading is designedin [19], utilizing stochastic geometry to model network interference and then disconnection andcollision probabilities. Such a model is further extended in [20], in which the authors propose ascheme that considers message replication and gateways with multiple receive antennas/decodersto attain time and spatial diversity. They demonstrate that the number of users and trafﬁc densitydirectly affects the performance of the LoRa network and that sending multiple message copies isbeneﬁcial for low-density networks. Both of these studies adopt equal radius SF allocation approaches.Unlike [19,20], our work considers K-means-based fair SF allocation of nodes in LoRa networks.Recently, several studies have addressed the problems associated with automatic repeat request(ARQ) and contributed to downlink reliability in LoRaWAN applications. The sequential transmission108Sensors 2019, 19, 4723of downlink frames, saturation of duty cycle, and half-duplex nature of LoRa gateway radios aremarked as the major shortcomings for the downlink transmission [34,35]. Furthermore, these worksalso highlight the signiﬁcance of gateway selection algorithm to prevent trafﬁc losses due to sequentialtransmission of downlink frames and duty cycle limitations.One experimental study evaluates the performance of a LoRa network at a 125 kHz bandwidthand SF7 for a sailing monitoring model, and the measurements show a 60.49% packet loss at themaximum distance of 3284 m [36]. Another LoRaWAN-based indoor environment monitoring systemcomposed of 331 sensor nodes is deployed at the University of Oulu, where the gateway is installed ata distance of ∼180 m and 24 m above the ground [37]. The measurements performed at SF7 show amaximum 11.33% packet error rate (PER), which can be due to co-spreading factor interference becauseall 331 end-devices use the same SF. As illustrated in [19], nodes using the same SF face co-spreadingfactor interference. The motivation behind our work is to propose a suitable SF allocation algorithmfor a large-scale LoRa network to efﬁciently utilize the different data rates. To enhance SF allocation,we propose a novel algorithm, based on the machine learning technique called K-means clustering,for effectively allocating the SFs.3. System ModelWe consider N¯ uniformly distributed smart devices inside an uplink class-A LoRaWAN networkwithout retransmissions, utilizing a single channel within radio range of R km and a circular area ofV = πR2 around a single gateway. Figure 1 illustrates a deployment with N¯ = 500 and R = 3 km.The gateway is at the origin, and nodes are distributed uniformly in V = 28.26 km2. Note that suchmodel captures the characteristics of telemetry applications such as those in smart cities and smartbuildings. For instance, the University of Oulu Smart Campus has a LoRaWAN network constantlymonitoring several sensors such as temperature, luminosity, and CO2 [37].-3 -2 -1 0 1 2 3-3-2-10123NodesGatewayFigure 1. Uniform distribution of N¯ = 500 nodes in a circular network area of radius R = 3 km,with the gateway (GW) at the origin.109Sensors 2019, 19, 4723The LoRa modulation bit rate is deﬁned as [14]Rb =44+ CRBW2SF, (1)where 44+CR is the effective coding rate, ranging from45 to48 , while CR denotes the LoRa coding rateconﬁguration, varying from 1 to 4. In our work, we assume CR = 1, and the LoRa uplink channelaggregated bit rate is expressed as bitrateU = ∑12i=7 Rbi = 12.17 kbps. For instance, Table 1 shows thecharacteristics of 9 byte LoRa packets with explicit header and CRC modes enabled and BW = 125 kHz.Table 1. Characteristics of the LoRa uplink model containing packets of 9 bytes at BW = 125 kHz.SF Bit Rate Receiver Sensitivity SNR Range(i) kbps (Rbi) dBm dB (qSF ) km7 5.47 −123 −6 l0 − l18 3.13 −126 −9 l1 − l29 1.76 −129 −12 l2 − l310 0.98 −132 −15 l3 − l411 0.54 −134.5 −17.5 l4 − l512 0.29 −137 −20 >l53.1. Uplink Outage ProbabilityThe uplink transmission of nodes is based on the ALOHA protocol, and the probability of collisionin ALOHA networks is high when many stations are connected [38]. In LoRa, simultaneous signalsof different SFs are quasi-orthogonal because the inter-SF rejection gain varies from 16 to 36 dB [39].Therefore, for the sake of simplicity, our work does not inspect inter-SF interference and focuses onco-SF interference only, which is stronger.In this paper, the uplink model includes the inﬂuence of Rayleigh fading and path loss attenuationas the baseline model [19] for performance analysis, where g(dk)= λ(4πdk)η is the path loss attenuationfunction, η ≥ 2 is the path loss exponent, λ is the wavelength, and hk is the fading in the link betweenthe k-th node and the gateway. Let us consider the transmitted signal of a single LoRa node s1(t)toexamine the impact of co-SF interference originated due to simultaneous transmission of nodes withsame SF. The mathematical expression of the received signal at the gateway can be expressed asr1(t)= g(d1)h1 ∗ s1(t)+N∑k=2χSFk(t)g(dk)hk ∗ sk(t)+ n(t), (2)where n(t)is additive white Gaussian noise with zero mean and variance N = −174 + NF +10log10(BW) dBm, NF is the noise ﬁgure of the receiver, and −174 dBm/Hz is the thermal noisespectral density constant.We consider that an outage of the received signal in an uplink channel can take place in the twoscenarios [19]. First, if the signal-to-noise ratio (SNR) of the received packet is less than the SF speciﬁcthreshold qSF, then the node is considered disconnected. Second, if the signal-to-interference ratio (SIR)between the target-received packet and any other concurrent signals of the same SF and frequencychannel is less than 6 dB, then it is considered as a collision.3.1.1. Outage Condition IThe distance of the end-device to the gateway in a wireless transmission domain is crucial.The instantaneous SNR can be expressed as SNR = P1|h1|2g(di)N , where P1 is the transmit power ofend-device 1 in mW and |h1|2 is the squared envelop of the channel coefﬁcient. Communication is110Sensors 2019, 19, 4723only possible when the SNR of the received signal at the gateway is less than the reception thresholdqSF. Thus, the ﬁrst outage condition, the connection probability, is deﬁned as [19]H1 = exp( N qSFP1g(d1)), (3)where d1 (in meters) is the distance of the desired end-device from the gateway.3.1.2. Outage Condition IIA collision in LoRa end-device transmission takes place if the SIR of the desired signal withrespect to interference from the same SF and frequency channel is less than 6 dB, i.e., if the desiredsignal is at least four times stronger than the interference. We model this outage condition basedon [19], where interference is approached by considering the strongest interfering device. Accordingto [19], the highest interference comes from the end-device k∗.The probability that no collision occurs or that the strongest interfering signal is at least 6 dBbelow the desired one, termed the capture probability, isQ1 = P[ |h1|2g(d1)|hk∗ |2g(dk∗)≥ 4∣∣∣∣ d1]= E|h1|2[P[Xk∗ <|h1|2g(d1)4∣∣∣∣ |h1|2, d1] ]. (4)The probability above depends on the distribution of Xk∗ = |hk∗ |2g(dk∗). The cumulative distributionfunction (CDF) of Xk∗ is derived in [19] and is denoted as FXk∗ . Thus,Q1 = E|h1|2[FXk∗( |h1|2g(d1)4)]=∫ ∞0e−zFXk∗(zg(d1)4)dz. (5)Moreover, in [19] the authors present an approximation for (5) that is only accurate at the edges of eachannulus. This paper considers only the exact probability in (5).3.1.3. Coverage ProbabilityThe probability that deﬁnes whether a selected end-device is in coverage and can successfullycommunicate with the gateway is termed the coverage probability. It is the product of H1 and Q1.The average coverage probability ℘c can be achieved by deconditioning the location of the individualnode by averaging over the network coverage area V = πR2, i.e., [19]℘c =2R2∫ R0H1(d1)Q1(d1)d1 dd1. (6)The average coverage probability of a individual SF annulus is also inspected. It indicates theprobability of an end-device at distance d1 in the annulus i by considering the connection and captureprobabilities and is deﬁned as [20]℘c,i =2(li+1 − li)2∫ li+1liH1(d1)Q1(d1)(d1 − li) dd1, (7)where li+1 is the radius of the outer circle and li is the radius of the inner circle of the ith annulus.4. Proposed SF Allocation AlgorithmIn this paper, we propose an SF allocation algorithm, i.e., an algorithm to deﬁne the rangeof each SF annulus. Our solution uses the K-means machine learning algorithm [40], used in theprocess of vector quantization in data mining by clustering. It is a non-deterministic, numerical, anditerative approach. The main objective of the K-means algorithm is to ﬁnd the minimum cost function,deﬁned as the distance between each point in the data set and its nearest centroid. The distance111Sensors 2019, 19, 4723between the cluster centers and data elements typically assumes the Euclidean distance. K-meansclustering method can efﬁciently achieve robust clustering results when dealing with large data sets.The K-means algorithm ﬁrst arbitrarily chooses K points from the data set, which indicate the initialcentroids. The remaining points are then clustered to the closest centroid, and the coordinates ofcentroids are recalculated, iteratively, until the cost function converges.Consequently, it is important to choose the appropriate number of centroids during theinitialization procedure because the area of each annulus π(l2i+1 − l2i ) increases towards the higherSFs in a strategy based on equal distance steps per SF, which results in the growth of node densitydue to uniform distribution. That is why it is essential to select the sequences for K-means iterationsthat can provide larger values of K clusters for higher SFs. In order to avoid an extensive number ofnodes in an individual SF, there should be a fair difference between the inner (li) and outer (li+1) radiiof annulus. In the proposed work, the annulus area is directly dependent on the difference betweenthe K clusters for two consecutive iterations.In our approach, we use ﬁve iterations of K-means. We start by computing the boundaries ofthe outermost SF ring, SF12, and then proceed to deﬁne the inner boundaries for lower SFs. For eachiteration, K clusters are selected to develop the centroids of end-devices in the LoRa network coveredby a single gateway. Four different mathematical sequences listed in Table 2—a Fibonacci series, squarenumbers, Wythoff array, and arithmetic series—are used to assign the values of K for each iteration.Table 2. K Cluster Values for K-Means Iterations.Iteration 1st 2nd 3rd 4th 5thSeries SF12 SF11 SF10 SF9 SF8Fibonacci series 34 21 13 8 5Square number 49 36 25 16 9Arithmetic series 34 28 22 16 10Wythoff array 37 32 24 16 11In our work, K-means operates iteratively. Each iteration deﬁnes the set of nodes at the outer SFring. In each iteration, the algorithm seeks the set of K centroids C that minimizes the average of thedistances between any node and its closest centroid, i.e.,C = arg minCk∈C1|EDk| ∑Xi∈EDkdist(Ck, Xi)2, (8)where EDk is the set of devices at the k-th iteration, Xi is a device in EDk, and Ck is the closest centroidof Xi. The function dist(x1, x2) computes the Euclidean distance between x1 and x2. This procedurereturns the collection of K centroids of network nodes, whereas C = {C1, . . . CK}. After computingthe centroids, the algorithm determines the boundary of C, so that [Cx,Cy] = boundary(C), whichdetermines the 2-D vector of border points around the Cartesian coordinates of the centroids. Then,it separates the nodes that are inside of the centroid boundary, forming the set I = [Ix, Iy], whereIx and Iy are vectors storing the coordinates of the inner nodes in each of the Cartesian dimensions.In the next step, the maximum absolute value of each dimension of I is calculated to set the radius asli =max|Ix |+max|Iy |2 , which deﬁnes the limit of the SF ring i. The procedure repeats to determine theboundaries of the remaining SF rings (l5, . . . l1).The steps involved in the proposed SF allocation technique are described in Algorithm 1. It repeatsthe process ﬁve times to allocate nodes for SF12–SF8.At the end, the remaining nodes use SF7. Initially,the SF12 outer limit is set to the network radius. In each iteration, the number of clusters is assignedto K depending on the chosen mathematical series (as mentioned in Table 2). For the ﬁrst iteration,the algorithm considers all of the N¯ nodes inside the set ED. In line 4, it computes the K-means of ED,which returns the centroids C = {C1, . . . CK} by (8). Since nodes in the set I are inside the boundary ofcentroids, line 7 computes the inner limit of SF ring li. This process is repeated iteratively until l1 is112Sensors 2019, 19, 4723calculated for the allocation of SF8 and the remaining nodes are assigned to SF7 (line 11). Note thatthe set of nodes ED is updated at the end of each iteration by removing the nodes that were alreadyallocated to an SF (line 9).Algorithm 1 K-Means-based SF AllocationInput: ED := N¯ uniformly deployed nodesOutput: L := {l0, l1, . . . , l6}1: l6 := R2: for i in {5, . . . , 1} do For each SF ring, starting from the outermost ring3: K := GetKfromSeries(i) Set number of centroids for this iteration4: C := Kmeans(ED, K) Compute the centroids5: B := boundary(C) Compute the boundary of C6: I := {x ∈ ED | x ∈ convB} Select nodes that are inside the boundary B7: li :=max(|Ix |)+max(|Iy |)2 Compute the new SF ring limit8: SFi+7 := {x ∈ ED | x /∈ Ball[(0, 0), li]} Allocate SFi+7 to nodes outside the circle of radius li9: ED := {x ∈ ED | x ∈ Ball[(0, 0), li]} Remove nodes outside the circle of radius li10: l0 := 011: SF7 := ED Allocate SF7 to remaining nodes12: return LAll of the iterations of the proposed algorithm for an example network are demonstrated inFigure 2. The radius of the network circular area is R = 3 km, and therefore, the outer limit of SF12 isl6 = R = 3 km. The ﬁrst iteration of the algorithm deﬁnes the inner boundary of SF12, l5, as shown inFigure 2. After excluding the devices inside the SF12 ring from EDk, the algorithm runs a new iterationand deﬁnes l4, i.e., the inner boundary of the SF11 ring. The iterations continue until l1 is deﬁned andthe complete network geometry is obtained.-3 -2 -1 0 1 2 3-3-2-10123 R l5-3 -2 -1 0 1 2 3-3-2-10123 l5 l4-3 -2 -1 0 1 2 3-3-2-10123 l4 l3-3 -2 -1 0 1 2 3-3-2-10123 l3 l2-3 -2 -1 0 1 2 3-3-2-10123 l2 l1-2 0 2-3-2-10123 l1 l0Figure 2. K-means iterations for SF allocation based on Fibonacci series and 500 nodes, where li andli+1 are the inner and outer radii of ith annulus, respectively.113Sensors 2019, 19, 4723Figure 3 depicts the clusters of nodes, centroids, and gateway at the origin for the last K-meansiteration. The nodes outside l1 are allocated to SF8, and the remaining nodes are assigned to SF7.The SF distribution of N¯ = 500 nodes based on the proposed approach considering the Fibonacci seriesfor clustering is shown in Figure 4. The number of nodes in SF rings depends on the chosen seriesbecause of the distinct number of clusters for each sequence.-1500 -1000 -500 0 500 1000 1500-1500-1000-500050010001500GatewayCluster 1Cluster 2Cluster 3Cluster 4Cluster 5Centroids l1Figure 3. 5th iteration with Fibonacci series. Nodes outside l1 use SF8, those inside use SF7.-3 -2 -1 0 1 2 3-3-2-10123SF12SF11SF10SF9SF8SF7GatewayFigure 4. SF allocation of 500 nodes with Fibonacci series.114Sensors 2019, 19, 4723The area of each annulus also varies according to the mathematical series, which also affects thenetwork performance. An important aspect to take into consideration is the selection of the numberof clusters (K); if the difference between the clusters of two consecutive iterations is too big, that willresult in a large number of end-devices in that speciﬁc region and, as a consequence, the probability ofcollisions and of co-SF interference will be high. In the same way, SF7 will have a larger coverage areaand more nodes if K is high for the last iteration.5. Numerical Results and DiscussionIn this section, we evaluate the scalability and performance of the proposed methodology bymeans of computer simulations. The results are based on a p0 = 1% duty cycle, BW = 125 kHz, η = 2.75path loss exponent, 868 MHz European frequency band, and network radius of R = 3 km. Table 3summarizes the parameters considered for the results.Table 3. System Parameters.Parameter Symbol ValueNodes N¯ 300–700Spreading Factor SF 7–12Bandwidth BW 125 kHzCarrier frequency f 868 MHzNoise ﬁgure NF 6 dBmTransmit power P1, 14 dBmDuty cycle p0 1%Path loss exponent η 2.755.1. SF Allocation and Scalability AnalysisAs discussed in Section 4, we used mathematical series to assign the numbers of clusters forK-means iterations. The Fibonacci series has the shortest ranges for SF7, 715 m for N¯ = 500. The distancebetween the SFs and the distribution of nodes can be changed by modifying the number of clusters (K).On the other hand, for the same number of nodes, the Square series has a longer range for SF7, whichis 1201 m, and it contains more end-devices. This type of conﬁguration is due to the higher valueof K clusters for iterations (see Table 2). While in the case of Fibonacci and equal-distance-based SFallocation in [19,20], the network has fewer nodes in SF7, and the number of nodes in each SF regionincreases considerably towards the higher SFs, as shown in Table 4. The average SF ranges for theFibonacci series, square series, Wythoff array and arithmetic series are shown in Table 5.Table 4. Comparison of the proposed approach with the reference method, based on the numberof nodes.Series N¯ SF7 SF8 SF9 SF10 SF11 SF12300 18 19 48 66 75 77Fibonacci series 500 29 35 79 107 124 129700 40 51 110 148 173 180300 44 24 43 52 63 77Arithmetic series 500 69 42 70 87 107 129700 94 61 98 121 149 180300 53 33 55 44 58 61Square numbers 500 81 57 87 75 99 105700 111 81 121 105 137 148300 49 24 45 52 60 73Wythoff array 500 77 42 75 87 100 123700 103 62 103 121 140 173300 9 25 42 59 76 92Reference model 500 14 42 70 98 126 152700 20 59 97 137 176 214115Sensors 2019, 19, 4723Table 5. Comparison of proposed approach with the reference method based on the distance ofindividual SF outer boundaries from the gateway (values in meters).Series N¯ SF7 SF8 SF9 SF10 SF11 SF12300 735 1049 1586 2112 2588 3000Fibonacci series 500 715 1060 1591 2112 2586 3000700 707 1071 1601 2112 2587 3000300 1144 1412 1806 2194 2590 3000Arithmetic series 500 1110 1403 1795 2183 2584 3000700 1099 1409 1801 2188 2588 3000300 1248 1591 2037 2336 2680 3000Square numbers 500 1201 1568 2004 2316 2670 3000700 1190 1568 2002 2313 2667 3000300 1209 1469 1872 2246 2613 3000Wythoff array 500 1168 1453 1857 2237 2607 3000700 1150 1450 1851 2231 2604 3000300 500 1000 1500 2000 2500 3000Reference model 500 500 1000 1500 2000 2500 3000700 500 1000 1500 2000 2500 3000The boxplot is a standard process to quantify the variability of data on the basis of ﬁve parameters,i.e., the minimum, ﬁrst quartile (25%), median, third quartile (75%), and maximum. Th distanceof the SFs boundary from the gateway for N¯ = 500 is demonstrated in Figure 5 for each of theconsidered series.SF7 SF8 SF9 SF10 SF11 SF1250010001500200025003000SF7 SF8 SF9 SF10 SF11 SF1210001500200025003000SF7 SF8 SF9 SF10 SF11 SF1210001500200025003000SF7 SF8 SF9 SF10 SF11 SF1210001500200025003000Figure 5. Boxplots representing the distance of SFs from the gateway. The red “+” signs indicateoutliers. SF12 has a constant distance of 3 km for all of the series.The median of every SF is identical to distances provided in Table 5. As depicted in the graphicalresults, SF7 and SF8 have a large disparity in range and number of nodes for different scenarios, whileSF11 and SF12 have nearly close coverage areas for all scenarios. Furthermore, we can also clearlyobserve that variation in K clusters selection has a direct effect on SF allocation based on the proposedmethodology. The number of nodes in each SF are illustrated in Table 4. Square-series-based networksdemonstrate ﬁve times more nodes in SF7 as compared to the reference model.116Sensors 2019, 19, 4723The large values of K clusters for the last iterations result in longer radii that directly increasethe region of SF7 and keep SF8 further away from the gateway. These different scenarios can be usedaccording to different situations and requirements of LoRa applications. An approach based on theFibonacci series is beneﬁcial for applications where fewer nodes in lower SFs are required, while theWythoff array, square series, and arithmetic series have wider regions for SF7, and thus can be used insetups where more nodes are required in SF7 inside the radio range of nearly 1200 m to provide highestdata rate (Rb = 5.47 kbps, see Table 1). Several studies examined the performance of LoRa networks andshow that the success probability of data packets decreases for higher SFs. In our work, we considerthe effect of changing the coverage range and varying the number of devices for individual SFs.5.2. Performance AnalysisAfter the application of the proposed SF allocation algorithm, we investigated the performanceof the resulting LoRa network. The theoretical results were veriﬁed by Monte Carlo simulations.In the ﬁgures, each marker represents the average over 105 random deployments of the Poisson pointprocess (PPP) for a single gateway LoRa uplink model, considering an end-device at d1 meters fromthe gateway. In Figure 6, the solid lines demonstrate the theoretical results, while marker points ofthe same color illustrate the simulation outputs. The simulated results align with the theoretical ones.Within the context of the previously discussed mathematical sequences, we considered and examinethe impact of different SF allocation scenarios on connection probability H1, capture probability Q1,and coverage probability H1Q1 against the distance from the gateway.0 500 1000 1500 2000 2500 300000.20.40.60.810 500 1000 1500 2000 2500 300000.20.40.60.810 500 1000 1500 2000 2500 300000.20.40.60.810 500 1000 1500 2000 2500 300000.20.40.60.81Figure 6. LoRa uplink performance with the proposed SF allocation algorithm for different seriesand N¯ = 500 nodes. (a) Square series with K = {49, 36, 25, 16, 9}. (b) Fibonacci series withK = {34, 21, 13, 8, 5}. (c) Wythoff with K = {37, 32, 24, 16, 11}. (d) Arithmetic series with K ={34, 28, 22, 16, 10}.As expected, the distance of the end-device from the gateway has considerable inﬂuence onconnection probability H1. In the case of the Fibonacci series, the model has a better success probabilityfor lower SFs as compared to the square series, arithmetic series, and Wythoff array. This fact is117Sensors 2019, 19, 4723due to the difference in a range of individual SF boundaries for said SF allocation schemes and thedistance-dependent SNR threshold qSF. Furthermore, in the scenario of the square series, SF7 has largecoverage areas of 1201 m (see Table 5), which affects path loss attenuation and the instantaneous SNR.The SNR threshold qSF, however, remains the same at−6 dB (see Table 1). As a consequence, the outagecondition in (3) slightly degrades the network connection probability. Although a performance boostis illustrated during transitions of end-devices into the next SF because of the lower value of qSF,the performance of the previous SF has a direct consequence on the next SF.Moving towards the capture probability Q1, unlike H1, it considers co-SF interference. Q1 declinesgradually with increasing SF, as illustrated in Figure 6. This trend is because of two major factorsincluding ToA and the number of nodes in each annulus. ToA grows exponentially with SF, thus forthe higher SFs, the wireless channel remains occupied for a long time slot, which increases the riskof collisions between simultaneously transmitted LoRa packets. In the same way, the number ofend-devices in an individual annulus increases for higher SFs due to the uniform distribution of nodesin the circular coverage area, as demonstrated in Table 4. As a result, the network experiences co-SFinterference that degrades the quality of transmission. For the cases with the square series, arithmeticseries, and Wythoff array (Figure 6a,c,d), the model has a larger coverage area and more nodes inSF7, resulting in higher co-spreading factor interference, which is the major reason behind the lowerperformance of the network for these speciﬁc scenarios. Although we are sacriﬁcing network qualityfor lower SFs with fewer nodes and high success probabilities, as presented in Figure 7, we improvedthe performance of the network for the higher SFs and regions with more nodes, where the networkperformance was weak in the baseline model from [19], which considers ﬁxed distance steps from thegateway to deﬁne the SF allocation. Moreover, SF allocation based on the square series (Figure 6a)has better network performance, which happens because of improved gain in Q1 for higher SFs ascompared to the Fibonacci series, arithmetic series, and Wythoff array.0 500 1000 1500 2000 2500 3000-0.3-0.2-0.100.10.20 500 1000 1500 2000 2500 3000-0.1-0.0500.050.10 500 1000 1500 2000 2500 3000-0.25-0.2-0.15-0.1-0.0500.050.10.150 500 1000 1500 2000 2500 3000-0.25-0.2-0.15-0.1-0.0500.050.10.15Figure 7. Comparison of performance gain of different series with respect to the baseline model forN¯ = 500, network size of R = 3 km with p0 = 1%, and path loss exponent η = 2.75.118Sensors 2019, 19, 4723In Figure 8, we present the performance of the square series based on the proposed SF allocationalgorithm in comparison with the baseline model. First, we observe that the capture probability Q1and coverage probability H1Q1 of the baseline model outperform the proposed algorithm in the regionof radius R > 1000 m from the gateway. Here, it is worth mentioning that there are fewer nodes inthis region as compared to the remaining area of the network. The proposed algorithm surpasses thebaseline model with a gain in capture probability Q1 (up to 53% for SF12 in Figure 8b). In addition, thenodes closer to the gateway always have better behavior in contrast with nodes far away, so that thenetwork can tolerate a lower success probability with a boost in performance of farther end-devices.As expected, there was more change in success probability in the higher SFs region as compared tobaseline model because of the fair distribution of SF by the proposed algorithm. Figure 8b shows thedifference between the baseline model and the square series in terms of outage probabilities throughthe course of distance. The zero level on the y-axis (success probability) shows no difference, whilethere is a positive/negative gain either on the upper or lower side of that level. There is up to 16.73%growth in Q1 demonstrated by the end-devices present in higher SFs.0 500 1000 1500 2000 2500 30000.20.30.40.50.60.70.80.910 500 1000 1500 2000 2500 3000-0.25-0.2-0.15-0.1-0.0500.050.10.150.2Figure 8. (a) Performance comparison of square series with the reference model for N¯ = 500.(b) Performance gain (success probability of proposed model − success probability of referencemodel). The proposed SF allocation approach sacriﬁces Q1, H1Q1 for the lower SFs but achievesgreater performance for the higher SFs.Moreover, we also consider the evaluation of the average coverage probability of the networksfor different numbers of nodes (N¯) ranging from 300 to 700, and results demonstrate that ℘c dropsexponentially towards higher N¯. Figure 9 depicts the average coverage probability of different numbersof end-devices. The SF allocation schemes deployed using the square series demonstrated a betterperformance gain than all other scenarios, including the reference model. It was followed by theWythoff array, arithmetic series, and Fibonacci series, in that order. The proposed SF allocationscheme overcomes the performance of the baseline model by an overall growth of around 5% in its ℘c.For instance, taking the square series into account, at N¯ = 500 there is a boost in the average coverageprobability (6) from ℘c = 41.9% to 46.81% compared with the reference model. On the other hand, SFallocation schemes deployed using the Fibonacci series showed the least-improved network coverageprobability compared to all other user distribution series.119Sensors 2019, 19, 4723300 400 500 600 7000.20.250.30.350.40.450.50.550.60.650.70.41140.3610Figure 9. Comparison of the coverage probabilities of the LoRa uplink as a function of the number ofnodes ranging from N¯ = 300 to 700 nodes for the network size of R = 3 km with p0 = 1% and path lossexponent η = 2.75.We also investigated the performance of the proposed model taking into account the variation ofdifferent parameters on Fibonacci series. As seen in Figure 10a,b, H1 is agnostic to the number of nodesand duty cycle. It is assessed at 0.1% and 1%, which is within the duty cycle range speciﬁed by ETSIfor LoRa applications [34,35]. Furthermore, node density and the duty cycle demonstrate a negativeimpact on Q1 because of co-SF interference caused by increasing medium usage from N¯ = 501 nodesto N¯ = 1005 nodes. Likewise, in Figure 10c, the path loss exponent η illustrates network connectiondegradation at 2.65 from 2.5 because H1 depends on the distance, while the the capture probability(Q1) is not dependent on the path loss exponent. In the case of one frequency channel, the transmitpower Pk of nodes can be up to 20 dBm (100 mW) [11,41]. In order to evaluate the effect of differenttransmit powers, in Figure 10d we raised the transmit power from 14 dBm to 19 dBm. The resultsdemonstrate that the transmit power of 19 dBm causes a better connection probability as comparedto 14 dBm. Nevertheless, variations of the path loss exponent and transmit power do not affect Q1considerably because it is much more dependent on the number of nodes.120Sensors 2019, 19, 47230 2000 4000 6000 8000 10000 1200000.20.40.60.810 2000 4000 6000 8000 10000 1200000.20.40.60.810 2000 4000 6000 8000 10000 1200000.20.40.60.810 2000 4000 6000 8000 10000 1200000.20.40.60.81Figure 10. Performance LoRa uplink considering Fibonacci series K-means clustering, and the impactof different parameters on success probabilities. (a) Density of users increased from N¯ = 501 to N¯ = 1005.(b) Duty cycle increased from p0 = 0.1% to p0 = 1%. (c) Path loss exponent from η = 2.5 to η = 2.65.(d) Transmit power from (Ptx = 14) dBm to Ptx = 19 dBm.5.3. DiscussionThe architecture of LoRaWAN consists of end-devices, a gateway, network server (NetServer),and application server [15]. The NetServer is mainly responsible for the overall management of thenetwork. The dynamic conﬁguration of the SF by the NetServer is already possible in LoRaWANduring the network join procedure or through speciﬁc MAC commands. In fact, these features areused in LoRaWAN when the adaptive data rate (ADR) mechanism is active. For our approach to beimplemented in practice, the NetServer could run our algorithm periodically or when the number ofconnected devices changes signiﬁcantly, and then issue the required MAC commands to reconﬁgurethe devices that need to change their SF. Therefore, as the current LoRaWAN speciﬁcation is alreadyable to dynamically allocate SFs, our proposal only changes the way the proper allocation is calculatedat the NetServer, and is therefore feasible in practice.6. ConclusionsThis paper has presented a novel SF allocation technique for a large-scale LoRa network using theK-means clustering machine learning algorithm. The authors also analyzed the impact of the distanceof end-devices from the gateway and the number of nodes in each SF on network performance. In thiswork, four different scenarios are considered, which have different distances for the SF boundariesand variations in the number of nodes in an individual SF. Such fair distribution results in a betteraverage coverage probability in the higher SFs, while dealing with the maximum number of nodes.Numerical ﬁndings show that our SF allocation algorithm outperforms the reference model notonly in terms of success probability but also in regards to fair resource distribution. The evaluatedtheoretical and simulation results are useful for an in-depth understanding of large and dense LoRa121Sensors 2019, 19, 4723networks. Our resource allocation method can handle dense and large circular coverage areas for LoRasensors using distinct numbers of clusters instead of equal-radius-based SF allocation [19,20], whilethe techniques in [28,29,33] are designed for short-range networks. The studies [19,20,28,29,33] onlyhighlighted the performance of networks for ﬁxed parameters. In contrast to them, our work inspectsdifferent scenarios by obeying the restrictions of ETSI standards.Author Contributions: Concept and methodology, H.A.; software, M.A.U. and J.I.; validation, A.H. and H.A.;Resources, A.H. and R.D.S.; writing original draft, M.A.U. and J.I.; writing revision, H.A., A.H., and R.D.S.;supervision, H.A.; Project Administration, H.A.Funding: This research has been supported in Finland by the Academy of Finland: 6Genesis Flagship (Grantno318927) and EE-IoT (no319008); as well as the BusinessFinland MOSSAF project. This work has been supportedin Brazil by CNPq, PrInt CAPES-UFSC “Automation 4.0”, and INESC P&D Brasil (Project F-LOCO, Energisa,ANEEL PD-00405-1804/2018).Conﬂicts of Interest: The authors declare no conﬂict of interest.References1. Chen, S.; Xu, H.; Liu, D.; Hu, B.; Wang, H. A Vision of IoT: Applications, Challenges, and OpportunitiesWith China Perspective. IEEE Internet Things J. 2014, 1, 349–359. [CrossRef]2. Sisinni, E.; Saifullah, A.; Han, S.; Jennehag, U.; Gidlund, M. Industrial Internet of Things: Challenges,Opportunities, and Directions. IEEE Trans. Ind. Inf. 2018, 14, 4724–4734. [CrossRef]3. Sotres, P.; Lanza, J.; Sánchez, L.; Santana, J.R.; López, C.; Muñoz, L. Breaking Vendors and City Locks througha Semantic-enabled Global Interoperable Internet-of-Things System: A Smart Parking Case. Sensors 2019,19, 229. [CrossRef] [PubMed]4. Rodrigues, J.J.P.C.; De Rezende Segundo, D.B.; Junqueira, H.A.; Sabino, M.H.; Prince, R.M.; Al-Muhtadi, J.;De Albuquerque, V.H.C. Enabling Technologies for the Internet of Health Things. IEEE Access 2018, 6,13129–13141. [CrossRef]5. Asadullah, M.; Ullah, K. Smart home automation system using Bluetooth technology. In Proceedings of the2017 International Conference on Innovations in Electrical Engineering and Computational Technologies(ICIEECT), Karachi, Pakistan, 5–7 April 2017; pp. 1–6.6. Saha, H.N.; Mandal, A.; Sinha, A. Recent trends in the Internet-of-Things. In Proceedings of the 2017 IEEE7th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA,9–11 January 2017; pp. 1–4.7. Internet of Things (IoT) Connected Devices Installed Base Worldwide from 2015 to 2025 (in Billions).Available online: https://www.statista.com (accessed on 16 October 2019).8. Yasmin, R.; Petäjäjärvi, J.; Mikhaylov, K.; Pouttu, A. On the integration of LoRaWAN with the 5G testnetwork. In Proceedings of the 2017 IEEE 28th Annual International Symposium on Personal, Indoor, andMobile Radio Communications (PIMRC), Montreal, QC, Canada, 8–13 October 2017; pp. 1–6.9. Mikhaylov, K.; Petrov, V.; Gupta, R.; Lema, M.A.; Galinina, O.; Andreev, S.; Koucheryavy, Y.;Valkama, M.; Pouttu, A.; Dohler, M. Energy Efﬁciency of Multi-Radio Massive Machine-Type Communication(MR-MMTC): Applications, Challenges, and Solutions. IEEE Commun. Mag. 2019, 57, 100–106. [CrossRef]10. Vangelista, L.; Zanella, A.; Zorzi, M. Long-range IoT technologies: The dawn of LoRa. In Proceedingsof the Future Access Enablers for Ubiquitous and Intelligent Infrastructures, Cham, Switzerland,23–25 September 2015; pp. 51–58.11. Petajajarvi, J.; Mikhaylov, K.; Roivainen, A.; Hanninen, T.; Pettissalo, M. On the coverage of LPWANs: Rangeevaluation and channel attenuation model for LoRa technology. In Proceedings of the 2015 14th InternationalConference on ITS Telecommunications (ITST), Copenhagen, Denmark, 2–4 December 2015; pp. 55–59.12. Lauridsen, M.; Vejlgaard, B.; Kovacs, I.Z.; Nguyen, H.; Mogensen, P. Interference Measurements in theEuropean 868 MHz ISM Band with Focus on LoRa and SigFox. In Proceedings of the 2017 IEEE WirelessCommunications and Networking Conference (WCNC), San Francisco, CA, USA, 19–22 March 2017; pp. 1–6.13. Cattani, M.; Boano, C.A.; Romer, K. An experimental evaluation of the reliability of lora long-rangelow-power wireless communication. J. Sens. Actuator Netw. 2017, 6, 7. [CrossRef]14. AN1200.22 LoRaTM Modulation Basics, Rev. 2. Semtech, 2015. Available online: http://www.semtech.com/uploads/documents/an1200.22.pdf (accessed on 31 July 2019).122Sensors 2019, 19, 472315. LoRa Alliance. Available online: http://www.lora-alliance.org (accessed on 30 September 2019).16. Phung, K.H.; Tran, H.; Nguyen, Q.; Huong, T.T.; Nguyen, T.L.; Nguyen, H. Analysis and assessmentof LoRaWAN. In Proceedings of the 2018 2nd International Conference on Recent Advances in SignalProcessing, Telecommunications & Computing (SigTelCom), Ho Chi Minh City, Vietnam, 29–31 January 2018;pp. 241–246.17. LoRa Technology Is Connecting Our Smart Planet. Available online: https://www.semtech.com/lora/lora-applications (accessed on 31 July 2019).18. de Castro Tomé, M.; Nardelli, P.H.J.; Alves, H. Long-Range Low-Power Wireless Networks and SamplingStrategies in Electricity Metering. IEEE Trans. Ind. Electron. 2019, 66, 1629–1637. [CrossRef]19. Georgiou, O.; Raza, U. Low-Power Wide-Area Network Analysis: Can LoRa Scale? IEEE Wirel. Commun.Lett. 2017, 6, 162–165. [CrossRef]20. Hoeller, A.; Souza, R.D.; Alcaraz López, O.L.; Alves, H.; de Noronha Neto, M.; Brante, G. Analysis andPerformance Optimization of LoRa Networks With Time and Antenna Diversity. IEEE Access 2018, 6,32820–32829. [CrossRef]21. Augustin, A.; Yi, J.; Clausen, T.; Townsley, W. A Study of LoRa: Long-Range & Low-Power Networks for theInternet-of-Things. Sensors 2016, 16, 1466.22. Haxhibeqiri, J.; De Poorter, E.; Moerman, I.; Hoebeke, J. A Survey of LoRaWAN for IoT: From Technology toApplication. Sensors 2018, 18, 3995. [CrossRef] [PubMed]23. Mikhaylov, K.; Petaejaejaervi, J.; Haenninen, T. Analysis of Capacity and Scalability of the LoRa Low-PowerWide-Area Network Technology. In Proceedings of the 22nd European Wireless Conference, Oulu, Finland,18–20 May 2016; pp. 1–6.24. Haxhibeqiri, J.; Karaagac, A.; Van den Abeele, F.; Joseph, W.; Moerman, I.; Hoebeke, J. LoRa indoorcoverage and performance in an industrial environment: Case study. In Proceedings of the 2017 22nd IEEEInternational Conference on Emerging Technologies and Factory Automation (ETFA), Limassol, Cyprus,12–15 September 2017; pp. 1–8.25. Lavric, A.; Petrariu, A.I.; Popa, V. Long Range SigFox Communication Protocol Scalability Analysis UnderLarge-Scale, High-Density Conditions. IEEE Access 2019, 7, 35816–35825. [CrossRef]26. Mekki, K.; Bajic, E.; Chaxel, F.; Meyer, F. A comparative study of LPWAN technologies for large-scale IoTdeployment. ICT Express 2019, 5, 1–7. [CrossRef]27. Noreen, U.; Bounceur, A.; Clavier, L. A study of LoRa low-power and wide-area network technology.In Proceedings of the 2017 International Conference on Advanced Technologies for Signal and ImageProcessing (ATSIP), Fez, Morocco, 22–24 May 2017; pp. 1–6.28. Cuomo, F.; Campo, M.; Caponi, A.; Bianchi, G.; Rossini, G.; Pisani, P. EXPLoRa: Extending the performanceof LoRa by suitable spreading factor allocations. In Proceedings of the 2017 IEEE 13th InternationalConference on Wireless and Mobile Computing, Networking and Communications (WiMob), Rome, Italy,9–11 October 2017; pp. 1–8.29. Cuomo, F.; Gámez, J.C.C.; Maurizio, A.; Scipione, L.; Campo, M.; Caponi, A.; Bianchi, G.; Rossini, G.; Pisani, P.Towards trafﬁc-oriented spreading factor allocations in LoRaWAN systems. In Proceedings of the 201817th Annual Mediterranean Ad Hoc Networking Workshop (Med-Hoc-Net), Capri, Italy, 20–22 June 2018;pp. 1–8.30. Kim, D.Y.; Kim, S. Data Transmission Using K-Means Clustering in Low Power Wide Area Networks withMobile Edge Cloud. Wirel. Pers. Commun. 2018, 105, 567–581. [CrossRef]31. Farhad, A.; Kim, D.; Pyun, J. Scalability of LoRaWAN in an Urban Environment: A Simulation Study.In Proceedings of the 2019 Eleventh International Conference on Ubiquitous and Future Networks (ICUFN),Zagreb, Croatia, 2–5 July 2019; pp. 677–681.32. Magrin, D.; Centenaro, M.; Vangelista, L. Performance evaluation of LoRa networks in a smart city scenario.In Proceedings of the 2017 IEEE International Conference on Communications (ICC), Paris, France, 21–25 May2017; pp. 1–7.33. Zhu, G.; Liao, C.-H.; Sakdejayont, T.; Lai, I.-W.; Narusue, Y.; Morikawa, H. Improving the Capacity of a MeshLoRa Network by Spreading-Factor-Based Network Clustering. IEEE Access 2019, 7, 21584–21596. [CrossRef]34. Vincenzo, V.D.; Heusse, M.; Tourancheau, B. Improving Downlink Scalability in LoRaWAN. In Proceedingsof the ICC 2019—2019 IEEE International Conference on Communications (ICC), Shanghai, China,20–24 May 2019; pp. 1–7.123Sensors 2019, 19, 472335. Hasegawa, Y.; Suzuki, K. A Multi-User ACK-Aggregation Method for Large-Scale Reliable LoRaWANService. In Proceedings of the ICC 2019—2019 IEEE International Conference on Communications (ICC),Shanghai, China, 20–24 May 2019; pp. 1–7.36. Li, L.; Ren, J.; Zhu, Q. On the application of LoRa LPWAN technology in sailing monitoring system.In Proceedings of the 13th Annual Conference on Wireless On-demand Network Systems and Services(WONS), Jackson, WY, USA, 21–24 February 2017; pp. 77–80.37. Yasmin,R.; Petäjäjärvi, J.; Mikhaylov, K.; Pouttu, A. Large and Dense LoRaWAN Deployment toMonitor Real Estate Conditions and Utilization Rate. In Proceedings of the 2018 IEEE 29th AnnualInternational Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC), Bologna,Italy, 9–12 September 2018.38. Abramson, N. The Aloha System—Another alternative for computer communications. In Proceedings of theFallJoint Computer Conference (AFIPS ’70 (Fall)), New York, NY, USA, 17–19 November 1970; pp. 281–285.39. Goursaud, C.; Gorce, J.M. Dedicated networks for IoT: PHY/MAC state of the art and challenges.EAI Endorsed Trans. Internet Things 2015, 1. [CrossRef]40. Na, S.; Xumin, L.; Yong, G. Research on k-means Clustering Algorithm: An Improved k-means ClusteringAlgorithm. In Proceedings of the 2010 Third International Symposium on Intelligent Information Technologyand Security Informatics, Jinggangshan, China, 2–4 April 2010; pp. 63–67.41. Bouguera, T.; Diouris, J.F.; Chaillout, J.J.; Jaouadi, R.; Andrieux, G. Energy Consumption Model for SensorNodes Based on LoRa and LoRaWAN. Sensors 2018, 18, 2104. [CrossRef] [PubMed]c© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open accessarticle distributed under the terms and conditions of the Creative Commons Attribution(CC BY) license (http://creativecommons.org/licenses/by/4.0/).124sensorsArticleUser Association and Power Control for EnergyEfﬁciency Maximization in M2M-Enabled UplinkHeterogeneous Networks with NOMAShuang Zhang 1,2 and Guixia Kang 1,2,*1 Key Laboratory of Universal Wireless Communications, Ministry of Education, Beijing University of Postsand Telecommunications, Beijing 100876, China; zhangshuang2015@bupt.edu.cn2 Wuxi BUPT Sensory Technology and Industry Institute CO. LTD, Wuxi 214000, China* Correspondence: gxkang@bupt.edu.cnReceived: 8 October 2019; Accepted: 25 November 2019; Published: 2 December 2019Abstract: To support a vast number of devices with less energy consumption, we propose a new userassociation and power control scheme for machine to machine enabled heterogeneous networkswith non-orthogonal multiple access (NOMA), where a mobile user (MU) acting as a machine-type communication gateway can decode and forward both the information of machine-typecommunication devices and its own data to the base station (BS) directly. MU association and powercontrol are jointly considered in the formulated as optimization problem for energy efﬁciency (EE)maximization under the constraints of minimum data rate requirements of MUs. A many-to-one MUassociation matching algorithm is ﬁrstly proposed based on the theory of matching game. By takingswap matching operations among MUs, BSs, and sub-channels, the original problem can be solvedby dealing with the EE maximization for each sub-channel. Then, two power control algorithms areproposed, where the tools of sequential optimization, fractional programming, and exhaustive searchhave been employed. Simulation results are provided to demonstrate the optimality properties of ouralgorithms under different parameter settings.Keywords: M2M; heterogeneous networks; non-orthogonal multiple access; energy efﬁciency;MU association; power control1. IntroductionThe increase of smartphones, laptops, and other mobile devices as well as data-hungry applications,need huge demands for ubiquitous coverage and very high data rates in cellular networks. However,homogeneous networks cannot satisfy these requirements [1]. Then, two-fold efforts have been spentto meet the stringent requirements. On one hand, researchers have proposed heterogeneous networks(HetNets) where different types of base stations (BSs), e.g., macro BSs (MBSs) and small BSs (SBSs) aredeployed in a multi-tier hierarchical structure. In this structure, all BSs have seamless coverage and reusefrequencies to achieve higher data rate [2,3]. On the other hand, the so-called non-orthogonal multipleaccess (NOMA) has been investigated as a potential technique to further improve the throughput ofnetwork [4–7]. Different from conventional orthogonal multiple access (OMA), NOMA serves multipleusers at the same time/frequency/codes resource by allocating different powers for them, and thesuperposition coded signal can be decoded at receivers by successive interference cancellation (SIC).Therefore, the combination of HetNets and NOMA will exhibit great potential to satisfy the 1000-timesincrease of mobile broadband data for the upcoming fifth generation (5G) communication systems andbeyond [3].However, the severe inter-tier and intra-tier interference make the NOMA-enabled HetNetschallenging to achieve. Resource management plays an important role to alleviate these interference [8].Sensors 2019, 19, 5307; doi:10.3390/s19235307 www.mdpi.com/journal/sensors125Sensors 2019, 19, 5307For downlink communication, speciﬁcally, some work focuses on the sum rate maximization andshows higher spectral efﬁciency (SE) can be achieved by NOMA when considering the intercellinterference [9–12]. Besides SE, energy efﬁciency (EE) is also a key performance metric investigated forresource allocation in NOMA-enabled HetNets [8,13,14]. Moreover, EE is more important in uplinkthan in downlink NOMA-enabled HetNets since the devices in uplink communications are oftenbattery-limited. It is a fact that the battery capacity has been improved at a very slow pace over thepast decades [15], and hence this increase cannot scale with the high energy consumption caused bythe increasing trafﬁc demands. Meanwhile, EE has emerged as a new prominent performance metricfor wireless communication networks designs due to the economic, operational, and environmentalconcerns [16,17]. Therefore, it is a stringent work to improve EE for uplink transmission.Machine-to-machine (M2M) communications, also known as machine-type communications(MTC), enable pervasive connections to support IoT. M2M communications are one of the potentialapplications of NOMA-enabled HetNets [18], since NOMA-enabled HetNets provide a practicalinfrastructure to offer massive access opportunities for such a huge number of devices, especiallyfor the cases in which each device only needs to send a small amount of data periodically in uplink.One of the challenges for HetNets with M2M communications is the access control, which can managethe engagement of massive MTC devices (MTCDs) to the core network. Among the existing accesssolutions, deploying MTC gateways (MTCGs) is an effective approach to connect M2M communicationand cellular communication [19–21]. When mobile user (MU) has more power and storage space thanMTCDs (e.g., smart sensors), the MU can be conﬁgured as the MTCG, as proposed in [22].Since 5G will be HetNets including various network models (e.g., cellular networks, wirelessnetworks (WSNs), and low power wireless area networks) to support high data rate and massivedevices [3], our work combing M2M communication and cellular network has a large signiﬁcance forthis heterogeneous scenario. The short distance communication in our system model can be realizedwith WSNs, which provide a new way to help the sink nodes in WSNs communicate to the corenetwork. For example, the MTCDs can be the sensors in an environmental monitoring WSN, and theycan transmit the collected data to the core network through a mobile device in cellular networks withNOMA. Therefore, our work also has a practical signiﬁcance for sensors work.Recently, there have been some studies addressing the aforementioned challenges of applyingNOMA in HetNets for EE maximization. In [23], a distributed user association algorithm based oninter-cell interference plus noise ratios of BS and a centralized user association based on the popularsize of BS were both proposed. After user association was determined, a power control algorithmwas proposed based on Lagrangian dual method, then a one-dimensional search algorithm wasused to search Lagrangian multiplier, which added algorithm complexity. Two speciﬁc exampleswere provided to demonstrate the effectiveness of uniﬁed NOMA-enabled heterogeneous ultra-densenetworks with user association and power control in [18]. An alternated energy efﬁcient resourceallocation algorithm based on ﬁxed power allocation was ﬁrst proposed in [13]. Then, two iterativeenergy-efﬁcient resource allocation algorithms were proposed to update for better EE based onLagrangian dual method. Joint base station association and power control optimization algorithmswere proposed based on coalition formation games and interior-point method in [24], but sub-channelallocation and fractional equation for EE maximization had not been considered. Moreover, the userassociation algorithms in the aforementioned work were all considered with ﬁxed power allocationﬁrstly, whereafter iterative algorithms were used to obtain the ﬁnal optimal value.There are also some studies on the usage of M2M communications in NOMA systems.For example, energy-efﬁcient resource allocation with hybrid division multiple-access NOMA forcellular-enabled M2M communications was researched in [25,26]. With MTCDs cluster formationknown beforehand, standard convex optimization and Lagrange duality methods were employedrespectively for power control in [25,26]. User clustering in NOMA-aided cellular M2M communicationsystems was researched in [27,28] with millimeter-wave and narrow-band IoT separately. A jointpower and sub-channel allocation for secrecy capacity algorithm was proposed in [29] to obtain the126Sensors 2019, 19, 5307suboptimal solution of the optimization problem. However, the aforementioned work deploys M2Muser in single-cell networks. The trend of more and more intensive network deployment motivates usto deploy M2M-enabled NOMA in the scenarios with multi-tier HetNets and new resource allocationneeds to be considered with the non-convexity caused by inter-cell interference in HetNets.In this paper, we focus on the uplink EE maximization via user association and power controlfor M2M-enabled HetNets using NOMA. In this scenario, one macro base station (MBS) is locatedin the cell center. Each small cell has one small base station (SBS) located in the cell center. MUsare distributed randomly in the cell. An MU acting as an MTCG can decode and forward boththe information of MTCDs and its own data to the BS. The EE (bits/Joule) maximization problemis formulated and solved to obtain the optimal MU association and power allocation. The maincontributions of this paper are summarized as follows:• We propose a new framework of M2M-enabled HetNets with NOMA. In this framework, controldata separation architecture, i.e., control information and data message are separated, which canreduce the signal overhead [30]. NOMA is adopted by the MTCDs to transmit the information toMUs which is regarded as the relay. MUs decode the overlaid information and simultaneouslytransmit received data to the BSs based on the NOMA principle.• In order to solve the EE maximization optimization problem, a BS and a sub-channel are includedin a couple, since a MU can only associate one BS at one sub-channel. Then, a many-to-one MUassociation algorithm is proposed based on matching game [31]. Through swap operation amongeach couple, the EE maximization problem can be tackled by solving the power control problemat each sub-channel. Compared with the previous studies on the algorithms (user association andpower allocation) [13,18,23,24], our algorithms are jointly optimized and ﬁxed power allocationis not required for initialization.• Two power control algorithms are proposed based on sequential optimization [32,33].The fractional programming [34] and sequential optimization are combined to develop a novelsequential fractional power control algorithm (SFPCA), from which the original problem istransformed to be convex and requires less computational complexity. The other algorithmcombines the exhaustive search method with sequential optimization, which can verify thecorrectness of SFPCA.The rest of this paper is organized as follows. The system model and problem formulation arefocused in Section 2. The MU association matching algorithm is proposed in Section 3. The powercontrol problem is solved in Section 4. Numerical results are provided in Section 5, and concludingremarks are given in Section 6.Notations: Lowercase and uppercase boldface letters denote vectors and matrices, respectively.We use uppercase decorated letters to denote sets. For an arbitrary set M, we always have thecorresponding uppercase M to the denote the cardinality of M, i.e., |M| = M, [·]T denotes thetranspose operator.2. System Model and Problem Formation2.1. System ModelAs shown in Figure 1, we consider uplink HetNets with M2M communications, where all MUsare anchored to the control base station (CBS). The CBS performs the MU association algorithm toselect the best serving BS for MUs and establishes a high BS–MU connection through backhaul links.Each MTCD selects the nearest MU as an MTCG. Since NOMA is adopted between MTCDs that selectthe same MU as their MTCGs, SIC is performed at the MU to gather the interference and channel gainwill be obtained by channel estimation at the MU. The HetNets consist of a set F = {0, · · · , F} of BSsand a set K = {1, · · · ,K} of MUs. Each MU is regarded as an MTCG, which can acts as a relay forsome MTCDs. Denote Uk as the speciﬁc set of MTCDs served by MU k (MUk). The index 0 denotes theMBS and other indexes stand for the SBSs in set F . Without special explanation, we always have f ∈ F ,127Sensors 2019, 19, 5307F = |F |. The system bandwidth shared by all BSs is divided into N orthogonal sub-channels, and eachone is assigned with bandwidth B. For convenience, hereinafter we always have n ∈ N = {1, 2, · · · N}to denote the sub-channel. MUs are served by BSs according to the BSs’ coverage.?????????? ??????????????????????????????????????????????????? ????????? ????????? ? ???????? ??????????????????????????????????????????????????????????????????????????????????????????? MU2 signal dectection MU1 signal dectection MU3 signal dectectionFigure 1. System Model.2.2. NOMA StrategyMultiple MTCDs can simultaneously transmit signals to the MU using NOMA. Since MUs andMTCDs use different transmission modes, we ignore the interference between MUs and MTCDs.The interference between MTCDs in different BSs is also not considered. According to the NOMAprinciple, the received signal of MUk isYk = ∑j∈Ukhjkk√qjk sjk + nk, (1)where hjk is the channel between MTCD jk and MUk; qjk and sjk denote the transmit power and messageof MTCD jk; and nk represents the additive zero-mean Gaussian noise with variance σ2. Uk representsthe set of MTCDs which are served by MUk. Without loss of generality, the channels are sorted by∣∣h1kk∣∣2/σ2k > ∣∣h2kk∣∣2/σ2k > · · · ∣∣hUkk∣∣2/σ2k > 0. Applying SIC in NOMA [33], the achievable datathroughput for MTCD jk at MUk is given byRjk = log2(1+Hjkkqjk1+ Ijk), (2)where Hjkk =∣∣∣hjkk∣∣∣2/σ2k , Ijk = ∑i∈{Uk∣∣∣Hikk ∣∣∣h2 f n∣∣∣2/σ2f n > · · ·∣∣∣hk f n∣∣∣2/σ2f n > · · · ∣∣∣hSfn f n∣∣∣2/σ2f n, where Sf n =∣∣∣S f n∣∣∣.Then the transmit data rate of MUk associated with BS f over SCn can be expressed asRk f n = Blog2(1+pk f nHk f n1+ Ik f n + φk f n), (3)where Hk f n =∣∣∣hk f n∣∣∣2/σ2f n. Ik f n is the interference that MUk receives from other MUs whose channelgains are smaller than that on SCn of BS f , which can be given byIk f n = ∑i∈{S f n|Hi f n 0, pk f n ≤ Pmax, ∀k ∈ K, f ∈ F , n ∈ N , (7c)where P is the transmit power vector with elements pk f n; Pmax is the maximum transmit power of eachMU; and Rreq is the minimum data rate requirement of a MU. Since each MU is regarded as an MTCGfor MTCDs, they should ensure the data rate that MTCDs can be uploaded to the SBS, therefore wehave constraint (7b) as the data rate requirement of MUk associated to BS f at SCn [22]. Constraint (7c)is used to guarantee the feasible value ranges of P.129Sensors 2019, 19, 53073. MU AssociationThe MTCDs associated to the corresponding MU are known beforehand. Since solving theoptimization problem is equal to obtain the optimal EE of each sub-channel, the MU associationwill become the matching problem among BSs, sub-channels and MUs to achieve sub-channel EEmaximization. Thus, we propose a MU association algorithm using matching game [30] in thefollowing parts.3.1. Matching Problem FormulationTo develop a low-complexity MU association algorithm, we ﬁrst regard a sub-channel and a BS asa couple, denoted as (n, f ). Then, the optimization problem is transformed to match the MUs to thecouples and allocate power appropriately, such that the EE can be maximized. Finally, the matchingproblem is a many-to-one problem between MUs and couples based on matching game, which isdescribed as follows.Deﬁnition 1. Given two disjoint sets, K = {1, · · · ,K} denotes the set for MUs, and M ={(1, 1) (1, 2) · · · , (2, 1) · · · , (n, f ), · · · (N, F)} represents the couples. A many-to-one matching Ψ isa mapping from the set K ∪M into the set of all subsets of K ∪M for f ∈ F , k ∈ K, n ∈ N satisfyingi) Ψ(k) ∈ M;ii) Ψ(n, f ) ⊂ K;iii) |Ψ(k)| = 1 |,Ψ(SCn, f )| = Sf n;iv) (n, f ) = Ψ(k) ⇔ k ∈ Ψ(n, f ).Condition i indicates that each MU matches with a sub-channel-BS couple. On the other hand,each couple matches a subset of MUs, which is illustrated in condition ii. Condition iii states a MU canonly associate one BS and choose one sub-channel while each couple matches Sf n MUs.The aim of each couple is to maximize its own EE. To this end, we exploit the swap operationinto our matching algorithm. A swap operation means two MUs matching with different couplesexchange their matchings based on different cases, while other MUs remain their matchings. The EEof the exchanged couples will be recomputed by the power control algorithm. Note that how toallocate power to obtain the optimal EE for a given sub-channel will be presented in the next section,and we assume it is known in advance. A swap operation will be approved and the matching willbe exchanged only when all EE of the sub-channels belonging to the exchanged couples increase ifthe swap is performed. The swap operation will be continued until no swap is further preferred.More details are described in Algorithm 1.3.2. Matching AlgorithmAlgorithm 1 contains a initialization phase and a swap matching phase. Considering the userfairness, the number of MUs accommodated by one sub-channel in a given BS is at most⌈KFN⌉. In theinitialization step, the basic idea is to associate the MU to the couple providing the largest channelgain. This will lead to either a higher data rate for the MU, or a lower transmit power. Since the valueof sub-channel gain between MU and the uncovered BSs is invalid and the maximized sub-channelgain is always chosen, there is no need to know whether the MUs are in the coverage of the exchangedBSs. However, in the swap matching phase, this judgment should be considered at ﬁrst to avoid theinvalid swap. Then, exchange will happen in the three cases. Iterations will continue until no swapoperation can be approved in a new round.130Sensors 2019, 19, 5307Algorithm 1 The MU association matching algorithm.Initialization phase: L =⌈KNF⌉, Kˆ = K1: for l = 1 : L do2: Mˆ = M, Count = 13: while (Count ≤ M) do4: Hk∗ f ∗n∗=argmax{Hk f n}∀k∈Kˆ,( f ,n)∈Mˆ. Assign k∗ to the couple ( f ∗, n∗), Kˆ = Kˆ\k∗, Mˆ =Mˆ\( f ∗, n∗), and set Count = Count + 15: end while6: end forSwap matching phase: Indicator = 17: while (Indicator) do8: for u = 1 : K do9: for k = 1 : K do10: if Ψ(k) = Ψ(u) then11: continue;12: else if MUk and MUu are both in the coverage of the BSs of each other then13: switch (Ψ(k),Ψ(u) )14: case MUk and MUu belonging to the same BS and different sub-channels:15: Calculate and compare the EE of the two sub-channels before and after the swap usingthe power control algorithm. If the EE of the two-subchannels both improve, exchangethe sub-channel, form the new couple, and set Indicator = 1.16: case MUk and MUu belonging to the different BSs and different sub-channels:17: Calculate and compare the EE of the two sub-channels before and after the swap usingthe power control algorithm. If the EE of the two sub-channels both improve, exchangethe couple, form the new couple, and set Indicator = 1.18: case MUk and MUu belonging to the different BSs and same sub-channels:19: Calculate the EE of the sub-channel before and after the swap using the power controlalgorithm. If the EE of the sub-channel has been improved, exchange the BS, form thenew couple, and set Indicator = 1.20: end switch21: end if22: end for23: end for24: end while3.3. Convergence and ComplexityTheorem 1. The proposed MU association and power control algorithm converges after a ﬁnite number ofswap operations.Proof of Theorem 1. For each swap operation, the matching changes from Ψex to Ψnow. We haveEEn,ex and EEn,now to denote the corresponding EE of of Ψex and Ψnow on SCn . Based on the aim ofswap operation, we have EEn,now > EEn,ex, that is, the EE of each sub-channel increases after eachswap matching. Since each sub-channel is orthogonal to each other, the system EE will increase owingto the improved EE of each sub-channels. Moreover, the system EE has an upper bound due to thelimited transmit power of each MU. Therefore, the MU association algorithm and power allocationconverge after a ﬁnite number of swaps.131Sensors 2019, 19, 53074. Power ControlIn this section, we will investigate the optimal power control design appearing in Algorithm 1 toobtain the maximum EE of SCn. Before we present the optimization problem for EEn maximization,we ﬁrst deal with Rn, which can be rewritten asRn(p f n) = B ∑f∈Flog2⎛⎜⎝1+∑k∈S f npk f nHk f n1+ φk f n⎞⎟⎠, (8)and we can also obtainRtot,k = ∑j∈UkRjk = log2(1+ ∑j∈UkHjkkqjk). (9)Due to the multi-interference in the sum-rate function in (8), EEn in (6) is non-convex and cannotbe directly solved by the generalized fractional programming approach. Then, we ﬁrst transformthe numerator into the difference of two non-negative functions and the EEn maximization can berewritten asmaxp f nη˜n =F+(p f n)− F−(p f n)∑f∈F∑k∈S f npk f n + Pc(10a)s.t. (7c), (11), (10b)withC+k f n(p f n)− C−k f n(p f n) ≥ 0, ∀k ∈ K, ∀ f ∈ F , ∀n ∈ N , (11)where p f n = [p1 f n, p2 f n, · · · , pk f n, · · · , pSfn Fn]T denotes the transmit power vector for MUs on SCn.Moreover, we haveF+(p f n) = B ∑f∈Flog2⎛⎝1+ ∑k∈S f npk f nHk f n + φk f n⎞⎠ ,F−(p f n) = B ∑f∈Flog2(1+ φk f n),C+k f n(p f n) = Blog2(1+ pk f nHk f n + Ik f n + φk f n)− Rtot,k − Rreq,C−k f n(p f n) = Blog2(1+ Ik f n + φk f n).(12)Note that F+, F−,C+, andC− are concave functions regarding to p f n, then the numerator of (10a)and the constraint functions in (10b) are expressed as the difference of concave functions, which are notconcave in general. Motivated by [31,32], where sequential optimization is used to solve the similarproblem as (10), we adopt this method and combine it with fractional programming and exhaustivesearch to propose two power control algorithms. Before introducing the two algorithms, we ﬁrstpresent the details of the sequential optimization theory in the next sub-section.132Sensors 2019, 19, 53074.1. Sequential Optimization TheorySequential optimization is a powerful tool that can tackle a difﬁcult optimization problem bysolving a sequence of approximate problems in simple forms with affordable complexity. Speciﬁcally,we give a formal maximization problem F¯ with a compact feasible set as [31,32], shown asmaxxf0(x) (13a)s.t. fi(x) ≥ 0, ∀i ∈ {1, · · · , I}, (13b)where f0(x) is the differentiable objective with constraints fi(x) ≥ 0. Let G(v) be the problem solved inthe v-th iteration by the sequential method to tackle problem F¯, which can be written asmaxxg(v)0 (x) (14a)s.t. g(v)i (x) ≥ 0, ∀i ∈ {1, · · · , I}, (14b)where g(v)0 (x) is the differentiable objective with the constraints g(v)i (x). Then, if g(v)0 (x) and g(v)i (x)are suitable continuous functions and constraints, they must satisfy the following two properties:1) g(v)0 (x) ≤ f0(x), g(v)i (x) ≤ fi(x) ∀x;2) g(v)0 ((x∗)(v−1)) = f0((x∗)(v−1)), g(v)i ((x∗)(v−1))≤ fi((x∗)(v−1)).(x∗)(v−1) is the optimal solution of the problem solved at iteration (v−1)-th. This means thesolution sequence {(x∗)}(v) of (14) monotonically increases the value of (13), i.e., f0((x∗)(v)) ≥f0((x∗)(v−1)) for all v, which guarantees the convergence of the sequential method. Next, if thefollowing third property is also satisﬁed:3) ∇g(v)0 ((x∗)(v−1)) = f0((x∗)(v−1)), ∇g(v)i ((x∗)(v−1)) =∇ fi((x∗)(v−1)).then every limit point of {x}(v) of (14) fulﬁlls the Karush–Kuhn–Tucker (KKT) conditions of problem F¯in (13).Therefore, if a maximization problem ﬁnds suitable approximate problems which can fulﬁll theabove three properties, its optimal value can be approximated by solving the monotonically increasedsequential problems. The critical issue is that the suitable approximate problems are solved easier thanthe original problem. In the rest of this section, we will ﬁrst ﬁnd the sequential approximate problemsto the numerator in problem (10).4.2. Sequential Fractional Power Control AlgorithmBased on sequential optimization, we should ﬁnd a sequence problem to approximateoptimization problem (10). To circumvent this issue, we obtain the following main result with theﬁrst-order Taylor expansion at p f n(v) of F−(p f n).Proposition 1. For any given p f n(v), the sequence approximation problem of (10), denoted by G(v) can bewritten asmax ηn =F+(p f n)− F˜(p f n)∑f∈F∑k∈S f npk f n + Pc(15a)s.t. C+k f n(p f n)− C˜(p f n) ≥ 0, (15b)(7c) , (15c)133Sensors 2019, 19, 5307with optimal solution p∗f n(v), whereF˜(p f n) = F−(p f n(v))−(∇F−(p f n(v)))T (p f n − p f n(v)) (16)C˜(p f n) = C−k f n(p f n(v))−(∇C−(p f n(v)))T(p f n − p f n(v))(17)If p(v)f n = p∗f n(v−1), ∀v ≥ 1 , then{ηnp∗f n(v)}(v)is monotonically increasing and converges to a value η˜n.Furthermore, any limit point of sequence{ηnp∗f n(v)}(v)that achieves η˜n fulﬁlls the KKT optimality conditionsof (10a).Proof of Proposition 1. As we know, any concave function is the upper-bounded of its ﬁrst-orderTaylor expansion at any point. Since F−(p f n) and C−(p f n) are concave functions, for any power vectorp(v)f n we haveF+(p f n)− F−(p f n)≥ F+(p f n)− F˜(p f n)= F+(p f n)− F−(p f n(v))−(∇F−(p f n(v)))T (p f n − p f n(v)),(18)C+(p f n)− C−(p f n)≥ C+(p f n)− C˜(p f n)= C+(p f n)− C−k f n(p f n(v))−(∇C−(p f n(v)))T(p f n − p f n(v)).(19)Hence, (15a) and (15b) are lower bounds of (10a) and (11), respectively. Since the lower boundsin (16) are tight when evaluated by p(v)f n , it follows that (15a) and (15b) are equal to (10a) and (11),respectively, for p f n = p(v)f n . Similarly, the gradients of (15a) and (15b) are equal to those of (10a) and(11), for p f n = p(v)f n . Thus, (15) fulﬁlls all the properties described in the above sub-section, whichcompletes the proof of this proposition.For any p(v)f n , problem (15) has a concave numerator and an afﬁne denominator, while theconstraint functions in (15b) and (15c) are both concave and afﬁne. Therefore, (15) is a single-ratioproblem, which can be solved by the generalized fractional programming. We adopt the widely usedDinkelbach’s algorithm to solve it. According to Dinkelbach’s method [33], we ﬁrst introduce thefollowing auxiliary functionT(p f n, ηn) = fn(p f n)− ηngn(p f n), (20)with fn(p f n) = F+(p f n)− F˜(p f n), and gn(p f n) = ∑f∈F∑k∈S f npk f n + Pc.134Sensors 2019, 19, 5307Theorem 2. Let η∗n , p∗f n and p∗k f n denote the optimal value, optimal solution and its elements of problem (15),respectively. Then, we haveη∗n =F+(p∗f n)−F˜(p∗f n)∑f∈F∑k∈S f np∗k f n+Pc= maxF+(p f n)−F˜(p f n)∑f∈F∑k∈S f npk f n+Pc, (21)if and only ifmax { T(p f n, η∗n) = fn(p f n)− η∗ngn(p f n) }= fn(p∗f n)− η∗ngn(p∗f n) = 0.(22)Proof of Theorem 2. Theorem 2 was proved in [33,36], and we omit it due to the limited space.The optimal η∗n can be obtained by Dinkelbach’s method, which is summarized in Algorithm 2.As shown in the algorithm, we need to solve the problem (23) for a given parameter ηn(c) in eachiteration. In Algorithm 2, ηn has been updated as ηn(c) in each iteration until convergence. or reachingthe maximum number of iterations. p f n(c) denotes the optimal power of the following problem in thec-th iteration, which can be obtained in Algorithm 3, as given bymaxp f nT(p f n, ηn(c)) = fn(p f n)− ηn(c)gn(p f n)s.t. C+k f n(p f n)− C˜(p f n) ≥ 0,(7c) ,(23)Algorithm 2 The Dinkelbach’s algorithm.Initialization phase:Set iteration c = 1, ηn(c) > 0, the maximum number of iterations Cmax, and error tolerance τ > 0.1: repeat2: Solve the equivalent problem (23) for a given ηn(c) to obtain the solution p f n(c).3: ηn(c) =F+(p f n(c))−F˜(p f n(c))∑f∈F∑k∈S f npk f n(c)+Pc,4: c = c + 1.5: until∣∣∣T(ηn(c−1),p f n(c−1))∣∣∣ ≤ τ or c > Cmax6: η∗n = ηn(c−1), p∗f n = p f n(c−1).Algorithm 3 The algorithm for solving problem (23).Initialization phase:Set p(0)f n , iteration index v= 0, the maximum iterations Vmax and error tolerance μ. Calculatefn(p f n(0))−ηn(c)gn(p f n(0)).1: repeat2: Solve the problem (23) to obtain the optimal solution p∗f n for given p(v)f n and ηn(c).3: v = v + 1.4: Set p(v)f n=p∗f n and cacluate fn(p f n(v))−ηn(c)gn(p f n(v)).5: until∣∣∣ fn(p f n(v))− ηn(c)gn(p f n(v)) − ( fn(p f n(v−1))− ηn(c)gn(p f n(v−1)))∣∣∣ ≤ μ6: p∗f n= p(v)f n .4.3. Computational Complexity AnalysisIn above sub-section, we have proposed the SFPCA including two steps, i.e., Algorithms 2and 3. The computational complexity of them are separately discussed. First, we use C to denote135Sensors 2019, 19, 5307the number of iterations for Algorithm 2, where C is bounded by Cmax. From Section V, we can seethat Algorithm 2 will converge after a few number of iterations. Then we discuss the computationalcomplexity of Algorithm 3, the complexity of this algorithm is mainly caused by (23), and denotedby X. The computational complexity of (23) is O(Sn3) [37], where Sn is the number of MUs at SCn.The complexity of Algorithm 3 is X = VO(Sn3), where V is the the number of iterations bounded byVmax. In summary, the computational complexity of the power control algorithm is O(CX).4.4. Sequential Exhaustive AlgorithmTo evaluate the performance of the SFPCA, a sequential exhaustive algorithm (SEA) combinedwith sequential optimization and exhaustive search is proposed in this section. The detailed proceduresof the compared algorithm is illustrated as follows. To solve the problem in an easier manner, weintroduce the auxiliary variable yn, as given byyn = B ∑f∈Flog2⎛⎝1+ ∑k∈S f n pk f nHk f n1+φk f n⎞⎠. (24)if we ﬁx yn , the objective function (15a) can be recast asmaxp f nyn∑f∈F∑k∈S f npk f ns.t. ∑f∈Flog2⎛⎜⎝1+∑k∈S f npk f nHk f n1+ φk f n⎞⎟⎠ ≥ yn.(25)Due to the multi-user interference, we cannot solve problem (25) by standard convex optimizationtools. Similar to SFPCA, sequential optimization is applied and the approximate problem can beshown asmaxp f nF+(p f n)− F˜(p f n)∑f∈F∑k∈S f npk f n + Pc(26a)s.t. F+(p f n)− F˜(p f n) ≥ yn, ∀n ∈ N , (26b)(7c), (11). (26c)It can be observed that since yn is ﬁxed, (26) is equivalent to minimize the linear function∑f∈F∑k∈Sf npk f n + Pc in the denominator, subject to convex constraints. Then, problem (26) can be solvedby plain convex programming. To implement an efﬁcient line search for yn, the bound of yn is given byyn = FSf nRreq≤ ∑f∈Flog2⎛⎜⎝1+∑k∈S f npk f nHk f n1+ φk f n⎞⎟⎠< ∑f∈Flog2⎛⎝1+ ∑k∈S f nPmaxHk f n⎞⎠=yn.(27)Then, the optimial p f n can be obtained by searching an appropriate value of yn with stepsize ε.136Sensors 2019, 19, 53075. Numerical Results and DiscussionsIn this section, the effectiveness of our proposed MU association and power control algorithmsin M2M-enabled HetNets with NOMA was demonstrated by Monte Carlo simulations. The HetNetsincluded one MBS and two SBSs, and the radius of the cells for them were 200 m and 80 m, respectively.MUswere randomly and uniformly distributed. The values of the simulation parameters are summarizedin Table 1.We considered the EE performance obtained from EE maximization and sum-rate maximizationwith different Pmax in Figure 2. The latter could be obtained in the ﬁrst iteration of Algorithm 2 due toq(1) = 0. In order to reﬂect the inﬂuence of qmax, we gave four schemes of different qmax. From theﬁgure, we can see that all of the four schemes had a “green point”, where EE and sum rate couldboth achieve their optimal values. Different from sum rate, EE became gradually ﬂat while sum ratedecreased after “green point” as Pmax grew. The reason is when the maximum EE is achieved, no moretransmit power is needed. For sum rate maximization, larger sum rate requires more transmit power,and its ratio (EE) may decrease, since the numerator (sum rate) and denominator (sum transmit power)both grow. We can also see that the EE decreased as qmax increased, because the increase of qmax meansthe data rate requirement of MUs increases. It is worth noting that even though MTCDs have lowerdata rate and transmit power, they can also have a strong inﬂuence on the overall uplink EE withtheir massive number. Furthermore, Algorithm 1 with higher EE had similar tendency as Algorithm 4,which proves the correctness of our algorithms.Figure 3 shows the EE performance with respect to different data rate requirements of MUs withthe different transmit power of MTCDs. The four curves all decreased as Rreq increased. This is due tothe fact that higher data rate will narrow the feasible value regions of the transmit power. Note thatthe four curves decreased slightly ﬁrst, when Rreq = 150 bps, the EE of the four schemes all declineddistinctly, since higher data rate requirement may require more transmit power, destroying the balanceof sum transmit power and sum rate. As explained above, the increase of qmax leads to the increase ofdata rate requirement, and the variation of EE is in line with the reason as the ﬁgure shown.Table 1. simulation parameters.Parameters Meanings ValuesF Number of BSs 3B The frequency bandwidth of each sub-channel 15 kHzK Number of MUs 40Uk Number of MTCDs of each MU 2σ2f n Noise variance 2 dBmμ, τ Error tolerance 10−3Pmax The maximum of transmit power of MU 0.2Wqmax The maximum of transmit power of MTCD 0.08Wα Path loss factor 3Pc The circuit power at each sub-channel 0.1WRreq The data rate requirement of each MU 100 bpsAlgorithm 4 Sequential exhaustive algorithm.Initialization phase:ε > 0, ω =y n−y nε1: for y f n ∈[y f n : ω :y f n)do2: p∗f n = argmin ∑f∈F∑k∈S f npk f n + Pc3: end for4: Obtain the optimal solution of p f n.137Sensors 2019, 19, 5307??? ???? ???? ???? ???? ??? ???? ???? ???? ????????????????????????????? ?????????????????????????????????????????????????????? ?????????????????????? ??????????????????????????? ????????????????????????????????????????????????Figure 2. EE versus the maximum transmit power for different schemes.??? ??? ??? ??? ??? ??? ??? ??? ??? ?????????????????????????????????????????? ??????????????????????????????????????????????????????????????????????????????????????????????????????????Figure 3. EE versus the date rate requirement of each MU for different schemes.Figures 4 and 5 shows the convergence property of Algorithms 2 and 3. For simplicity, the numericalresults in two figures are from a random chosen sub-channel, where ηn(c) = 1. In Figure 4, we can see thatthe number of iterations are limited within four times. To show the influence of p(0)f n , we give the differentvalues of p(0)f n in Figure 5, where∣∣∣ fn(p f n(v))− ηn(c)gn(p f n(v)) − ( fn(p f n(v−1))− ηn(c)gn(p f n(v−1)))∣∣∣ =W(p f n(v)). It is shown that the initial values have an effect on the number of iterations. Specifically,when p f n(0) = 0× Pmax, less than 11 times is needed to reach the convergence. Although the initialvalues affect the number of iterations, it does not affect the final results.To show the relationship between the different numbers of MUs and MTCDs and the EE, we haveFigure 6. It is not surprise to see that the EE performance of all these schemes increases as Pmax grows.From the four schemes, we can ﬁnd out that the EE of K = 40 is much larger than that of K = 15, sincethe NOMA scheme can obtain much higher EE by supporting multiple MUs, and they can choose thesuitable couples by swap operations for better EE. From the Algorithm 1, we know that power control138Sensors 2019, 19, 5307algorithm needs to be executed after each swap operation, that is, the number of iterations increaseswith the increase of K and more process time are required. From the Figure 6, we can also see that theEE of Uk = 2 is larger than that of Uk = 3 under the same K, i.e., K has much greater impact on EEthan Uk, since the NOMA scheme can obtain much higher EE by supporting multiple MUs and theincreasing Uk represents the increase data rate requirement of MUs.Figure 7 presents the cumulative distribution function (CDF) of the number of swap operationsof different scenarios when the matching algorithms reached convergence. From the ﬁgure we cansee that more swap operations were needed for a larger number of MUs and sub-channels, such as,K = 40, N = 3 needed more swap operations than that K = 40, N = 2 and K = 15, N = 3. Especially,less than 70 swap operations were needed for K = 40 and N = 3.? ? ? ? ? ? ? ? ? ???????????????????????????????????????????????????????????????????????????????????????????????Figure 4. Convergence property of Algorithm 2.? ? ? ? ? ? ? ? ? ? ?? ????????????????????????????????????? ???????????????????????????????????????????????????????????????????????????????????????????????????????Figure 5. Convergence property of Algorithm 3.139Sensors 2019, 19, 5307??? ???? ???? ???? ???? ??? ???? ???? ???? ???????????????? ?????????????????????????????????????????????????????????????????????????????Figure 6. EE versus the maximum transmit power for the different number of MUs and MTCDs.? ?? ?? ?? ?? ?? ?? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????Figure 7. CDF of the number of swap operations for convergence.6. ConclusionsThis work investigated the uplink EE maximization problem in M2M-enabled HetNets withNOMA, where a MU acting as an MTCG can decode and forward both the information of MTCDsand its own data to the BS directly. Due to the limited spectrum resource, each BS shared the samesub-channels and NOMA was adopted between MUs in the same BS and sub-channel. The EEmaximization problem was formulated, where MU association and power control were combined witheach other. To solve it, aMU associationmatching algorithmwas proposed based on thematching game.Under a given MU association, the uplink EE maximization was transformed into the EE maximizationof each sub-channel. Two power control algorithms were provided to obtain the suboptimal powersolutions based on sequential optimization. Simulation results showed that our proposed algorithmsperformed better than EE performance. It is known that cellular network is a key way to connect theM2M communications to the core network; our proposed scheme provided a new strategy for MTCDsto connect the cellular network with regard to MUs as their MTCGs based NOMA, and the power140Sensors 2019, 19, 5307control of MTCDs was also considered as the constraints for the EE optimization problem. In fact, largescale devices are a more realistic scenario for 5G and next generation network, and since the numberof MTCDs is considered on a small scale in this paper, the extension of our algorithms for large scaledevices is one of the future works. Furthermore, the research of high computation complexity of theproposed algorithms for large scale devices is also a signiﬁcant problem.Author Contributions: The main contributions of S.Z. were to create the main ideas and execute performanceevaluation by extensive simulation, while G.K. worked as the advisor to discuss, create, and advise the main ideasand performance evaluations together. All authors read and approved the ﬁnal version of the paper.Funding: This work was supported by the National Science and Technology Major Project of China (No.2017ZX03001022) and National Natural Science Foundation of China (61471064).Conﬂicts of Interest: The authors declare no conﬂict of interest.References1. Ghosh, A.; Mangalvedhe, N.; Ratasuk, R.; Mondal, B.; Cudak, M.; Visotsky, E.; Thomas, T.A.; Andrews, J.G.;Xia, P.; Jo, H.S.; et al. Heterogeneous cellular networks: From theory to practice. IEEE Commun. Mag.2012, 6, 54–64. [CrossRef]2. Damnjanovic, A.; Montojo, J.; Wei, Y.; Ji, T.; Luo, T.; Vajapeyam, M.; Yoo, T.; Song, O.; Malladi, D. A surveyon 3GPP heterogeneous networks. IEEE Wirel. Commun. 2011, 3, 10–21. [CrossRef]3. Ericsson, A.B. 5G Radio Access, Research and Vision; Saito, Y., Benjebbour, A., Eds.; June 2013.4. Shin, W.; Vaezi, M.; Lee, B.; Love, D.J.; Lee, J.; Poor, H.V. Non-orthogonal multiple access in multi-cellnetworks: Theory, performance, and practical challenges. IEEE Commun. Mag. 2017, 10, 2–9. [CrossRef]5. Ding, Z.; Yang, Z.; Fan, P.; Poor, H.V. On the performance of non-orthogonal multiple access in 5G systemswith randomly deployed users. IEEE Signal Process. Lett. 2014, 12, 1501–1505. [CrossRef]6. Ding, Z.; Peng, M.; Poor, H.V. Cooperative non-orthogonal multiple access in 5G systems. IEEE Commun. Lett.2015, 8, 1462–1465. [CrossRef]7. Zhang, N.; Wang, J.; Kang, G.; Liu, Y. Uplink non-orthogonal multiple access in 5G systems.IEEE Commun. Lett. 2016, 3, 458–461. [CrossRef]8. Zhang, H.; Fang, F.; Cheng, J.; Long, K.; Wang, W.; Leung, V.C.M. Energy-efﬁcient resource allocation inNOMA heterogeneous networks. IEEE Wirel. Commun. 2018, 2, 48–53. [CrossRef]9. Mokdad, A.; Azmi, P.; Mokari, N. Radio resource allocation for heterogeneous trafﬁc in GFDM-NOMAheterogeneous cellular networks. IET Commun. 2016, 12, 1444–1455. [CrossRef]10. Zhao, J.; Liu, Y.; Chai, K.K.; Nallanathan, A.; Chen, Y.; Han, Z. Spectrum allocation and power control fornon-orthogonal multiple access in HetNets. IEEE Trans. Wirel. Commun. 2017, 9, 5825–5837. [CrossRef]11. Ni, D.; Hao, L.; Tran, Q.T.; Qian, X. Power allocation for downlink NOMA heterogeneous networks.IEEE Access 2018, 6, 26742–26752. [CrossRef]12. Ni, D.; Hao, L.; Tran, Q.T.; Qian, X. Transmit power minimization for downlink multi-cell multi-carrierNOMA networks. IEEE Commun. Lett. 2018, 12, 2459–2462. [CrossRef]13. Fang, F.; Cheng, J.; Ding, Z. Joint energy efﬁcient subchannel and power optimization for a downlink NOMAheterogeneous network. IEEE Trans. Veh. Technol. 2019, 68, 1351–1364. [CrossRef]14. Moltafet, M.; Azmi, P.; Mokari, N.; Javan, M.R.; Mokdad, A. Optimal and fair energy efficient resource allocationfor energy harvesting-enabled-PD-NOMA-based HetNets. IEEE Trans. Wirel. Commun. 2018, 3, 2054–2067.[CrossRef]15. Pentikousis, K. In search of energy-efﬁcient mobile networking. IEEE Commun. Mag. 2010, 1, 95–103.[CrossRef]16. IMT-2020(5G) Promotion Group. 5G Vision and Requirements; May 2014.17. Buzzi, S.; Chih-Lin, I.; Klein, T.E.; Poor, H.V.; Yang, C.; Zappone, A. A survey of energy-efﬁcient techniquesfor 5G networks and challenges ahead. IEEE J. Sel. Areas Commun. 2016, 4, 697–709. [CrossRef]18. Qin, Z.; Yue, X.; Liu, Y.; Ding, Z.; Nallanathan, A. User association and resource allocation in uniﬁed NOMAenabled heterogeneous ultra dense networks. IEEE Commun. Mag. 2018, 6, 86–92. [CrossRef]19. Yang, Z.; Xu, W.; Xu, H.; Shi, J.; Chen, M. Energy efﬁcient non-orthogonal multiple access for machine-to-machine communications. IEEE Commun. Lett. 2017, 4, 817–820. [CrossRef]141Sensors 2019, 19, 530720. Shirvanimoghaddam, M.M.; Dohler, M.; Johnson, S.J. Massive non-orthogonal multiple access for cellularIoT: potentials and limitations. IEEE Commun. Mag. 2017, 9, 55–61. [CrossRef]21. Chen, S.; Ma, R.; Chen, H.; Zhang, H.; Meng, W.; Liu, J. Machine-to-machine communications in ultra-densenetworks-a survey. IEEE Commun. Surv. Tutor. 2017, 3, 1478–1503. [CrossRef]22. Yang, Z.; Pan, Y.; Xu, W.; Guan, R.; Wang, Y.; Chen, M. Energy efﬁcient resource allocation for machine-to-machine communications with NOMA and energy harvesting. In Proceedings of the 2017 IEEE Conferenceon Computer Communications Workshops (INFOCOM WKSHPS), Atlanta, GA, USA, 1–4 May 2017; pp. 1–6.[CrossRef]23. Xu, B.; Chen, Y.; Carrion, J.R.; Zhang, T. Resource allocation in energy-cooperation enabled two-tier NOMAHetNets towards green 5G. IEEE J. Sel. Areas Commun. 2017, 12, 2758–2770. [CrossRef]24. Qian, L.P.; Wu, Y.; Zhou, H.; Shen, X.S. Joint uplink base station sssociation and power control for small-cellnetworks with non-orthogonal multiple access. IEEE Trans. Wirel. Commun. 2017, 19, 5567–5582. [CrossRef]25. Yang, Z.; Xu, W.; Pan, Y.; Pan, C.; Chen, M. Energy efﬁcient resource allocation in machine-to-machinecommunications with multiple access and energy harvesting for IoT. IEEE Internet Things J. 2018, 5, 229–245.[CrossRef]26. Li, Z.; Gui, J. Energy-efﬁcient resource allocation with hybrid TDMA–NOMA for cellular-enabledmachine-to-machine communications. IEEE Access 2019, 7, 105800–105815. [CrossRef]27. Lv, T.; Ma, Y.; Zeng, J.; Mathiopoulos, P.T. Millimeter-wave NOMA transmission in cellular M2Mcommunications for Internet of Things. IEEE Internet Things J. 2018, 5, 1989–2000. [CrossRef]28. Shahini, A.; Ansari, N. NOMA aided narrowband IoT for machine type communications with user clustering.IEEE Internet Things J. 2019, 6, 7183–7191. [CrossRef]29. Han, S.; Xu, X.; Tao, X.; Zhang, P. Joint power and sub-channel allocation for secure transmission inNOMA-based mMTC networks. IEEE Syst. J. 2019, 13, 2476–2487. [CrossRef]30. Mohamed, A.; Onireti, O.; Imran, M.A.; Imran, A.; Tafazolli, R. Control-data separation architecture forcellular radio access networks: A survey and outlook. IEEE Commun. Surv. Tutor. 2016, 18, 446–465.[CrossRef]31. Bodine-Baron, E.; Lee, C.; Chong, A.; Hassibi, B.; Wierman, A. Peer effects and stability in matching markets.In Proceedings of the 4th Symposium on Algorithmic Game Theory (SAGT), Amalﬁ, Italy, 17–19 October 2011;pp. 117–129. [CrossRef]32. Aydin, O.; Jorswieck, E.A.; Aziz, D.; Zappone, A. Energy-spectral efﬁciency tradeoffs in 5G multi-operatornetworks with heterogeneous constraints. IEEE Trans. Wirel. Commun. 2017, 9, 5869–5881. [CrossRef]33. Zappone, A.; Bjornson, E.; Sanguinetti, L.; Jorswieck, E. Globally optimal energy-efﬁcient power control andreceiver design in wireless networks. IEEE Trans. Signal Process. 2017, 11, 2844–2859. [CrossRef]34. Dinkelbach, W. On nonlinear fractional programming. Manag. Sci. 1967, 7, 492–498. [CrossRef]35. Ali, M.S.; Tabassum, H.; Hossain, E. Dynamic user clustering and power allocation for uplink and downlinknon-orthogonal multiple access (NOMA) systems. IEEE Access 2016, 4, 6325–6343. [CrossRef]36. Li, Y.; Sheng, M.; Yang, C.; Wang, X. Energy efﬁciency and spectral efﬁciency tradeoff in interference-limitedwireless networks. IEEE Commun. Lett. 2013, 10, 1924–1927. [CrossRef]37. Kha, H.H.; Tuan, H.D.; Nguyen, H.H. Fast global optimal power allocation in wireless networks by localD.C. programming. IEEE Trans. Wirel. Commun. 2012, 2, 510–515. [CrossRef]© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open accessarticle distributed under the terms and conditions of the Creative Commons Attribution(CC BY) license (http://creativecommons.org/licenses/by/4.0/).142sensorsArticleA Dynamic Access Probability Adjustment Strategyfor Coded Random Access SchemesJingyun Sun 1, Rongke Liu 1,* and Enrico Paolini 2,*1 School of Electronics and Information Engineering, Beihang University, 37 Xueyuan Road, Haidian District,Beijing 100191, China; sunjingyun@buaa.edu.cn2 Department of Electrical, Electronic, and Information Engineering, University of Bologna,via Dell’Universitá 50, 47522 Cesena (FC), Italy* Correspondence: rongke_liu@buaa.edu.cn (R.L.); e.paolini@unibo.it (E.P.);Tel.: +86-10-8233-9475 (R.L.); +39-0547-339137 (E.P.)Received: 13 August 2019; Accepted: 25 September 2019; Published: 27 September 2019Abstract: In this paper, a dynamic access probability adjustment strategy for coded random accessschemes based on successive interference cancellation (SIC) is proposed. The developed protocolconsists of judiciously tuning the access probability, therefore controlling the number of transmittingusers, in order to resolve medium access control (MAC) layer congestion states in high load conditions.The protocol is comprised of two steps: Estimation of the number of transmitting users during thecurrent MAC frame and adjustment of the access probability to the subsequent MAC frame, based onthe performed estimation. The estimation algorithm exploits a posteriori information, i.e., availableinformation at the end of the SIC process, in particular it relies on both the frame conﬁguration(residual number of collision slots) and the recovered users conﬁguration (vector of recovered users)to effectively reduce mean-square error (MSE). During the access probability adjustment phase,a target load threshold is employed, tailored to the packet loss rate in the ﬁnite frame length case.Simulation results revealed that the developed estimator was able to achieve remarkable performanceowing to the information gathered from the SIC procedure. It also illustrated how the proposeddynamic access probability strategy can resolve congestion states efﬁciently.Keywords: congestion; estimation; irregular repetition slotted ALOHA; medium access control;random access; successive interference cancellation1. IntroductionIn machine-type and Internet-of-Things (IoT) communications, users generate a large amount ofbursty trafﬁc to transmit over a shared communication medium. Coordinated multiple access schemesturn to be impractical and generally inefﬁcient in such scenarios. For this reason, random accessschemes have attracted a renewed interest, as they provide a practical way for uncoordinated users tocontend for channel resource.Pure ALOHA scheme [1] was proposed in 1968 to share a channel among a number of userssending packets as soon as they have data to transmit. Classical slotted ALOHA [2] is a distributedrandom access scheme in which time is divided into slots of equal duration with each transmissionstarting only at the beginning of a time slot. In both variants, an absence of coordination among usersmay lead to collisions (two or more packets are received in overlapping time windows). All packetsinvolved in a collision are often reported as useless and are retransmitted after a random delay,according to some probability distribution, or (in the framed case) in the next frame. As a result,pure ALOHA and slotted ALOHA suffer from a throughput penalty and an under-utilization ofchannel resource. The optimal normalized throughput of pure ALOHA is 0.18 and the throughput ofslotted ALOHA is increased to 0.37.Sensors 2019, 19, 4206; doi:10.3390/s19194206 www.mdpi.com/journal/sensors143Sensors 2019, 19, 4206The expression of “coded random access” refers to a set of random access schemes that combinethe packet repetition of users with successive interference cancellation (SIC) at the receiver. The ﬁrstcoded random access scheme is collision resolution diversity slotted ALOHA (CRDSA) [3], where eachuser sends two packet replicas in two random slots of the frame, and then SIC is applied to recover thecollided packets in an iterative fashion. After CRDSA, CRDSA++ [4] was proposed to further improvethroughput by increasing the number of packet replicas. In [5], where irregular repetition slottedALOHA (IRSA) was proposed, the SIC-based random access process is conveniently described by abipartite graph, establishing a bridge between the SIC procedure and the iterative erasure decodingof graph-based codes. In IRSA, the packet repetition rate is irregular from user to user and is chosenindependently by each active user according to a suitably designed probability distribution. Since then,coded random access emerged as a new paradigm and has been the subject of several investigationsover the past few years (e.g., [6–12] and references therein). As a result, the throughput has substantiallyincreased which makes it a practical and efﬁcient solution to support uncoordinated access.Despite their numerous advantages, coded random access schemes exhibit lower critical points intrafﬁc load. In other words, the throughput of these schemes is maximized for load values less than1 and, for larger values of the load, it decreases very rapidly. Congestion occurs when the numberof active users is greater than the receiver processing capacity. Several control methods for randomaccess schemes have been investigated, which may be classiﬁed into two kinds: Dynamic framelength based methods and dynamic access probability based methods. In dynamic framed slottedALOHA (DFSA) systems, the frame size is adjusted dynamically according to the estimated numberof active users in order to maximize the system efﬁciency [13–17]. In dynamic access probability basedschemes, on the other hand, an access controller is required to adjust the users access probability underhigh trafﬁc loads in order to limit the number of transmitting users [18–21]. However, in [18–20],the estimation process was simply based on the status of frame slots before the application of SICand in [21], the estimation is assumed to be ideal at the receiver. Furthermore, the proposed randomaccess control mechanisms in [18] are based on random access schemes without SIC at the receiver,which is not applicable for coded random access schemes. In both [19,20], users directly employ theload threshold from [5], which is obtained via asymptotic analysis (frame length and user populationsize tending to inﬁnity, their ratio remaining constant). When applied to the ﬁnite frame length case,asymptotic load thresholds tend to be beyond the actual critical point, which may yield considerablethroughput losses.In this paper, a dynamic access probability based strategy for coded random access schemesis proposed to resolve congestion. The proposed strategy performs two main tasks: Estimation ofnumber of transmitting users in the current frame and the adjustment of access probability in the nextframe based on the estimation results. In our previous work [22–24] techniques for a more reliableestimation of the number of transmitting users in coded random access schemes were developedand more specially, the number of transmitting users in the current frame was estimated using aposteriori information gathered throughout the SIC process. A posteriori estimation was consideredfor CRDSA in [22], for IRSA in [23] and for CRDSA over a packet and slot erasure channel in [24].Notably, [22–24] were entirely focused on the estimation process, without any attempt to exploit itwithin a dynamic access probability adjustment protocol. The usage of a target load threshold tailoredto the ﬁnite frame length case and the introduction of a state judgment to avoid not fully reliableestimation in high trafﬁc load conditions are other original features of this manuscript.The system model and some preliminary deﬁnitions are provided in Section 2. The estimationalgorithm for the number of transmitting users in the current frame is addressed in Section 3.1,while the access probability adjustment strategy is proposed in Section 3.2. Numerical results areillustrated in Section 4 and concluding remarks are given in Section 5.144Sensors 2019, 19, 42062. Preliminaries2.1. System ModelWe consider a scenario where multiple users contend for access to a single central receiver.The medium access control (MAC) layer is organized into frames and the random access scheme is aslotted one. We denote random variables by capital letters, while their realizations and deterministicquantities are denoted by lower case letters. The frame length is ﬁxed and divided into m time slotswith equal duration.Active users are the ones who have packets to transmit. Congestion occurs when the numberof active users is too large in comparison to the available resources (a more precise deﬁnition ofcongestion is given in Section 3.2). We use the subscript (k) to represent the index of the MAC frame.If there is no congestion or the congestion is resolved, the index (k) is re-initialized to (0) in the nextframe, otherwise, it keeps counting. As such, a frame index k ≥ 1 indicates that we are in the k-thframe of the current congestion event.User population size is npop. The number of active users is unknown to the receiver and ismodeled by a random variable Na; the number of active users at the beginning of frame k is N(k)a .No new user activates before the current congestion has been resolved. Denoting by Δ(k), the numberof users that are recovered while processing frame k, for k ≥ 1 we have:N(k)a = N(k−1)a − Δ(k−1). (1)Transmitting users are the ones who are allowed to transmit their packets in the frame. Let T(k)abe the number of transmitting users during frame k. Moreover, denote by p(k)ac the access probabilityof the active users during frame k. At the beginning of the k-th frame, each active user becomes atransmitting one with probability p(k)ac , independently of other active users. Hence, the conditionalexpected value of T(k)a is:E[T(k)a |n(k)a ] = n(k)a p(k)ac . (2)Each transmitting user is frame- and slot- synchronous and attempts at most one packet transmissionper frame.In every frame corresponding to k = 0, all active users transmit their packets to the receiver, i.e.,we have p(0)ac = 1 and t(0)a = n(0)a . The instantaneous channel load over frame k is deﬁned as:G(k) =t(k)am(3)and represents the average number of packet transmissions per slot. The throughput over frame k isdeﬁned as:T(k)h =t(k)am(1− PL) (4)representing the average number of successfully recovered packets per slot by the receiver. The quantityPL in Equation (4) is the packet loss rate over the frame, which is expressed as:PL = 1− δ(k)t(k)a. (5)145Sensors 2019, 19, 4206In IRSA, each transmitting user sends L packet replicas to slots picked uniformly at random.The number of replicas, named user degree, is a discrete random variable probability mass function(p.m.f.) {Λl}, where Λl = P(L = l) is the probability that a user generates l packet replicas. Userschoose their replica factor (i.e., user degree) L independently of each other, with no coordination,and the values of user degree are according to distribution {Λl}. We also represent {Λl} in polynomialform, as Λ(x) = ∑dmaxl=1 Λl xl , where dmax is the maximum number of packet replicas per user. Bothinformation about the transmitting user index (assuming users are indexed from 1 to npop) and pointersto the slots where the other replicas have been transmitted are included in the header of each packetreplica. CRDSA can be seen as IRSA with Λ(x) = xdmax .In this paper, a classical collision channel model is adopted. After packet replica transmissions,each slot takes one of the following three states: Empty slot (no packet replica transmitted in thatslot), singleton slot (only one packet replica transmitted in that slot), and collision slot (two or morethan two packet replicas transmitted in that slot). The receiver can always correctly classify the stateof each slot. Collision slots provide no information to the receiver about the number and content ofcollided packet replicas directly. However, as soon as the contribution of interference, generated bysome transmitting users on the slot, is canceled and only one packet replica is left in it, the slot statusis updated to singleton. Similarly, if all of the packet replicas transmitted in the slot are recoveredby the receiver, the slot status (singleton slot or collision) is updated to empty. Packet replicas fromsingleton slots are always correctly received, which means that packet losses may only be generatedby unresolved collisions.After transmissions, the pointers to twin replicas in the header of the packet enable SIC at thereceiver. At ﬁrst, the receiver stores the content of the frame. Then, the receiver performs iterativelythe following procedure, consisting of two subsequent steps:1. Pick out the singleton slots in the frame. For each singleton slot, extract the transmitting userindex, the content of the packet replica, and positions of other twin replicas. Identiﬁed users in thisstep become recovered users;2. For each user recovered at step 1, remove the user’s contribution of interference in the slotswhere the packet replicas have been transmitted. A new singleton slot will appear if, after interferencecancellation, they contain only one replica.The iterative SIC procedure terminates when all slots are empty ones, in which case SIC succeeds,or when no singleton slot can be found but collision slots still exist, in which case it fails. At the end ofthe SIC procedure, the residual number of empty slots in the frame is denoted by Me, and the residualnumber of collided slots per frame by Mc. Obviously, we have Me + Mc = m.Example 1. With reference to Figure 1, ta = 4 users transmit their packets to a frame with m = 5 slots. Useru1 generates three replicas of his packet, and sends them to s1, s3, and s4, respectively. Each of the other usersgenerate two replicas of the corresponding packets and transmit them as illustrated in the ﬁgure. At the receiver,slots s1 and s4 are singleton slots and the left s2, s3, and s5 are collison slots.Figure 2 provides a graphical interpretation (ﬁrst proposed in [5]) of the iterative SIC procedure performedon the frame of Figure 1. In the presented graph, “slot nodes" represent slots and “user nodes" represent users.In the ﬁrst SIC iteration, s1 and s4 are singleton slots and the corresponding packet replicas are correctly received,making u1 a recovered user. The pointer to slot s3, where the twin of the replica in s1 has been transmitted,is extracted (step 1). After the interference from recovered user u1 in slot s3 is canceled and only one packetreplica is left in s3, making s3 a new singleton slot (step 2). Then a second iteration is triggered. After three SICiterations, users u2 and u3 remain unrecovered, there are no singleton slots in the frame, and SIC terminateswith failure.The feedback frame conﬁguration signal is {0, 1, 0, 0, 1} which indicates that s1, s3 and s4 are empty slotsand that s2 and s5 are unresolved collision slots. Receiving this feedback signal, u2 and u3 become aware thattheir packets have not been successfully received.146Sensors 2019, 19, 4206u4u3u2u1s1 s2 s3 s4 s5u4,1 u4,2u3,1 u3,2u2,1 u2,2u1,1 u1,2 u1,3Figure 1. Example of a MAC frame with ta = 4 transmitting users and m = 5 slots. User u1 sends threepacket replicas and the other users each send two packet replicas. Slots s1 and s4 are singleton slotsand the left s2, s3, and s5 are collision slots.slot nodesuser nodesFigure 2. Example of successive interferece cancellation (SIC) procedure corresponding to Figure 1.Squares correspond to slots and circles correspond to users.2.2. Threshold Deﬁnition and NotationThroughout the paper we deﬁne a load threshold G◦ as the maximum load such that the packetloss rate falls below a given target value P◦L . In other words, when the instantaneous load G is belowG◦, we have PL ≤ P◦L , otherwise we have PL > P◦L .In Table 1, some examples of probability distributions Λ(x) are shown with the correspondingtarget load threshold values. The ﬁrst two rows in the table represent CRDSA schemes, where eachuser transmits the same number of replicas. The last two rows represent IRSA schemes, where thenumber of replicas per user is irregular. The values of G◦ have been obtained via a Monte Carlosimulation, for MAC frame length m = 200 and target packet loss rate P◦L = 0.01.Table 1. Load threshold G◦ for different probability distributions Λ(x), for MAC frame length m = 200,and packet loss rate target P◦L = 0.01.Distribution, Λ(x) G◦Λ1(x) = x2 0.35Λ2(x) = x4 0.69Λ3(x) = 0.5x2 + 0.28x3 + 0.22x8 0.705Λ4(x) = 0.25x2 + 0.6x3 + 0.15x8 0.76147Sensors 2019, 19, 4206Figure 3 shows the packet loss rate PL versus instantaneous load G for the distributions in Table 1and frame length m = 200. As previously remarked, the SIC process in IRSA can be described by abipartite graph, where unresolved collisions are associated with graphical structures known, in thelow-density parity-check (LDPC) coding jargon, as stopping sets. It is well known that the impactof small stopping sets on the ﬁnite-length performance is strictly related to the fraction of degree-2variable nodes in its bipartite graph and a similar role is played by degree-2 users in IRSA. As observedin the ﬁgure, the limitation of degree-2 repetition has a better error ﬂoor performance, but a poorerwaterfall performance. The detailed packet loss rate performance analysis for IRSA schemes have beenaddressed in [5].0 0.2 0.4 0.6 0.8 110−410−310−210−1100GPLΛ1Λ2Λ3Λ4Figure 3. Packet loss rate PL versus instantaneous load G for frame length m = 200 and the distributionsin Table 1.2.3. Combinatorial ParametersWe denote by |v| = ∑ni=1 |vi| the 1 norm of a real-valued vector v = (v1, . . . , vn). Moreover,given a second vector w = (w1, . . . ,wn) whose elements are nonnegative integers, we use the compactnotation vw for vw11 · · · vwnn .Leto = (o1, . . . , odmax) be a vector whose elements are all nonnegative integers. Let M(o, b) bethe set of all |o| × b binary matrices M, with rows and columns indexed from 1 to |o| and from 1 to b,respectively, that fulﬁll the following properties: 1. The matrix M has the structure:M = [MT1 MT2 · · · MTdmax ]Twhere Mi has dimension oi × b and all of its rows have Hamming weight i. 2. Every column of M hasHamming weight at least 2.Example 2. Leto = (o1, o2, o3) = (0, 1, 3) and b = 5. Each matrix in M ∈ M(o, b) has dimension 4× 5.Its row indexes should be thought as partitioned into the two subsets {1} and {2, 3, 4}. The row of index 1has weight 2, and the rows of indexes 2, 3, and 4 have weight 3. Every column of M has weight of at least 2.An example of matrix M ∈ M(o, b) is:148Sensors 2019, 19, 4206M =⎛⎜⎜⎜⎝1 0 0 1 00 1 0 1 11 1 1 0 00 0 1 1 1⎞⎟⎟⎟⎠ .The following lemma provides a formal expression for the cardinality of the set M(o, b).Lemma 1. For given o and b, let h(o, b) be the cardinality of the set M(o, b). Moreover, let x =(x1, x2, . . . , x|o|) and:q = (1, . . . , 1︸︷︷︸o1, 2, . . . , 2︸︷︷︸o2, . . . , dmax, . . . , dmax︸︷︷︸odmax). (6)Deﬁne the multivariate polynomials A(x) and Bj,l(x) as:A(x) =|o|∏i=1(1+ xi)−(1+|o|∑i=1xi)(7)and:Bj,l(x) =( |o|∑i=1xi)l( |o|∏i=1(1+ xi))j. (8)Then, we have:h(o, b) = coeff((A(x))b,xq) (9)=b∑j=0b−j∑l=0(bj)(b− jl)(−1)b−jcoeff(Bj,l(x),xq) (10)where coeff(P(x),xr) is the coefﬁcient of xr in the multivariate polynomial P(x).Proof. LetcT = (c1, . . . , c|o|)T be the generic column and deﬁne a multivariate enumerating functionfor valid columns (i.e., columns with weight of at least 2):A(x) = ∑c:|c|≥2xc. (11)It is easy to recognize that an equivalent expression for A(x) is the one shown in Equation (7).This is because (1+ x1) · · · (1+ x|o|) provides the sum of all monomials in the variables x1, . . . x|o|with a unitary coefﬁcient, to which we subtract all monomials of degrees 0 and 1 as required by thecondition of validity.Considering now b columns and applying properties of generating functions, coeff((A(x))b,x w)is the number of |o| × b binary matrices such that all matrix columns are valid and such thatthe weight of row i is wi. This immediately leads to Equation (9). The equivalent expression ofEquation (10) is obtained by simple algebraic manipulation of the multivariate polynomial (A(x))b.149Sensors 2019, 19, 4206In particular, it is obtained by applying Newton’s binomial formula twice and by exploiting the identitycoeff(∑i αiPi(x),x w) = ∑i αicoeff(Pi(x),x w).3. Dynamic Access Probability AlgorithmIn this section we introduce the proposed multiple access strategy based on a dynamic adjustmentof the users access probability. Section 3.1 addresses estimation of the number of transmitting users;Section 3.2 exploits the developed estimator to perform congestion detection and resolution viadynamic access probability adjustment.3.1. Number of Transmitting Users EstimationIn this subsection, we exploit frame conﬁguration information at the end of SIC to estimatethe number t(k)a of transmitting users in the k-th frame, when an SIC failure occurs. For the sake ofnotational simplicity, the superscript (k) is temporarily omitted.The total number of transmitting users is denoted by ta. We also denote by ta,l the numberof such users that employ the replica factor l. Clearly, we have ta = ∑dmaxl=1 ta,l . The vectorta = (ta,1, ta,2, . . . , ta,dmax) is referred to as transmitting users conﬁguration at the beginning of theframe. The number of transmitted users that are recovered at the end of the SIC process is denoted byδ ≤ ta. Out of these δ recovered users, δl ≤ ta,l are the ones using replica factor l, so that δ = ∑dmaxl=1 δl .The vector δ = (δ1, δ2, . . . , δdmax) is referred to as the recovered users conﬁguration at the end of SIC.Hereafter we develop a compact expression for the a posteriori probability distribution ofthe conﬁgurationta of transmitting users, given the number mc of residual collision slots and theconﬁgurationδ of recovered users observed at the end of SIC. This probability is denoted by P(ta|mc,δ).Note that, as transmitting users pick their slots uniformly at random, it is sufﬁcient to condition tothe number of collision slots (and not to their positions in the frame). The corresponding probabilitydistribution of the number ta of transmitting users is given by:P(ta|mc,δ) = ∑ta :|ta |=taP(ta|mc,δ). (12)A maximum a posteriori (MAP) estimator for the number of transmitting users then returnsthe value:tˆa = argmaxtaP(ta|mc,δ). (13)Theorem 1. The a posteriori probability distribution of the conﬁgurationta of the transmitting users fulﬁlls:P(ta|mc,δ) ∝(taδ)h(ta −δ,mc)∏dmaxl=1 (ml )ta,lP(ta) (14)where h(o, b) is given by Lemma 1, (taδ) = ∏l (ta,lδl), and P(ta) is the a priori probability that the transmittingusers conﬁguration equalsta.Proof. From Bayes’ rule we have:P(ta|mc,δ) = P(mc,δ|ta)P(ta)P(mc,δ)∝ P(mc,δ|ta)P(ta). (15)150Sensors 2019, 19, 4206Let T(ta,mc,δ) be the number of ways in which |ta| transmitting users with conﬁguration ta cantransmit their packet replicas in the frame so that, at the end of SIC, there are mc unresolved collisionslots and a recovered users conﬁguration δ. Moreover, let T(ta) be the number of ways in which |ta|transmitting users with conﬁgurationta can place their packet replicas in the frame. The conditionalprobability P(mc,δ|ta) can be expressed as:P(mc,δ|ta) = T(ta,mc,δ)T(ta). (16)The quantity T(ta) is readility shown to be given by:T(ta) =dmax∏l=1(ml)ta,l. (17)To develop an expression for T(ta,mc,δ), we proceed as follows. At the end of SIC, |ta −δ| transmittingusers with conﬁguration ta −δ remain unrecovered. The number of ways in which these users transmittheir packet replicas to mc slots, forming mc collisions (at least two replicas per slot) is h(ta −δ,mc).If we let g(δ,mc) be the number of ways in which |δ| transmitting users with conﬁguration δ can placetheir packet replicas in a frame with m − mc free slots and mc unresolvable collision slots, so thatSIC can recover all of them, we can write (no formal expression for g(δ,mc) is provided because thisparameter, not depending onta does not play any role in the estimation process of Equation(13)):T(ta,mc,δ) =(taδ)(mmc)h(ta −δ,mc)g(δ,mc). (18)Incorporating Equation (17) and Equation (18) into Equation (16) and then Equation (16) intoEquation (15), and omitting all terms not depending on ta, we obtain Equation (14).Although Equations (13) and (14) deﬁne an exact MAP estimator, computing h(ta −δ,mc) turnsout to be a complex task, becoming already intractable for frame sizes in the order of a few tens.For this reason we employ an approximated MAP estimator. In the approximation, all packet replicas,even from the same user, are regarded as distinguishable packets. Equivalently, each user chooses lslots with replacement. In this approximate setting, we haveP(ta|mc,δ) ∝(taδ)h((∑dmaxl=1 (tl − δl)l),mc)m∑dmaxl=1 tl lP(ta) (19)where (∑dmaxl=1 (tl − δl)l) represents a vector with only one element, corresponding too = (o1) in h(o, b).The value of h((∑dmaxl=1 (tl − δl)l),mc) is the number of ways in which ∑dmaxl=1 (tl − δl)l packet replicas aresent to mc slots, such that each slot receives not less than 2 packet replicas [25]; m∑dmaxl=1 tl l is the totalnumber of ways in which ∑dmaxl=1 tl l packet replicas can be accommodated into the m slots.As an estimator performance measure we consider the MSE, deﬁned as:M = E[2] (20)where = tˆ(k)a − t(k)a is the estimation error.151Sensors 2019, 19, 42063.2. Access Probability Adjustment StrategyWe say that we have a congestion on frame k whenevern(k)a > G◦m, (21)where we recall that n(k)a is the number users that are active on frame k. Our purpose is to exploit thedeveloped estimator to detect congestion states and dynamically adjust the users access probability toimprove overall efﬁciency. Congestion states are resolved by tuning the access probability to controlthe number of transmitting users in the next frame.The proposed scheme is based on the deﬁnition of three possible states for a frame, namely:• Not fully reliable estimate. In high load conditions, SIC typically stops prematurely with arelatively small number of recovered users. We say that the estimate tˆ(k)a is not fully reliable whenthe number of users recovered by processing the frame is smaller than the number of users thatcould not be recovered:δ(k) < t(k)a − δ(k) (22)or, equivalently,δ(k) < t(k)a /2. (23)• Congestion with reliable estimate. The number of active users is above threshold G◦m, but thenumber of users recovered by processing the frame is not less than the number of users that couldnot be recovered:n(k)a > G◦m and δ(k) ≥ t(k)a − δ(k). (24)• No congestion. The number of active users is below threshold G◦m:n(k)a ≤ G◦m. (25)In the ﬁrst case, a large number of transmitting users is unrecovered, and the packet loss rate islarger than 0.5. As illustrated in the numerical results section, the estimation MSE increases with thenumber of transmitting users and the estimate is therefore regarded as not suitable to design the accessprobability pac in the subsequent frame. In contrast, in the last two cases the access probability in thenext frame is calculated directly by employing the estimate of the number of transmitting users.In the generic frame k, after all transmitting users have performed the transmission of theirpacket replicas, the receiver performs the SIC procedure. At the end of SIC, the receiver executes theprocedure described in Algorithm 1. This procedure is executed regardless of the SIC terminationstatus (success or failure). An explanation of Algorithm 1 is provided in the following.152Sensors 2019, 19, 4206Algorithm 1: Receiver procedure1 if m(k)c > 0&&P(t(k)a >2δ(k) |m(k)c ,δ (k))P(t(k)a ≤2δ(k) |m(k)c ,δ (k))> 1 then2 p(k+1)ac ← Gomnpop−∑ki=0 δ(i) ;3 k ← k + 1;4 else5 if m(k)c == 0 then6 tˆ(k)a = δ(k);7 else8 calculate tˆ(k)a according to Equation (19)9 end10 nˆ(k)a = tˆ(k)a /p(k)ac ;11 if nˆ(k)a > G◦m then12 p(k+1)ac ← Gomnˆ(k)a − δ(k);13 k ← k + 1;14 else15 p(1)ac ← 1;16 k ← 0;17 end18 end19 broadcast k, p(k)ac , and C(k) to the users;The ﬁrst step (line 1) consists of detecting whether Equation (23) is fulﬁlled or not. When SICsucceeds (m(k)c = 0), the estimation is perfect. The algorithm jumps to line 6 and simply sets tˆ(k)a = δ(k).In case of an SIC failure (m(k)c > 0), the algorithm applies a two-hypotheses MAP detector, whosedevelopment is presented in Appendix A, to decide whether Equation (23) holds (in which caseestimation is considered unreliable) or not. Concretely, ifP(t(k)a > 2δ(k)|m(k)c ,δ (k))P(t(k)a ≤ 2δ(k)|m(k)c ,δ (k))> 1, (26)then Equation (23) is assumed to hold and the estimation Equation (19) is regarded as not reliableenough. Otherwise, the estimate tˆ(k)a is employed to design the access probability in the next frame.When Equation (26) is satisﬁed, a ’not fully reliable estimate’ state is detected and the numberof transmitting users is detected to be large enough to create a congestion but the relatively largeestimation MSE prevents from relying on tˆ (k)a to reliably adjust the access probability in the next frame.At the end of frame k a number ∑ki=0 δ(i) of active users have been recovered since the beginning ofcongestion. Therefore, at the beginning of the subsequent frame, the number of unresolved activeusers fulﬁlls n(k+1)a ≤ npop − ∑ki=0 δ(i). To make the expected number of transmitting users in thesubsequent frame below the target number G◦m, we set the access probability according to (line 2)p(k+1)ac =G◦mnpop −∑ki=0 δ(i). (27)This way, the conditional expected number of transmitting users in the next frame is153Sensors 2019, 19, 4206E[T(k+1)a |n(k+1)a ] = n(k+1)a p(k+1)ac=n(0)a −∑ki=0 δ(i)npop −∑ki=0 δ(i)G◦m, (28)where n(0)a −∑ki=1 δ(i) represents the actual number of unrecovered active users at the beginning offrame k + 1.When the estimation result is detected to be reliable, an acceptable estimation MSE is assumedby the receiver, which exploits tˆ(k)a (equal to δ(k) in case of a SIC success or by Equation (19) in caseof a failure) to obtain an estimate of the number of active users on frame k. Speciﬁcally, the receiverperforms (line 10):nˆ(k)a = tˆ(k)a /p(k)ac . (29)The estimate nˆ(k)a is compared with the threshold G◦m (line 11). If nˆ(k)a > G◦m the receiverdeclares a congestion with a reliable estimate state. The system is suffering from congestion, but mostof (or all of) transmitting users have been recovered by SIC. The relatively low estimation MSE allowsconﬁdently using nˆ(k)a to set the access probability for the next frame. If the access probability is keptunchanged in the subsequent frames, the number of transmitting users will deviate progressively fromthe target G◦m, leading to a low throughput. To make efﬁcient use of channel resources, we increasethe access probability in such a way as to maintain the number of transmitting users close to thetarget G◦m in the next frame. From Equation (1), the estimated number of unrecovered active usersat the beginning of the frame k + 1 is nˆ(k+1)a = nˆ(k)a − δ(k). The target conditional expected number oftransmitting users in frame k + 1 isE[T(k+1)a |n(k+1)a ] = G◦m. (30)Thus, the access probability over frame k + 1 is set to (line 12):p(k+1)ac =G◦mnˆ(k)a − δ(k). (31)If nˆ (k)a < G◦m, a no congestion state is detected. The frame index k is re-initialized to 0 and theusers access probability is set to be 1 (lines 15 and 16).As a last step (line 19), the receiver broadcasts to the users the index of the next frame (index ofthe current frame increased by 1 if a congestion is detected and 0 otherwise), the access probability tobe employed by active users in the next frame, and the list of indexes of collision slots at the end ofSIC in the current frame. Upon receiving feedback from the receiver, users behave as follows:• If k > 0 (congestion), in the next frame each backlogged user attempts access to the frame withprobability equal to the new access probability. Each non-backlogged user is prevented fromtransmitting new packets;• If k = 0 (no congestion), users that are in a backlog state retransmit their packet. Users that arenot backlogged take their normal access activity.Users are updated by the receiver about congestion or no congestion simply through the index k.Moreover, each of them knows whether or not it is in a backlog state simply by looking at the list ofcollision slot indexes C(k). Note that, if k = 0 (no congestion) is broadcasted by the receiver, this doesnot necessarily mean that SIC has succeeded as there may be a small number of users unrecovered154Sensors 2019, 19, 4206even though the system is not suffering from congestion. In this case, we simply let backlogged usersretransmit packet replicas with probability 1 in the subsequent frame, together with possible freshreplicas from newly activated users.4. Numerical ResultsThis section is organized into two subsections. In Section 4.1 we show results on the estimation ofthe number of transmitting users, while in Section 4.2 we address the performance achieved by theproposed scheme.4.1. Estimation of Transmitting UsersIn this section, we present Monte Carlo simulation results using the approximated estimatordiscussed at the end of Section 3.1. Let the frame length be m = 200 and the user population size benpop = 400. Users are assumed to activate independently of each other at the beginning of every newframe. In each run, after users transmissions, SIC is applied at the receiver and then the developedapproximated estimator is applied.Figure 4 shows the average throughput and throughput standard deviation versus theinstantaneous load G for IRSA with Λ(x) = 0.5x2 + 0.28x3 + 0.22x8 [5]. The maximum averagethroughput is achieved at a value of G that is approximately equal to 0.8. However, the realizationsof the per-frame throughput ﬂuctuate around its statistical mean, the throughput standard deviationrepresenting a reliable measure of the bobbing range (i.e., dispersion). A large standard deviationmakes the average throughput a not fully meaningful parameter since, due to the per-frame throughputﬂuctuations, we have a higher probability that the system falls into a not fully reliable estimate state.In this respect, the peak average throughput is not necessarily a good working point, as the statisticalmean alone is not able to capture the probability of falling into such an “outage” state.0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1G00.20.40.60.81Average Throughput00.20.4Throughput Standard DeviationFigure 4. Average throughout and throughput standard deviation versus the instantaneous load G forIRSA with Λ(x) = 0.5x2 + 0.28x3 + 0.22x8.Figure 5 shows the estimation performance after SIC iterations, letting the SIC-based receiver rununtil no active user is recovered. As a comparison, we also consider the estimation performance usingthe frame conﬁguration before SIC iterations, which is reviewed in Appendix B. In the ﬁgure, the solidline is relevant to the proposed estimation making use of the frame conﬁguration and recovered usersconﬁguration after SIC iterations. Moreover, a dashed line corresponds to the estimation based onthe initially received frame, before SIC is applied. As observed in the ﬁgure, the proposed estimation155Sensors 2019, 19, 4206algorithm is able to reduce the MSE effectively over the whole range of G values. It is also worthnoting that the performance of the proposed estimator relies on the SIC performance. In low loadconditions, the SIC procedure stops with a large number of users recovered, so in this region theproposed estimation algorithm is more effective. In contrast, in high load conditions SIC almost alwaysstops prematurely, recovering a small number of users, leading the proposed estimation algorithm tobe less effectively.0 0.2 0.4 0.6 0.8 110−210−1100101102GMεBefore SICAfter SICFigure 5. Mean squared error (MSE) versus G for IRSA with Λ(x) = 0.5x2 + 0.28x3 + 0.22x8.4.2. Dynamic Access Probability Simulation ResultsIn this subsection, we present the simulation results for dynamic access probability based codedrandom access schemes using the mentioned estimation methods. The frame length is m = 200and the user population size is npop = 2000. Moreover, the considered IRSA distribution isΛ(x) = 0.5x2 + 0.28x3 + 0.22x8. Each non-backlogged user activates, independently of the otherusers, with probability π = 0.8 at the beginning of every frame with k = 0. At the ﬁrst frame, there areno backlogged users. The target trafﬁc threshold G◦ is set to 0.65, 0.705, 0.80, and 0.938 respectively,of which G◦ = 0.708 is associated with P◦L = 0.01 and G◦ = 0.938 is the asymptotic threshold of theconsidered IRSA distribution [5]. The initial access probability is p(0)ac = 1. We analyzed the systemperformance, during congestion resolution periods, through numerical simulations. Every simulationconsisted of a sufﬁciently large number of runs and, in each run, the simulation was stopped when thecongestion was resolved.As a benchmark, consider transmission without any dynamic access probability adjustmentprocess. The expected number of active users (transmitting users) in the initial frame is 1600.The average repetition rate is 3.6, corresponding to an expected number of 6480 packet replicastransmitted over the 200 slots. At the receiver, we have a vanishing probability to ﬁnd singleton slotscapable of triggering the SIC process. Without dynamic access probability adjustment, the packet lossrate becomes very close to 1 and the throughput very close to 0, meaning that almost no users arerecovered in the subsequent frames, making system congestion unresolvable.Figures 6–8 show that the proposed access probability algorithm works well to resolve congestion.The users access probability is adjusted dynamically to track the number of active users. At frame1, the access probability is decreased quickly to avoid working in the high load region. In this way,the estimator can provide a reliable estimate at the end of the frame and the receiver is able to performan accurate access probability design for the users in the next frame. Then the access probability isadjusted dynamically to make the number of transmitting users around the target G◦m. It is increased156Sensors 2019, 19, 4206slowly as some users are recovered by the receiver in each transmission. Each curve is plotted up tothe maximum value of k for which congestions remain unresolved, which is different for the differentchoices of the target load threshold.0 5 10 15 20 25 30k00.10.20.30.40.50.60.70.80.91p acGo=0.65Go=0.705Go=0.80Go=0.938Figure 6. Access probability pac versus frame index k for IRSA with Λ(x) = 0.5x2 + 0.28x3 + 0.22x8.0 5 10 15 20 25 30k00.10.20.30.40.50.60.7T hGo=0.65Go=0.705Go=0.80Go=0.938Figure 7. Throughput performance Th versus frame index k for IRSA with Λ(x) = 0.5x2 + 0.28x3+ 0.22x8.0 5 10 15 20 25k10-410-310-210-1100P LGo=0.65Go=0.705Go=0.80Go=0.938Figure 8. Packet loss rate PL versus frame index k for IRSA with Λ(x) = 0.5x2 + 0.28x3 + 0.22x8.Back to Figure 3, we have seen that the IRSA scheme tends to show a packet loss rate ﬂoor atlow offered trafﬁc regimes, the ﬂoor appearing around PL = 10−2 (corresponding to G = 0.705) for157Sensors 2019, 19, 4206Λ(x) = 0.5x2 + 0.28x3 + 0.22x8. For larger values of G (corresponding to the waterfall packet loss rateregion), the packet loss rate increases rapidly: A PL 0.08 is achieved at G = 0.80 and a PL 0.5is achieved at G = 0.938. Consequently, in Figure 8, systems with target G◦ = 0.65 and G◦ = 0.705have a similar packet loss rate performance, and they perform better than those with target G◦ = 0.80and G◦ = 0.938. Furthermore, due to estimation errors and to ﬂuctuations of the actual number oftransmitting users, we observe a minor packet loss rate deviation between Figure 8 and Figure 3.For example, in Figure 8, the packet loss rate with target G◦ = 0.80 is around 0.2, while in Figure 3,the packet loss rate at G◦ = 0.80 is approximately equal to 0.008.As a ﬁnal remark, recall that the throughput is deﬁned as G(1 − PL). The inﬂuence of PL atG ≤ 0.705 is small, so that the per-frame throughput T(k)h is approximately equal to the instantaneousload G. That is why in Figure 7, the throughput performance with target G◦ = 0.705 is better than thatwith target G◦ = 0.65. However, for the cases G◦ = 0.8 and G◦ = 0.938, the inﬂuence of PL can not beignored any more. The system performance is worse even though the load target G◦ is higher, sincethe packet loss rate is now considerably higher.5. ConclusionIn this paper, we proposed a technique to estimate the number of transmitting users in eachframe of an IRSA-based coded random access system. The estimated number of transmitting usersin the current frame was exploited to adjust the users access probability in the next frame. Frameconﬁguration information as well as recovered users conﬁguration information at the end of theSIC procedure were employed to make the estimation more accurate. Numerical results revealedhow the derived dynamic access probability strategy could resolve congestion efﬁciently, with astable throughput and a target packet loss rate performance for a proper choice of the parameterG◦. Interesting directions of investigation include the exact efﬁcient evaluation of the h(o, b) function(addressed in Lemma 1), to make the optimum estimator applicable to large communication networks.Adjusting the frame length dynamically in situations of slowly varying trafﬁc load over a large scaleis another direction of investigation that, to the best of the authors’ knowledge, has not been so faraddressed in the coded random access context.Author Contributions: All the authors contributed extensively to the work presented in this paper and to writingthe paper itself. J.S. and E.P. conceived the idea and developed the proposed approaches. E.P. gave adviceon the research and helped in editing the paper. E.P. and R.L. improved the quality of the manuscript andcompleted revision.Funding: This research was funded by the China Scholarship Council (grant No. 201706020024).Conﬂicts of Interest: The authors declare no conﬂict of interest.Appendix A. Justiﬁcation and Implementation of Equation (26)This appendix justiﬁes and addresses the implementation of the two-hypotheses MAP detectionrule Equation (26). Let the two hypotheses be H0 and H1. Moreover, let r represent the observation.The optimum detection rule consists of making the decision Hˆ0 when P(H0|r) > P(H1|r) and ofmaking the decision Hˆ1 otherwise.In our case, H0 corresponds to a ’not fully reliable estimate’ state (satisﬁed Equation (23)); H1corresponds to a ’reliable estimate’ state (Equation (23) not holding). The observation is (m(k)c ,δ (k)).Hence, we have:P(t(k)a > 2δ(k)|m(k)c ,δ (k))P(t(k)a ≤ 2δ(k)|m(k)c ,δ (k))Hˆ0≷Hˆ11, (A1)158Sensors 2019, 19, 4206where P(t(k)a > 2δ(k)|m(k)c ,δ (k)) = ∑t (k)a : t (k)a >2δ(k) P(t(k)a |m(k)c ,δ (k)) and P(t(k)a ≤ 2δ(k)|m(k)c ,δ (k)) =∑t(k)a : t(k)a ≤2δ(k)P(t(k)a |m(k)c ,δ (k)). The probability P(t(k)a |m(k)c ,δ (k)) can be expressed as:P(t(k)a |m(k)c ,δ (k)) = ∑ta :|t (k)a |=t(k)aP(t (k)a |mc,δ (k)), (A2)where P(t (k)a |mc,δ (k)) comes from the estimator Equation (14).Appendix B. Estimation Using Collision Slots before SICThe user’s repetition rate is Λ(x). Deﬁne Λ′(1) as the average user repetition rate given byΛ′(1) = ∑l lΛl . It is easy to verify that the probability that a generic user sends a packet replica withina given slot is Λ′(1)/m. As the users send packet replicas randomly, the slot degree distribution isbinomially distributed. The probability that a slot has l collided users is given by:Ψl =(tal)(Λ′(1)m)l (1− Λ′(1)m)ta−l. (A3)Before SIC iterations, the probability pe that a given slot is empty, the probability ps that agiven slot is singleton and the probability pc that a given slot is a collision one can be expressedrespectively as:pe =(1− Λ′(1)m)ta, (A4)ps = taΛ′(1)m(1− Λ′(1)m)ta−1(A5)and:pc = 1− pe − ps. (A6)An estimation for Ta using frame conﬁguration before SIC performs:tˆa = argmaxtaP(ta|wc), (A7)where wc is the number of collision slots before SIC iterations. Following Bayes’ rule, P(ta|wc) may bedeveloped as:P(ta|wc) ∝ P(wc|ta)P(ta)∝ pcwc(1− pc)m−wc . (A8)References1. Abramson, N. The ALOHA system: Another alternative for computer communications. In Proceedings ofthe Fall Joint Computer Confonference, New York, NY, USA, 17–19 November 1970; pp. 281–285.2. Roberts, L.G. ALOHA packet systems with and without slots and capture. ACM SIGCOM Comput.Commun. Rev. 1975, 5, 28–42. [CrossRef]3. Casini, E.; De Gaudenzi, R.; Herrero, O.D.R. Contention resolution diversity slotted ALOHA (CRDSA):An enhanced random access scheme for satellite access packet networks. IEEE Trans. Wirel. Commun. 2007, 6,1408–1419. [CrossRef]159Sensors 2019, 19, 42064. De Gaudenzi, R.; Herrero, O.D.R. Advances in random access protocols for satellite networks. In Proceedingsof the 2009 International Workshop on Satellite and Space Communications, Tuscany, Italy, 9–11 September2009; pp. 331–336.5. Liva, G. Graph-based analysis and optimization of contention resolution diversity slotted ALOHA.IEEE Trans. Commun. 2011, 59, 477–487. [CrossRef]6. Amat, A.G.; Liva, G. Finite Length Analysis of Irregular Repetition Slotted ALOHA in the Waterfall Region.IEEE Commun. Lett. 2018, 22, 886–889. [CrossRef]7. De Gaudenzi, R.; Herrero, O.D.R.; Gallinaro, G.; Cioni, S.; Arapoglou, P.D. Random access schemes forsatellite networks, from VSAT to M2M: A survey. Int. J. Satellite Commun. Netw. 2018, 36, 66–107. [CrossRef]8. Alvi, S.; Durrani, S.; Zhou, X. Enhancing CRDSA with Transmit Power Diversity for Machine-TypeCommunication. IEEE Trans. On Vehic. Tech. 2018, 67, 7790–7794. [CrossRef]9. Paolini, E.; Liva, G.; Chiani, M. Coded slotted ALOHA: A graph-based method for uncoordinated multipleaccess. IEEE Trans. Inf. Theory 2015, 61, 6815–6832. [CrossRef]10. Paolini, E.; Stefanovic´, Cˇ.; Liva, G.; Popovski, P. Coded random access: Applying codes on graphs to designrandom access protocols. IEEE Commun. Mag. 2015, 53, 144–150. [CrossRef]11. Taghavi, A.; Vem, A.; Chamberland, J.-F.; Narayanan, K.R. On the design of universal schemes for massiveuncoordinated multiple access. In Proceedings of the 2016 IEEE International Symposium on InformationTheory (ISIT), Barcelona, Spain, 10–15 July 2016; pp. 345–349.12. Sandgren, E.; i Amat, A.G.; Brännström, F. On frame asynchronous coded slotted ALOHA: Asymptotic,ﬁnite length, and delay analysis. IEEE Trans. Commun. 2017, 65, 691–704. [CrossRef]13. Schoute, F. Dynamic frame length ALOHA. IEEE Trans. Commun. 1983, 31, 565–568. [CrossRef]14. Chen, W.-T. An accurate tag estimate method for improving the performance of an RFID anticollisionalgorithm based on dynamic frame length ALOHA. IEEE Trans. Autom. Sci. Eng. 2009, 6, 9–15. [CrossRef]15. Eom, J.-B.; Lee, T.-J. Accurate tag estimation for dynamic framed-slotted ALOHA in RFID systems.IEEE Commun. Lett. 2010, 14, 60–62. [CrossRef]16. Wu, H.; Zeng, Y. Bayesian tag estimate and optimal frame length for anti-collision ALOHA RFID system.IEEE Trans. Autom. Sci. Eng. 2010, 7, 963–969. [CrossRef]17. Zanella, A. Estimating collision set size in framed slotted ALOHA wireless networks and RFID systems.IEEE Commun. Lett. 2012, 16, 300–303. [CrossRef]18. Rivero Angeles, M.E.; Lara Rodriguez, D.; Cruz-Perez, F.A. Random-access control mechanisms usingadaptive trafﬁc load in ALOHA and CSMA strategies for EDGE. IEEE Trans. Vehic. Tech. 2005, 54, 1160–1186.[CrossRef]19. Lee, M.W.; Lee, J.K.; Lim, J.S. R-CRDSA: Reservation-Contention Resolution Diversity Slotted ALOHA forSatellite Networks. IEEE Commun. Lett. 2012, 16, 1576–1579. [CrossRef]20. Noh, H.J.; Lee, J.K.; Lim, J.S. Performance evaluation of access control for CRDSA and R-CRDSA under hightrafﬁc load. In Proceedings of the 2013 IEEE Military Communications Conference, San Diego, CA, USA,18–20 November 2013; pp. 1365–1370.21. Sun, J.; Liu, R. Irregular Repetition Slotted ALOHA with Priority (P-IRSA). In Proceedings of the 2016 IEEEVehicular Technology Conference (VTC Spring), Nanjing, China, 15–18 May 2016; pp. 1–5.22. Sun, J.; Liu, R.; Paolini, E. Detecting the Number of Active Users in Coded Random Access Systems.In Proceedings of the 2018 IEEE 29th Annual International Symposium on Personal, Indoor and MobileRadio Communications (PIMRC), Bologna, Italy, 9–12 September 2018; pp. 1–7.23. Sun, J.; Liu, R.; Paolini, E. Detecting the Number of Active Users in IRSA Access Protocols. In Proceedings ofthe 2018 IEEE 29th Annual International Symposium on Personal, Indoor and Mobile Radio Communications(PIMRC), Bologna, Italy, 9–12 September 2018; pp. 1972–1976.24. Sun, J.; Liu, R.; Paolini, E. Unrecovered Users Distribution in Coded Random Access Systems with Erasures.In Proceedings of the 2019 IEEE International Conference on Communications (ICC), Shanghai, China,20–24 May 2019; pp. 1–6.25. Schmetterer, L. An introduction to combinatorial analysis by J. Riordan. Phys. Today 1959, 12, 158.c© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open accessarticle distributed under the terms and conditions of the Creative Commons Attribution(CC BY) license (http://creativecommons.org/licenses/by/4.0/).160sensorsArticleOn the Capacity of 5G NR Grant-Free Schedulingwith Shared Radio Resources to SupportUltra-Reliable and Low-Latency CommunicationsM. Carmen Lucas-Estañ *, Javier Gozalvez and Miguel SepulcreDepartment of Communications Engineering, Universidad Miguel Hernández de Elche (UMH),Avda. de la Universidad s/n, 03202 Elche, Spain* Correspondence: m.lucas@umh.es; Tel.: +34-965-222-424Received: 14 June 2019; Accepted: 10 August 2019; Published: 16 August 2019Abstract: 5G and beyond networks are being designed to support the future digital society,where numerous sensors, machinery, vehicles and humans will be connected in the so-calledInternet of Things (IoT). The support of time-critical verticals such as Industry 4.0 will be especiallychallenging, due to the demanding communication requirements of manufacturing applicationssuch as motion control, control-to-control applications and factory automation, which will requirethe exchange of critical sensing and control information among the factory nodes. To this aim,important changes have been introduced in 5G for Ultra-Reliable and Low-Latency Communications(URLLC). One of these changes is the introduction of grant-free scheduling for uplink transmissions.The objective is to reduce latency by eliminating the need for User Equipments (UEs—sensors, devicesor machinery) to request resources and wait until the network grants them. Grant-free schedulingcan reserve radio resources for dedicated UEs or for groups of UEs. The latter option is particularlyrelevant to support applications with aperiodic or sporadic traﬃc and deterministic low latencyrequirements. In this case, when a UE has information to transmit, it must contend for the usageof radio resources. This can lead to potential packet collisions between UEs. 5G introduces thepossibility of transmitting K replicas of the same packet to combat such collisions. Previous studieshave shown that grant-free scheduling with K replicas and shared resources increases the packetdelivery. However, relying upon the transmission of K replicas to achieve a target reliability levelcan result in additional delays, and it is yet unknown whether grant-free scheduling with K replicasand shared resources can guarantee very high reliability levels with very low latency. This is theobjective of this study, that identiﬁes the reliability and latency levels that can be achieved by 5Ggrant-free scheduling with K replicas and shared resources in the presence of aperiodic traﬃc, and asa function of the number of UEs, reserved radio resources and replicas K. The study demonstratesthat current Fifth Generation New Radio (5G NR) grant-free scheduling has limitations to sustainstringent reliability and latency levels for aperiodic traﬃc.Keywords: grant-free; scheduling; URLLC; ultra-reliable and low-latency communications; 5G;deterministic; time-critical; reliability; latency; aperiodic traﬃc; Industry 4.01. Introduction5G networks are being designed with the objective to support a broad range of verticals such asmanufacturing, transport, health, energy and entertainment. To this aim, important changes have beenintroduced to increase data rates (enhanced mobile broadband, or eMBB), eﬃciently support largeamounts of devices (massive machine type communications, or mMTC) and guarantee unprecedentedreliability and latency levels (Ultra-Reliable and Low-Latency Communications or URLLC) [1].Supporting URLLC is particularly relevant for many Industry 4.0 manufacturing applications, such asSensors 2019, 19, 3575; doi:10.3390/s19163575 www.mdpi.com/journal/sensors161Sensors 2019, 19, 3575motion control (requires a maximum latency of 1 ms and a reliability of 1–10−6 [2]), control-to-controlapplications (maximum latency of 4 ms and a reliability of 1–10−8 [1]) and factory automation(maximum latency between 0.25 ms and 2.5 ms and reliability requirements up to 1–10−9 [3]). Theseapplications require the exchange of information between sensors, actuators and controllers throughan industrial sensor and control network. 5G has the potential to provide the connectivity requiredby the Industry 4.0 to digitalize factories and to support data-intensive services while ubiquitouslyguaranteeing low latency and reliable connections. This has actually been acknowledged through theestablishment of the 5G Alliance for Connected Industries and Automation (5G-ACIA) [4].5G has introduced signiﬁcant changes to support URLLC [5]. Some of these changes focus at theRadio Access Network level, since the medium access mechanisms account for an important part ofthe total end-to-end transmission delay [6]. This is for example the case of the grant-based schedulingprocess for uplink (UL) transmissions in legacy LTE (Long Term Evolution) 4G networks. Grant-basedscheduling requires a User Equipment (UE) and a Base Station (BS) to exchange scheduling requests(SRs) and grant messages before transmitting any data. This process alone already results in an averagedelay of up to 11.5 ms when considering a Transmission Time Interval (TTI) equal to 1 ms and anSR periodicity of 10 ms [3]. Reducing the slot duration can reduce this delay. However, additionalscheduling changes have been necessary to sustain the URLLC requirements that characterize somevertical applications, such as those in Industry 4.0. In particular, Release 15 and 16 of the 3rd GenerationPartnership Project (3GPP) standards have introduced the concept of grant-free scheduling (alsoreferred to as Conﬁgured Grant for 5G New Radio [7]) to support URLLC.With grant-free scheduling, the BS reserves resources for UL transmissions and informs the UEsof the reserved resources. When a UE wants to initiate a UL transmission, it directly utilizes thereserved resources, without sending an SR and waiting for the subsequent grant message from the BS.Recent studies have shown that grant-free scheduling in 5G NR considerably reduces the end-to-endlatency [8]. The 3GPP standards introduce the possibility for grant-free scheduling to reserve resourcesto dedicated UEs, or to a group of UEs. In the ﬁrst case, each resource is reserved for a speciﬁc UE,and only this UE can utilize the resource at any time. This approach is adequate for periodic traﬃc sincethe resource allocations can be planned, and resources can then be utilized eﬃciently. Such planning isnot possible in the case of aperiodic, sporadic or uncertain traﬃc. Sharing dedicated resources by agroup of UEs is hence an interesting option to optimize the usage of the radio resources in the presenceof aperiodic traﬃc. In this case, UEs have to contend for their usage, and collisions are possible. 5G NRintroduces the possibility to transmit K replicas of the same packet in consecutive slots to combatpotential collisions. However, relying on the transmission of K replicas to achieve a target reliabilitylevel can result in additional delays. It is yet unknown whether 5G NR grant-free scheduling withK-repetitions and shared resources can satisfy critical applications and guarantee very high reliabilitylevels with very low latency. In this context, this study presents an in-depth analysis of the reliabilityand latency levels that can be achieved with existing 5G NR grant-free scheduling solutions as afunction of the number of UEs, the number of reserved radio resources, and the number of replicasK. To this aim, the study analytically quantiﬁes the probability of successfully delivering a packetwhen using grant-free scheduling with K-repetitions and shared resources. In addition, the studyanalyzes the impact of self-collisions. Self-collisions occur when a UE has to transmit a new packet,and the transmission of the K replicas of the previous packet has not ﬁnished. If this happens, the newpacket must be stored, and its transmission is delayed until all replicas of the previous packet havebeen transmitted. This study demonstrates for the ﬁrst time that self-collisions have a non-negligibleimpact upon the capacity of 5G NR grant-free scheduling to support stringent URLLC reliability andlatency levels.2. Related WorkThe 5G NR standard introduces the use of grant-free scheduling (also referred to as ConﬁguredGrant [7]). With grant-free scheduling, the network pre-conﬁgures the radio resources and assigns162Sensors 2019, 19, 3575them to UEs without waiting for UEs to request resources. UEs can utilize the pre-assigned resourcesas soon as they have data to transmit. This is in contrast to grant-based scheduling, where UEs mustrequest access to radio resources through the transmission of Scheduling Requests (SR). The BS assignsthe radio resources to the UEs and notiﬁes them using grant messages. UEs must wait to receive thesegrant messages before transmitting any data. Grant-free scheduling eliminates all delays introduced bythe handshaking present in grant-based scheduling. Grant-free scheduling also improves the energyconsumption of the UEs, reduces their complexity, and decreases the signaling overhead comparedwith grant-based scheduling ([8,9]). Grant-free scheduling can assign dedicated or shared resources tothe UEs. The BS decides whether resources are dedicated to speciﬁc UEs, or are shared by a group ofUEs [10]. Reserving resources to dedicated UEs is an interesting approach when we can plan aheadwhat is the demand for resources. This is for example the case of periodic traﬃc. However, reservingresources to dedicated users can be highly ineﬃcient if the traﬃc demand is uncertain or aperiodic,and it is not possible to anticipate when these resources will be needed. In this case, it is possible toshare radio resources by a group of UEs. This option ensures a more eﬃcient utilization of resources,and the possibility to satisfy URLLC communication requirements. However, users must contend forthe resources, and collisions can happen if two or more UEs simultaneously contend for the sameresources. 5G NR introduces the possibility of transmitting K replicas of the same packet in consecutiveslots to combat collisions and thus increase the probability of a correct reception [11,12].The study in [13] analyzes the performance of the K replicas scheme. The authors proposetransmitting the ﬁrst copy of a packet using dedicated resources, and the following replicas usingshared resources. The proposal also exploits shared diversity and advanced receiver processingtechniques to reduce the impact of packet collisions. The proposal achieves adequate reliability levelsand reduces the number of reserved (shared) radio resources, compared to a conﬁguration that reservesresources to dedicated UEs. The study in [14] also transmits the ﬁrst copy of a packet using dedicatedresources. However, it does not consider the transmission of K replicas of a packet. Instead, the authorspropose to retransmit the original packet in a shared resource only if the ﬁrst transmission is notsuccessful. This requires a handshaking between the UEs and the BS to exchange acknowledgementmessages. This handshaking increases the latency, and can compromise the capability to adequatelysupport URLLC applications with stringent latency requirements. In [15], the authors study theoptimum number of replicas (K) necessary to achieve a target reliability level within a deterministiclatency deadline. The study focuses upon aperiodic traﬃc and the case in which a group of UEsshare resources. The authors show that randomly choosing the resource for each replica increases theprobability of correctly delivering a packet. However, the study focuses on reliability levels up to1–10−5 while some critical Industry 4.0 applications require higher reliability levels.Previous studies have shown that transmitting K-repetitions of a packet increases the receptionrate. However, this can be done at the expense of an ineﬃcient use of the radio resources due topacket collisions or the unnecessary reservation of resources when the ﬁrst replicas are correctlydelivered. Latency requirements may also impose restrictions on the number of replicas that can betransmitted, and consequently on the reliability levels that may be achieved. In this context, severalrecent contributions have analyzed slight modiﬁcations to the K-repetitions scheme. For example, [16]proposed adaptively conﬁguring the number of replicas transmitted based on the channel conditions.The objective is to utilize the radio resources eﬃciently by avoiding unnecessary retransmissionswhen the channel quality is good. A similar objective is sought in [17] where authors proposeconditions to stop the transmission of replicas. Other interesting proposals in 3GPP standardizationworking groups include: the transmission of replicas within mini-slots (to reduce the latency) [18],the possibility for transmitting replicas across the slot border, or the concept of periodicity boundary [19].These studies propose interesting variants of the K-repetitions scheme. However, it is yet unknownwhether 5G NR grant-free scheduling with K-repetitions and shared resources can really supportURLLC communications with strict reliability and latency requirements under the presence of aperiodicor sporadic traﬃc. This traﬃc is critical in many verticals, for example in Industry 4.0. In this context,163Sensors 2019, 19, 3575this study conducts an in-depth evaluation of 5G NR grant-free scheduling with K-repetitions andshared resources in the presence of aperiodic or sporadic traﬃc. The study identiﬁes the reliabilityand latency levels that can be achieved with 5G NR grant-free scheduling, and identiﬁes its currentlimitations. The study analyzes the impact of the number of UEs in the network, the number of reservedradio resources, and the number of replicas K. The study also analyzes for the ﬁrst time the impact ofself-collisions. The conducted analysis helps to identify the reliability and latency levels that can beachieved based on network deployments and conﬁguration options for 5G NR grant-free scheduling.It should be noted that 3GPP standards deﬁne the possibility of utilizing grant-free schedulingand transmitting K replicas, but do not deﬁne a speciﬁc scheme to be implemented. This study is basedon the implementation of 5G NR grant-free scheduling with K replicas and shared resources proposedin [15]. This implementation is chosen because it has been speciﬁcally designed to guarantee stringentURLLC latency and reliability requirements. To this aim, the implementation transmits original packetsand all of the replicas using grant-free scheduling on shared radio resources. A diﬀerent approachis proposed in [13] where dedicated resources are used to transmit the original packets, and sharedresources are used for the following replicas. This approach can increase the delay compared to [15]if grant-based scheduling is utilized to allocate the dedicated resources. The eﬃcient utilizationof resources could also be compromised if dedicated resources were reserved for each UE whensupporting applications with aperiodic traﬃc. The implementation of 5G NR grant-free schedulingwith K–repetitions and shared resources proposed in [15] is therefore better suited to support URLLCapplications with aperiodic or sporadic traﬃc.3. Grant-Free SchedulingThis paper uses grant-free scheduling with K-repetitions and shared resources to evaluate thereliability and latency levels that can be achieved in the presence of aperiodic traﬃc. Following [20],reliability for URLLC services is deﬁned as the percentage of data packets that are successfully deliveredbefore the latency deadline L established by the service or application. Following 3GPP standards [11],UEs transmit the same data packet in K consecutive transmission slots with a duration Tslot. The UErandomly selects an RB (Resource Block) for each transmission from the U RBs available per Tslot.This is illustrated in Figure 1 that represents the time/frequency resource grid map in 5G NR, where theunit is an RB. In 5G NR, a wideband channel is divided into sub-frames, slots and RBs. An RB is thesmallest unit of frequency resources that can be allocated to a UE. Without loss of generality, this studyconsiders a numerology μ equal to 3 with a subcarrier spacing of 120 kHz [21]. An RB is then 1440 kHz(Δf ) wide in frequency (12 sub-carriers of 120 kHz) and lasts for one time slot with the duration Tslotequal to 0.125 ms.The reliability at the medium access level that can be achieved with grant-free scheduling withK-repetitions and shared resources depends upon two main factors. The ﬁrst factor is the possibilitythat a packet is not correctly received due to the collision of all its K replicas with other transmissions;this is due to the random selection of the RB for the transmission of each replica. The study in [15]showed that the possibility to successfully deliver a packet increases with the number K of replicas.The second factor is the eﬀect of self-collisions. A self-collision occurs when a UE has to transmit anew packet, and the transmission of the K replicas of the previous packet has not ﬁnished. If thishappens, the new packet must be stored, and its transmission is delayed until all the replicas of theprevious packet have been transmitted. This delay can result in the case that the new packet cannotbe delivered within the latency limit, and hence self-collisions can impact the reliability of URLLCservices. It is important then that the reliability (or probability that a packet is correctly receivedbefore the latency deadline) of grant-free scheduling with K-repetitions and shared radio resourcesis computed considering both the eﬀect of collisions from other UEs, and the eﬀect of self-collisions.In this case, the reliability or probability Prel that a packet is correctly received by the BS must considerthe probability Psc that the transmission of the K replicas of a packet is not completed before the latencydeadline L due to the eﬀect of self-collisions. For the packets that are not aﬀected by the eﬀect of164Sensors 2019, 19, 3575self-collisions, it must be considered the probability Pc that a packet is not correctly received due to thecollision of all its K replicas with other transmissions. Hence, Prel can be expressed as:Prel = 1− (Psc + (1− Psc) · Pc) (1)In [15], its authors presented an expression to approximate the probability Pc of the collision ofthe K replicas of a packet with the transmission of other UEs. The expression was derived in scenarioswhere N UEs share the same pool of RBs. However, [15] did not analyze the impact of self-collisions,since the study only considered low values of K (equal to or lower than 4). For these low values,self-collisions might not have an impact upon the reliability, as will be later shown. In this paper,we analytically derive the exact probability of any collision of the K replicas of a packet with packetstransmitted by other UEs (Pc). We also quantify the impact of self-collisions (Psc), and analyticallycompute the reliability that can be achieved by grant-free scheduling with K-repetitions and sharedresources (Prel). These analytical expressions are a valuable contribution to the community since theycan be easily utilized to evaluate 5G NR grant-free scheduling. The availability of these exact analyticalexpressions is particularly useful when considering applications with very demanding reliability andlatency URLLC requirements. This is the case of certain Industry 4.0 applications. For example, motioncontrol requires a maximum latency of 1 ms and a reliability of 1–10−6. Control-to-control applicationsrequire a maximum latency of 4 ms and a reliability of 1–10−8. Factory automation applications usuallydemand maximum latency values in the range 0.25–2.5 ms and reliability levels up to 1–10−9. In thiscase, simulations can be very computationally expensive if we want to compute the packet receptionrate (1 − Pc) with reliability demands in the order of 1–10−6 to 1–10−9. In these scenarios, errors arevery rare, and we need long and computationally expensive simulations to achieve accurate results.The analytical methodology utilized in this study is then an adequate and eﬃcient tool for scenarioswith demanding URLLC communication requirements.f/?ft/Tslot????u???U? ? ? ??? i ???????? ?????????????????????????????Figure 1. Illustration of the Fifth Generation New Radio (5G NR) resource grid map: Transmission of adata packet with four repetitions and a random selection of Resource Blocks (RBs) per slot.3.1. Collisions with Other UEsFirst, we focus on the probability Pc that a packet is not correctly received due to the collisions ofits K replicas with the packets transmitted by other UEs. To this end, we consider UL transmissionsand N UEs within a single cell with aperiodic traﬃc. Packets are generated by each UE following aPoisson distribution with exponential inter-arrival time. The average packet inter-arrival time is equalto 1/λ, where λ is the average number of packets generated per second. We consider the transmissionof small packets with a size of 32 bytes [22], and we assume without loss of generality that each packetrequires only one RB.The probability Pg that one or more packets are generated for a UE in a time period Tslot is equal to:Pg = 1− exp(−Tslot · λ) (2)165Sensors 2019, 19, 3575We deﬁne Ri as the set of UEs for which a new packet could be generated in a slot si (the slot has atime duration equal to Tslot). Here, ni is the number of UEs that do have a new packet to transmit in si.This ni can then take any value between 0 and the cardinality of Ri. The probability Ptx(ni,Ri) that niUEs from the set Ri of UEs have new packets to be transmitted in si with duration Tslot is equal to:Ptx(ni,Ri) =( |Ri|ni)· Pgni ·(1− Pg)|Ri |−ni (3)where |Ri| represents the number of elements or the cardinality of the set Ri.A packet will not be successfully delivered to the BS if all its K replicas collide with thetrans-missions of other UEs. A UE has an active transmission in si if it generated a new data packet inthe previous slots si−(K−1), . . . , si−1, and si. If this is the case, then the UE would be transmitting one ofthe K replicas in si. We denote as nacti the number of UEs with active transmissions in si. The probabilityPnrc(nacti ,U) that nacti UEs do not collide with a given UE is equal to the probability that they do notselect the same RB at a given slot for their next transmission as the UE under study. Pnrc(nacti ,U) isgiven by:Pnrc(nacti ,U) =(U − 1U)nacti(4)Equations (2)–(4) are necessary to compute the probability Pc that a packet is not correctly receivedat the BS due to the collision of all its K replicas with the transmissions of other UEs. To compute Pc,let us consider the case of a particular UE1 that has to transmit the K replicas of a packet in slots si,si+1, . . . , si+K−1. For the sake of clarity, we consider an example with K = 4, and si corresponding to s3.Pc is then equal to the probability of collision of the 4 replicas transmitted in s3, s4, s5, and s6, which isrepresented by Prc(s3, s4, s5, s6):Pc = Prc(s3, s4, s5, s6) (5)To determine Prc(s3, s4, s5, s6), we ﬁrst study the probability Prc(s3) that the replica of the packettransmitted in s3 collides with a transmission from any other UE. Prc(s3) is given by the probabilitythat one or more UEs (in addition to UE1) have an active transmission in s3 (i.e., nact3 ≥ 1), and that oneor more of the nact3 UEs select the same RB as UE1 for their transmission. nact3 is equal to n0 + n1 + n2 +n3, and the probability Prc(s3) has to consider all possible combinations of n0, n1, n2 and n3 that resultin nact3 ≥ 1. The probability P(nact3 ≥ 1) can then be expressed as:P(nact3 ≥ 1) =nmax0∑n0=nmin0⎧⎪⎪⎪⎪⎨⎪⎪⎪⎪⎩Ptx(n0,R0) ·nmax1∑n1=nmin1〈Ptx(n1,R1) ·nmax2∑n2=nmin2⎡⎢⎢⎢⎢⎢⎢⎢⎢⎣Ptx(n2,R2) ·nmax3∑n3=nmin3Ptx(n3,R3)⎤⎥⎥⎥⎥⎥⎥⎥⎥⎦〉⎫⎪⎪⎪⎪⎬⎪⎪⎪⎪⎭ (6)where nmaxi and nmini represent the maximum and minimum possible values of ni in each slot, and areequal to:nmaxi = |Ri|, ∀i ≤ 3 (7)nmini ={1 if i = 3& |Ri| = N − 10 otherwise, i ≤ 3 (8)where Ri is the set of UEs that could have a new packet to be transmitted in si. Ri is equal to the totalnumber of UEs (N) minus UE1 and all active UEs in the slot previous to si. The cardinality of Ri is thenequal to:|Ri| = N − 1−∑i−1j=max{i−3,0} nj, i ≤ 3 (9)166Sensors 2019, 19, 3575It should be noted that nmini is equal to 0 or 1 in order to guarantee that nact3 is equal to or higherthan one. nacti can be expressed as:nacti =∑ij=max{i−3,0} nj, i ≤ 3 (10)To achieve ﬁnally the expression of Prc(s3), we need to incorporate to the expression of P(nact3 ≥ 1)in (6) the probability that one or more of the nact3 UEs select the same RB as UE1 for their transmissions.This probability is equal to 1 − Pnrc(nact3 ,U). Prc(s3) is then calculated as:Prc(s3) =nmax0∑n0=nmin0⎧⎪⎪⎪⎨⎪⎪⎪⎩Ptx(n0,R0) ·nmax1∑n1=nmin1〈Ptx(n1,R1) ·nmax2∑n2=nmin2⎡⎢⎢⎢⎢⎢⎢⎣Ptx(n2,R2) ·nmax3∑n3=nmin3{Ptx(n3,R3) · (1− Pnrc(nact3 ,U))}⎤⎥⎥⎥⎥⎥⎥⎦〉⎫⎪⎪⎪⎬⎪⎪⎪⎭ (11)The probability of collision of the replica transmitted in s4 depends upon the number nact4 ofUEs with active transmissions in s4. This nact4 depends on the number n1, n2, n3 and n4 of UEs thathave new packets to transmit in s1, s2, s3, and s4, respectively. The probability that UEs have newpackets to transmit in s1, s2, and s3 is already included in (11) (Ptx(n1,R1), Ptx(n2,R2), and Ptx(n3,R3)respectively). In this context, Prc(s3) and Prc(s4) are not independent, and they must be calculatedjointly. We then compute the joint probability Prc(s3, s4) that the replicas transmitted in s3 and s4collide with transmissions from other UEs. Computing Prc(s3, s4) only requires including in (11) theprobability that there are UEs with new packets to be transmitted in s4 (i.e., Ptx(n4,R4)), and theprobability that one or more of the active nact4 UEs in s4 select the same RB for their transmission thanUE1. Prc(s3, s4) can then be expressed as:Prc(s3, s4) =nmax0∑n0=nmin0⎧⎪⎪⎪⎨⎪⎪⎪⎩Ptx(n0,R0) ·nmax1∑n1=nmin1〈Ptx(n1,R1) ·nmax2∑n2=nmin2[Ptx(n2,R2)·nmax3∑n3=nmin3〈{Ptx(n3,R3) · (1− Pnrc(nact3 ,U))}·nmax4∑n4=nmin4{Ptx(n4,R4) · (1− Pnrc(nact4 ,U))}〉⎤⎥⎥⎥⎥⎥⎥⎦〉⎫⎪⎪⎪⎬⎪⎪⎪⎭(12)where nact4 , |R4|, nmax4 and nmin4 are deﬁned as:nact4 =∑4j=1nj (13)|R4| = N − 1−∑3j=1nj (14)nmax4 = |R4| (15)nmin4 ={1 if |R4| = N − 10 otherwise(16)The process followed to account for possible collisions of the replicas transmitted in s5 and s6 issimilar to that considered for s4. Pc can then be expressed as follows when K = 4:Pc =nmax0∑n0=nmin0⎧⎪⎪⎪⎨⎪⎪⎪⎩Ptx(n0,R0) ·nmax1∑n1=nmin1〈Ptx(n1,R1) ·nmax2∑n2=nmin2[Ptx(n2,R2)·nmax3∑n3=nmin3〈{Ptx(n3,R3) · (1− Pnrc(nact3 ,U))}·nmax4∑n4=nmin4[{Ptx(n4,R4) · (1− Pnrc(nact4 ,U))}·nmax5∑n5=nmin5〈{Ptx(n5,R5) · (1− Pnrc(nact5 ,U))}·nmax6∑n6=nmin6{Ptx(n6,R6) · (1− Pnrc(nact6 ,U))}〉⎤⎥⎥⎥⎥⎥⎥⎦〉⎤⎥⎥⎥⎥⎥⎥⎦〉⎫⎪⎪⎪⎬⎪⎪⎪⎭(17)167Sensors 2019, 19, 3575where nacti , |Ri|, nmaxi and nmini ∀i ∈ [0, 2·K−1] are deﬁned as:nacti =∑ij=max{i−(K−1),0} nj (18)|Ri| = N − 1−∑i−1j=max{i−(K−1),0} nj (19)nmaxi = |Ri| (20)nmini ={1 if i ≥ K − 1 & |Ri| = N − 10 otherwise(21)The process illustrated for K = 4 can be followed to compute Pc for any value of K. As shownin (22), Pc can be computed using the auxiliary function hi(K,N,U) deﬁned in (23) with i equal to cero.To simplify the notation, hi(K,N,U) is also represented as hi in (22) and (23). As it can be observedin (23), h0 depends on h1, and in general, hi depends on hi+1, until h2K−1.Pc(K,N,U) = h0(K,N,U) = h0 (22)hi =⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩nmaxi∑ni=nmini[Ptx(ni,Ri) · hi+1] if i ∈ [0, K)nmaxi∑ni=nmini[Ptx(ni,Ri) · (1− Pnrc(nact5 ,U)) · hi+1]if i ∈ [K, 2 ·K − 1)nmaxi∑ni=nmini[Ptx(ni,Ri) · (1− Pnrc(nact5 ,U))]if i = 2 ·K − 1(23)The parameters nacti , |Ri|, nmaxi and nmini in (23) correspond to those expressed in (18)–(21).3.2. Self-CollisionsThe eﬀect of self-collisions is illustrated in Figure 2. We may suppose that a UE starts transmittinga packet p1 that was generated before t0. Let us then suppose then that a second packet p2 is generatedbefore the K replicas of the previous packet p1 have been transmitted. This is a self-collision. If aself-collision happens, p2 can be stored, and its transmission will start after the UE has transmitted theKth replica of p1 (i.e., at t1 in Figure 2). The transmission of the K replicas of p2 will ﬁnish at t2 that isequal to:t2 = 2 ·K · Tslot + t0 (24)The transmission of the K replicas of p2 may ﬁnish after the latency deadline L, due to the time p2being stored as the K replicas of p1 are being transmitted. We then analyze the probability Psc thatthe transmission of K replicas of a packet is not completed before L due to the eﬀect of self-collisions.This probability depends upon the number of replicas K and on the time instant at which p2 wasgenerated. Figure 2 illustrates how self-collisions aﬀect the probability of completing the transmissionof p2 before L, with L equal to 1 ms. L = 1 ms implies that the maximum number of replicas K that canbe transmitted per packet is 8. However, it is possible to transmit less than 8 replicas, and Figure 2represents the case in which K is set equal to 4, 6 or 8. p2 can be transmitted before the deadline L if itis generated at any time instant after t2 − L, where t2 is the time at which the transmission of the Kreplicas of p2 is ﬁnished (the transmission of p2 starts when the transmission of the K replicas of p1has ﬁnished at t1). If p2 is generated before t2 − L, it is not possible to complete the transmission ofthe K replicas of p2 before the latency deadline L. Psc can then be computed as the probability that thetime between the generation of two consecutive packets at a UE falls within the interval [0, Δt], where168Sensors 2019, 19, 3575Δt represents the time diﬀerence between t2 − L and the time tp1 at which p1 is generated (see (26)).Psc can then be expressed as:Psc(Δt) =∫ Δt0λ · e−t·λ · dt (25)Δt = t2 − L− tp1 = 2 ·K · Tslot − L− tp1 (26)tp? p? p? p? p? p? p? p? p? p? p?p? p? p? p? p?K??p?????t?t? t?p??t????tp? p? p? p? p? p? p? p? p? p? p?p?K??t?t? t?p? p??ttp? p? p? p? p? p? p? p?K??t? t?????t?p? p??tFigure 2. Scenarios with possible self-collisions (L = 1 ms and K = 4, 6 and 8).As shown in (25) and (26), the negative eﬀect of self-collisions increases with the value of K, since Kinﬂuences the time a packet might be stored until the transmission of the previous packet is ﬁnished.However, increasing the number K of replicas transmitted for each packet is preferred, in order tocombat possible collisions with other UEs sharing the same pool of radio resources. The next sectionwill analyze both the eﬀect of collisions from other UEs and the eﬀect of self-collisions to analyze thereliability achievable with the grant-free scheduling with K-repetitions and shared radio resources.4. ValidationThis section validates the analytical expressions derived in Section 3.1 to calculate the probabilityPc that a packet is not correctly received due to packet collisionswith otherUEs. To this aim, we comparethe results achieved with the analytical expressions, with that obtained through simulations.We have implemented a system level simulator in Matlab™ that accurately models the 5G NRgrant-free scheduling process with K-repetitions and shared resources. The simulator emulates a singlecell with N UEs that generate aperiodic traﬃc. Each UE models the packet traﬃc arrival, using aPoisson distribution with exponential inter-arrival time. The average packet inter-arrival time is equalto 1/λ, where λ is the average number of packets generated per second. The simulator implements thetime/frequency resource grid map of 5G NR. The time and frequency duration of RBs is conﬁgurablebased on the considered 5G NR numerology μ. It is possible to also conﬁgure the number U of RBsavailable per time slot. The number K of replicas can also be conﬁgured in the simulation platform.We have conducted a large number of simulations to ensure the accuracy of the simulation results,and compare them to those obtained with our analytical expressions and methodology. Simulationsare here shown for K equal to 2, 4 and 8, λ equal to 0.1 packets, μ equal to 3, and U equal to 6 RBsper slot. UEs transmit small packets with a size of 32 bytes [22] that can be transmitted in a singleRB. Figure 3 compares the value of Pc achieved analytically and through simulations for a varyingnumber N of users in the cell. The ﬁgure shows that the results achieved analytically preciselymatch those obtained through the simulations. Similar trends have been observed for other values ofthe parameters. The results achieved clearly validate the proposed methodology and the analyticalexpressions presented in Section 3.1.169Sensors 2019, 19, 3575Figure 3. Comparison of analytical and simulation results for diﬀerent latency requirements L andnumber of repetitions K (U = 6, λ = 0.1 packets).It is important to highlight that this study focuses on URLLC applications that demand veryhigh reliability levels. In simulations, we compute the number of packets for which the K replicashave collided with those packets transmitted by other UEs, and then compute the achieved reliability(Prel = 1− Pc). It is rare that all K replicas of a packet collide with transmissions from other UEs forlow values of N. This is particularly the case when K increases. In this context, the computational costof simulations signiﬁcantly increases if we want to achieve accurate statistical results. This explainswhy simulation results are not shown for values of N below 30 when K = 8. It also highlights thevalue of our analytical expressions and methodology to estimate the performance of 5G NR grant-freescheduling for demanding URLLC applications and aperiodic traﬃc.5. Performance EvaluationThis section evaluates the capacity of 5G NR grant-free scheduling with K-repetitions and sharedresources to meet the reliability and latency requirements of URLLC services. To this aim, we usethe analytical expressions that are derived in Section 3 and were validated in the previous section.Reliability for URLLC services is deﬁned as the percentage Prel of data packets that are successfullyreceived by the BS before the latency deadline established by the service or application. In this study,we analyze ﬁrst the reliability, considering only the eﬀect of collisions from other UEs. This studyanalyzes then the impact of self-collisions on the capacity of 5G NR grant-free scheduling withK-repetitions and shared resources to achieve the reliability levels demanded by URLLC services.This is particularly relevant, as this study extends the state of the art by evaluating the capacity of 5GNR grant-free scheduling to sustain reliability levels even higher than 1–10−9. This study also evaluatesthe performance of 5G NR grant-free scheduling as a function of the number of UEs, the number ofreserved radio resources, and the number K of replicas.The performance of 5G NR grant-free scheduling is evaluated considering a single cell with NUEs. Packets are generated by each UE following a Poisson process with exponentially inter-arrivaltime. The average packet inter-arrival time is equal to 1/λ, where λ is the average number of packetsgenerated per second. UEs transmit small packets with a size of 32 bytes [22]. Radio resources aredivided in 6 × 12 subcarriers (i.e., U = 6) with a subcarrier spacing of 120 kHz (i.e., Tslot = 0.125 ms).Figure 4 shows the probability Pc that a packet is not correctly received at the BS due to the collisionsfrom other UEs experienced by all of the replicas of a packet (This would correspond to the reliabilityachieved with 5G NR grant-free scheduling if there were no self-collisions, i.e., Psc = 0 and Prel = 1 − Pc).The ﬁgure shows the value of Pc that can be achieved as a function of the number of UEs for latencyrequirements (L) of 0.25, 0.5, 0.75 and 1 ms. We focus on services with the most stringent latencyrequirements, given the challenge to satisfy high reliability levels when latency decreases [23]. For eachvalue of L, the grant-free scheduling scheme is executed with the maximum possible number of replicas170Sensors 2019, 19, 3575K that can be transmitted within the required latency. For example, if the maximum latency L that canbe tolerated is equal to 1 ms, the maximum number of replicas K that can be transmitted within 1 ms isequal to 8 (L = 1 ms corresponds to 8·Tslot when Tslot = 0.125 ms). Figure 4 also shows the performanceachieved for two values of λ (0.1 and 1 packet(s)). The results depicted in Figure 4 clearly show thatreducing the probability Pc of not receiving a packet to values as low as 10−9, (and hence reachingreliability levels of 1–10−9 when the eﬀect of self-collisions is not considered), can only be achievedwith high values of K and values of L equal to 0.75 or 1 ms. Figure 4 also shows that the probability Pcincreases with the number of UEs, since the risk of collision is higher. As a result, the capacity of 5G NRgrant-free scheduling to support high reliability levels is signiﬁcantly decreased as the number of UEsto be supported increases. Figure 4 also shows that the diﬃculty in supporting high reliability levelsincreases with λ, since the probability Pc increases as a result of a higher risk of collision between UEs. (a) (b) Figure 4. Pc as function of the number of User Equipments (UEs) and for diﬀerent latency requirementsL: (a) λ = 0.1 packets; (b) λ = 1 packet.Figure 5 depicts the number of UEs that can be supported with a given latency requirement (L)and a reliability of Prel = 1 − Pc when Psc = 0. It is important to remember that L establishes themaximum number of replicas K that can be transmitted. The results (the number of supported UEs)for each value of L in Figure 5 have been obtained for the maximum value of K permitted by L (Kequal to 2, 4, 6 and 8 for L equal to 0.25, 0.5, 0.75 and 1 ms, respectively). The Release 15 of the 3GPPstandards [22] establishes URLLC requirements with a latency of L = 1 ms and a reliability target of1–10−5. Figure 5 shows that grant-free scheduling with K-repetitions and shared resources can achievea reliability equal to 1–10−5 with only K = 2 if we do not consider self-collisions. Grant-free schedulingwith K = 2 can also guarantee a latency as low as 0.25 ms. For low values of the packet generationrate (i.e., λ = 0.1 packets), grant-free scheduling with 2 repetitions can support up to 34 UEs with areliability of 1–10−5 and L = 0.25 ms if we do not consider self-collisions. The number of UEs thatcan be supported decreases with λ, since the risk of collision with other UEs increases when each UEtransmits more packets per second. For example, only 4 UEs can be supported with L = 0.25 ms and areliability of 1–10−5 when λ = 1 packet. If the latency requirement is relaxed to 0.5 ms or even higher,grant-free scheduling can support more than 500 UEs with only K = 4 when λ = 0.1 packets. If λincreases, grant-free scheduling can only guarantee the required reliability for 500 UEs if the latencyrequirement is 1 ms, and each UE can transmit 8 replicas of the same packet. These results showthat the reliability and latency levels that can be achieved with grant-free scheduling depend uponconﬁguration parameters (e.g., K), the traﬃc (e.g., λ) and the number of UEs supported. An adequateconﬁguration and optimization of grant-free scheduling based on the network conditions could helpsupport stringent reliability and latency levels. However, it is important to note that these results areachieved without considering self-collisions. The impact of self-collisions might be non-negligiblewhen, for example, K and/or λ increase.The Release 16 of 3GPP standards for 5G NR [2] deﬁnes use cases with higher reliabilityrequirements (up to 1–10−6). Some Industry 4.0 applications (e.g., factory automation) require even171Sensors 2019, 19, 3575higher reliability levels (up to 1–10−9), as discussed in [3]. It is then important analyzing whethergrant-free scheduling with K-repetitions and shared resources can guarantee reliability levels of theorder of 1–10−9. Figures 4 and 5 show that grant-free scheduling can only guarantee very high reliabilitylevels with high values of K, which limits the latency requirements (L) that can be satisﬁed. For example,a probability to correctly receive a packet equal to 1–10−7 cannot be guaranteed when L < 0.5 ms,even for the lower packet generation rates. If the reliability requirement increases to Prel = 1–10−9,grant-free scheduling can only support 5 UEs with L = 0.75 ms and λ = 0.1 packets. It can support86 UEs if the latency requirement is relaxed to 1 ms. However, if λ increases to 1 packet then grant-freescheduling can only support 10 UEs with a reliability of 1–10−9 even if L is equal to 1 ms. (a) (b) Figure 5. Number of UEs supported with diﬀerent requirements (L and Prel = 1 − Pc, when Psc = 0):(a) λ = 0.1 packets; (b) λ = 1 packet.Figure 6 shows the impact of U upon the performance of the grant-free scheduling scheme withK-repetitions and shared resources. U is the number of available RBs (Resource Blocks) per Tslot.In particular, Figure 6 depicts the number of UEs that can be supported with a given reliability andlatency L when U decreases and λ is set equal to 0.1 packets (the reliability is equal to Prel = 1 − Pc whenthe eﬀect of self-collisions is not taken into account, i.e., Psc = 0). Figure 6 shows that the number ofUEs that grant-free scheduling with K-repetitions can support for a given set of requirements stronglydepends upon the number of RBs available. UEs randomly select an RB for each transmission from theU RBs available per slot. The probability that several UEs select the same RB for their transmissionsincreases when the number of RBs per slot decreases. Consequently, the probability Pc that a packetis not correctly received due to packet collisions, increases. In addition, the number of UEs that canachieve a target reliability level also decreases when the number of RBs per slot decreases. For example,443 UEs can be supported with L = 0.5 ms (and hence K = 4) and Pc = 10−5 when U is equal to 5 RBs.This number decreases to 69 UEs when U decreases to 3 RBs. This is a signiﬁcant reduction of 84%.This reduction increases when the reliability demand increases. For example, 86 UEs can be supportedwith Pc = 10−9 and L = 1 ms (and hence K = 8) when U is equal to 6. However, only 6 UEs can achievethese values of Pc and L if U decreases to 4 (i.e., a 93% reduction).172Sensors 2019, 19, 3575Figure 6. Number of UEs supported for a given L and Prel = 1 − Pc with Psc = 0 as a function of thenumber U of available RBs per Tslot (λ = 0.1 packets).All previous results have been derived without considering the eﬀect of self-collisions.Self-collisions were illustrated in Figure 2, and the probability of self-collision was derived in Section 3.2.As previously described, if a packet p2 is generated before the K replicas of the previous packet p1have been transmitted, p2 will be stored and transmitted after completing the transmission of the Kreplicas of p1. Due to the time that p2 is stored, the transmission of its K replicas may ﬁnish after thelatency deadline L. As presented in Section 3.2, it is not possible to complete the transmission of the Kreplicas of p2 before the latency deadline L if p2 is generated before t2 − L (t2 is the time at which thetransmission of the K replicas of p2 is ﬁnished as shown in Figure 2). This results in that the probabilityPsc (the probability that the transmission of K replicas of a packet is not completed before L due tothe eﬀect of self-collisions) is equal to the probability that the time between the generation of twoconsecutive packets at a UE falls within the interval [0, Δt], where Δt represents the time diﬀerencebetween t2 − L and the time tp1 at which p1 is generated (see (25) and (26)).We consider that packets are generated following a Poisson process with exponential inter-arrivaltime. As a result, Δt is homogeneously distributed between Δt1 and Δt2. For K = 4 in Figure 2, Δt1 isequal to 0 and Δt2 is equal to Tslot, since p1 can be homogeneously generated between t0 and t0 − Tslot.When K = 6, Δt1 is equal to 4 ·Tslot, and Δt2 is equal to (4+1) ·Tslot, since p1 can be homogeneouslygenerated between t0 and t0 − Tslot. Similarly, Δt1 and Δt2 are equal to 8 ·Tslot and (8+1) ·Tslot forK = 8. Table 1 shows the value of Psc given in (26) when Δt is equal to Δt1 or Δt2 considering L = 1ms and K = 4, 6 and 8. Δt = Δt1 corresponds to the scenario where self-collisions are less probable,while Δt = Δt2 corresponds to the case in which they are more probable.The results in Table 1 show that the probability of self-collision is non-negligible. For example,Psc can reach values equal to 1.25 × 10−4 and 9.99 × 10−4 when K is equal to 4 and 8, respectively,and λ = 1 packet. It is also important to highlight that a comparison of results in Figure 4 and Table 1shows that Psc can be actually higher than Pc. This is for example the case when K = 8: Pc is lower than10−7 and 10−5 for λ equal to 0.1 and 1 packet(s), respectively (Figure 4), while Psc is approximately equalto 10−4 and 10−3 (Table 1). Grant-free scheduling can hence be limited by the eﬀect of self-collisions,in particular when K increases. It is then important that the reliability (or probability that a packet iscorrectly received before the latency deadline) of grant-free scheduling with K-repetitions and sharedradio resources is computed considering both the eﬀect of collisions from other UEs and the eﬀect ofself-collisions following (1).Table 1. Psc for L = 1 ms.KΛ = 0.1 Packets Λ = 1 PacketΔt = Δt1 Δt = Δt2 Δt = Δt1 Δt = Δt24 0 1.25 × 10−5 0 1.25 × 10−46 5.00 × 10−5 6.25 × 10−5 5.00 × 10−4 6.25 × 10−48 9.99 × 10−5 1.13 × 10−4 9.99 × 10−4 1.13 × 10−3173Sensors 2019, 19, 3575Figure 7 plots 1 − Prel for diﬀerent values of K and L when considering both Pc and Psc. The resultsare plotted consideringΔt=Δt1 for computingPsc. Δt=Δt1 corresponds to the casewhere self-collisionsare less probable. Figure 4 shows that it is necessary to transmit a high number of replicas K within Lto combat collisions from other UEs and correctly receive a packet at the BS. For example, Figure 4shows that K must be equal to 8 in order to achieve Prel = 1–10−9 when Psc = 0 and λ is equal to 1packet. However, Table 1 showed that the eﬀect of self-collisions increases with K even to the pointthat self-collisions limit the reliability that can be achieved. This is actually shown in Figure 7 when weconsider L = 1 ms. In principle, it could be possible to satisfy a 1 ms latency requirement if we transmit4, 6 or 8 replicas of a packet. Figure 7 shows that if K = 4 and Δt = Δt1 (for computing Psc in (26)),the impact of self-collisions is not relevant, and the reliability levels of 1–10−5 can be satisﬁed for morethan 500 UEs and 80 UEs when λ is equal to 0.1 and 1 packet(s), respectively; these results are in linewith those observed in Figure 4 for K = 4. However, when K is equal to 6 or 8, the eﬀect of self-collisionsbecomes more relevant (Table 1), and Figure 7 shows that it can actually limit the maximum reliabilitythat can be achieved independently of the number of UEs. In fact, the maximum reliability that can beachieved is approximately equal to 1 − Psc. In this case, for K = 8 and λ = 1 packet/s, the maximumreliability (when Psc is computed considering Δt = Δt1) that can be achieved is 1–10−3 when the latencyrequirement L is equal to 1 ms. It should be noted that reliability levels even higher than 1 − Pc = 1–10−9were achieved when the eﬀect of self-collisions was not considered (Figure 4). The results discussed sofar correspond to the scenario where Psc has been computed considering Δt = Δt1. This correspondsto the scenario where self-collisions are less probable. Figure 7 also shows the reliability that can beachieved with L = 1 ms and K = 4 when Δt = Δtavg. This Δtavg is the average value of Δt. Δtavg = (Δt1 +Δt2)/2, since Δt is homogeneously distributed between Δt1 and Δt2. Figure 7 shows that in this case it isnot possible to achieve a reliability higher than 1–6.3 × 10−5 and 1–6.3 × 10−4 when λ is equal to 0.1 and1 packet(s). Figure 7 also shows that the reliability becomes again nearly independent of the number ofUEs that are being supported. The degradation of reliability experienced from Δt = Δt1 to Δt = Δtavg isagain due to a major relevance of the eﬀect of self-collisions when we compute the reliability. (a) (b) Figure 7. Reliability for diﬀerent latency requirements L and number of repetitions K (U = 6): (a) λ = 0.1packets; (b) λ = 1 packet.Expressions in (25) and (26) show that Psc also depends upon the latency requirement L. The eﬀectof self-collisions is more relevant when the latency requirement is stricter. For example, Figure 7 showsthat the eﬀect of self-collisions already limits the maximum reliability that can be achieved whenK = 4 if the latency requirement is equal to 0.5 ms. Latency requirements signiﬁcantly inﬂuence thereliability levels that can be satisﬁed. This is the case because latency requirements limit the number Kof replicas that can be sent for each packet. Figure 4 shows that the maximum reliability level that canbe guaranteed depends on the latency requirements when only considering Pc. Figure 7 also shows174Sensors 2019, 19, 3575that the eﬀect of self-collisions becomes more relevant with stricter latency requirements. These resultsshow that it is a challenge guaranteeing high reliability demands with very low latency levels.The results in Figure 7 demonstrate that current 5G NR grant-free scheduling with K-repetitionsand shared resources cannot guarantee some of the more demanding reliability and latency levels.However, it is important emphasizing that other proposals cannot meet such requirements either,and these actually perform worse than the implementation analyzed in this study. This is actuallythe case for the proposals that transmit the ﬁrst copy of a packet in dedicated resources for the UEs.These resources can be reserved using grant-based scheduling (such as in [14]) or semi-persistentscheduling (such as in [13]). Grant-based scheduling requires the UE to send an SR to the BS, and waitfor the BS to reply with a grant message. The exchange of these messages between the UE and the BSis illustrated in Figure 8 This handshaking generates a non-negligible Ttotal latency that is equal to:Ttotal = 2 TL1/L2 + Talign + 2 Tproc + 3 Ttx = 2.3 ms (27)where TL1/L2 is the L1/L2 processing latency at the BS and the UE, Talign is the alignment latency (thealignment latency is the time elapsed from the moment the UE is ready to transmit to the actual timethe transmission starts), Tproc is the processing latency (this latency represents the latency betweenthe reception of the SR and the transmission of the grant message), and Ttx is the time required totransmit the SR and grant messages. Following [24], we consider TL1/L2 = Talign = Ttx = 1 TTI, and Tproc= 2.33 TTI. These values are a best-case scenario, since they represent reduced processing times thatcan be achieved with 3GPP Release 15 compared to Release 14. Equation (27) shows that the totallatency (2.3 ms) introduced by the grant-based scheduling process to assign dedicated resources to UEsis higher than the latency achieved with the 5G NR grant-free scheduling implementation analyzed inthis study. For example, Figure 7 shows that this implementation can guarantee latency levels below1 ms (this latency is guaranteed with a reliability up to 1–10−5 when K = 4, λ = 0.1 packets, U = 6,and Δt = Δtavg).???????????????????????????? ???????????????? ???????????? ?????????????????????????????????????????????????????????? ???????????? ?Figure 8. Latency introduced in grant-based scheduling.The alternative to grant-based scheduling is Semi-Persistent Scheduling (SPS). In this case, UEs areassigned dedicated resources for a period of time. During this period, UEs can utilize the resourceswithout requesting permission from the BS. This avoids the latency introduced by grant-basedscheduling. However, semi-persistent scheduling ineﬃciently utilizes the radio resources when thetraﬃc is aperiodic. This is the case, because it is not possible to predict when UEs will need resources.To illustrate this eﬀect, let us consider a scenario with N = 300 users that generate aperiodic traﬃc(λ = 0.1 packets). We shall then suppose that users request a maximum latency of 1 ms and a reliabilitylevel equal to 1–10−5. Satisfying this demand requires reserving 300 RBs (one per UE) in a 1 ms timewindowA lower number of resources would be necessary if traﬃc was periodic and we could estimatewhen each UE would require resources for their transmission. In this case, several UEs could sharethe same RB if they generate their packets at diﬀerent time instants. This would reduce the totalnumber of RBs necessary to serve all users. This is not possible in the case of aperiodic traﬃc, since175Sensors 2019, 19, 3575we cannot predict when a UE would need radio resources. Figure 7 shows that our implementationof 5G NR grant-free scheduling with 4-repetitions and shared resources can support 300 UEs (withtheir latency and reliability demands) with only 48 RBs in a time window of 1 ms. This is 84% lessradio resources than if we reserve dedicated resources per UE (with aperiodic traﬃc) for their ﬁrsttransmission using semi-persistent scheduling. These results clearly show that the implemented5G NR grant-free scheduling with shared resources can better support URLLC applications withaperiodic traﬃc and stringent communication requirements than other existing proposals. However,the conducted analysis (e.g., Figure 7) has also shown that new solutions will be needed to guaranteevery demanding reliability and latency levels such as those foreseen for some URLLC services in 3GPPRelease 16.6. ConclusionsThis paper has analyzed the capacity of 5G NR grant-free scheduling to support URLLC serviceswith strict reliability and latency levels such as those demanded by Industry 4.0. The study hasfocused on aperiodic or sporadic traﬃc and an implementation of 5G NR grant-free scheduling withK-repetitions and shared radio resources. This implementation has been chosen, since sharing radioresources is an attractive option for aperiodic traﬃc. In addition, the K-repetitions scheme can combatpossible packet collisions betweenUEs that share radio resources. This study has analyzed the reliabilityand latency levels that can be achieved with existing 5G NR grant-free scheduling with shared radioresources as a function of the number of UEs, the number of reserved radio resources, and the number ofreplicas K. To this aim, this study has derived analytical expressions that quantify the exact probabilityof collision with packets transmitted by other UEs, and the impact of self-collisions. It is importantto emphasize that this study is the ﬁrst one that has evaluated the impact of self-collisions. Packetcollisions and self-collisions have then been taken into account to derive analytically the reliability thatcan be achieved by existing 5G NR grant-free scheduling with shared resources. The derived analyticalexpressions have been validated against simulations. These expressions are a valuable contribution tothe community, since they can be easily utilized to evaluate 5G NR grant-free scheduling.This study has demonstrated that current 5G NR grant-free scheduling solutions cannot guaranteehigh reliability levels with strong latency requirements. This is partly due to the fact that strong latencyrequirements limit the number of replicas K that can be transmitted. In addition, self-collisions have anon-negligible impact that even limits the reliability that can be achieved when K increases. The impactof self-collisions also increases with the latency requirements. The obtained results demonstratethat new solutions are necessary for 5G NR grant-free scheduling to be able to support applicationswith stringent URLLC latency and reliability requirements under the presence of aperiodic traﬃc.In particular, the transmission of K replicas per packet might be inadequate to support aperiodic traﬃcwith very low latency levels due to the impact of self-collisions. Consequently, other approaches shouldbe designed to minimize collisions between UEs sharing radio resources. This study has shown thatthese new solutions cannot be based either on grant-based or semi-persistent scheduling. Grant-basedscheduling introduces additional latency due to the exchange of messages between the UEs and the BSfor assigning the radio resources. Semi-persistent scheduling with dedicated resources ineﬃcientlyutilizes the available resources when considering dedicated resources and aperiodic traﬃc. Innovativegrant-free scheduling solutions are hence necessary to meet the URLLC requirements identiﬁed for3GPP Release 16 and beyond. This could include, for example, the use of sensing mechanisms or fullduplex techniques that can reduce packet collisions.Author Contributions: Conceptualization, M.C.L.-E. and J.G.; methodology, M.C.L.-E. and J.G.; validation,M.C.L.-E. and J.G.; formal analysis, M.C.L.-E., J.G. and M.S.; investigation, M.C.L.-E.; writing—original draftpreparation, M.C.L.-E.; writing—review and editing, J.G. and M.S.; funding acquisition, J.G. and M.S.Funding: This work has been funded by the European Commission through the FoF-RIA Project AUTOWARE:Wireless Autonomous, Reliable and Resilient Production Operation Architecture for Cognitive Manufacturing176Sensors 2019, 19, 3575(No. 723909), and the Spanish Ministry of Economy, Industry, and Competitiveness, AEI, and FEDER funds(TEC2017-88612-R).Conﬂicts of Interest: The authors declare no conﬂict of interest.References1. 3GPP. Technical Speciﬁcation Group Services and System Aspects; Study on Communication for Automation inVertical Domains; 3GPP: Sophia Antipolis, France, 2018; Release 16, 3GPP TR 22.804 V16.2.0.2. 3GPP. Technical Speciﬁcation Group Radio Access Network; Study on Physical Layer Enhancements for NRUltra-Reliable and Low Latency Case (URLLC); 3GPP: Sophia Antipolis, France, 2018; Release 16, 3GPP TR38.824 V1.0.0.3. Klessig, H.; Ashraf, S.A.; Almeroth, B.; Riedel, I.; Puschmann, A.; Elste, T.; Simsek, M.; Schulz, P.; Matthe, M.;Fettweis, G.; et al. Latency Critical IoT Applications in 5G: Perspective on the Design of Radio Interface andNetwork Architecture. IEEE Commun. Mag. 2017, 55, 70–78.4. 5G Alliance for Connected Industries and Automation (5G-ACIA). 5G for Connected Industries and Automation,2nd ed.; 5G Alliance for Connected Industries and Automation (5G-ACIA): Frankfurt, Germany, 2019.5. Popovski, P.; Nielsen, J.J.; Stefanovic, C.; De Carvalho, E.; Ström, E.; Trillingsgaard, K.F.; Bana, A.-S.;Kim, D.M.; Kotaba, R.; Park, J.; et al. Wireless Access for Ultra-Reliable Low-Latency Communication:Principles and Building Blocks. IEEE Netw. 2018, 32, 16–23. [CrossRef]6. 3GPP. Technical Speciﬁcation Group Radio Access Network; Evolved Universal Terrestrial Radio Access (E-UTRA);Study on Latency Reduction Techniques for LTE; 3GPP: Sophia Antipolis, France, 2016; 3GPP TR 36.881V14.0.0.7. 3GPP. Technical Speciﬁcation Group Radio Access Network; NR; Medium Access Control (MAC) ProtocolSpeciﬁcation; 3GPP: Sophia Antipolis, France, 2018; Release 15, 3GPP TS 38.321 V15.4.0.8. Berardinelli, G.; Mahmood, N.H.; Abreu, R.; Jacobsen, T.; Pedersen, K.; Kovacs, I.Z.; Mogensen, P. ReliabilityAnalysis of Uplink Grant-Free Transmission Over Shared Resources. IEEE Access 2018, 6, 23602–23611.[CrossRef]9. 3GPP. Discussion on Conﬁgured Grant for NR-U; 3GPP: Sophia Antipolis, France, 2018; 3GPP TSG-RAN,R1-1810329.10. Li, Z.; Uusitalo, M.A.; Shariatmadari, H.; Singh, B. 5G URLLC: Design Challenges and System Concepts.In Proceedings of the 2018 15th International Symposium on Wireless Communication Systems (ISWCS),Lisbon, Portugal, 28–31 August 2018; pp. 1–6.11. 3GPP. Technical Speciﬁcation Group Radio Access Network; NR; Physical Layer Procedures for Data; 3GPP:Sophia Antipolis, France, 2018; Release 15, 3GPP TS 38.214 V15.4.0.12. Wu, Y.; Zhang, L.; Wang, C.; Chen, Y. Performance Evaluation of Grant-Free Transmission for Uplink URLLCServices. In Proceedings of the 2017 IEEE 85th Vehicular Technology Conference (VTC Spring), Sydney,Australia, 4–7 June 2017; pp. 1–6.13. Kotaba, R.; Manchon, C.N.; Balercia, T.; Popovski, P. Uplink Transmissions in URLLC Systems with SharedDiversity Resources. IEEE Wirel. Commun. Lett. 2018, 7, 590–593. [CrossRef]14. Abreu, R.; Mogensen, P.; Pedersen, K.I. Pre-Scheduled Resources for Retransmissions in Ultra-Reliable andLow Latency Communications. In Proceedings of the 2017 IEEE Wireless Communications and NetworkingConference (WCNC), San Francisco, CA, USA, 19–22 March 2017; pp. 1–5.15. Singh, B.; Tirkkonen, O.; Li, Z.; Uusitalo, M.A. Contention-Based Access for Ultra-Reliable Low LatencyUplink Transmissions. IEEE Wirel. Commun. Lett. 2018, 7, 182–185. [CrossRef]16. Jacobsen, T.; Abreu, R.; Berardinelli, G.; Pedersen, K.; Mogensen, P.; Kovacs, I.Z.; Madsen, T.K. SystemLevel Analysis of Uplink Grant-Free Transmission for URLLC. In Proceedings of the 2017 IEEE GlobecomWorkshops (GC Wkshps), Singapore, 4–8 December 2017; pp. 1–6.17. 3GPP. Grant-Free Transmission for UL URLLC; 3GPP: Sophia Antipolis, France, 2017; 3GPP TSG-RAN,R1-1706919.18. 3GPP. Enhancement of Uplink Grant-free transmission for NR URLLC; 3GPP: Sophia Antipolis, France, 2018;3GPP TSG-RAN, R1-1810176.19. 3GPP. Enhancement of Conﬁgured Grant for NR URLLC; 3GPP: Sophia Antipolis, France, 2018; 3GPP TSG-RAN,R1-1812162.177Sensors 2019, 19, 357520. 3GPP. Technical Speciﬁcation Group Radio Access Network; NR; Service Requirements for the 5G System; 3GPP:Sophia Antipolis, France, 2018; Stage 1, Release 15, 3GPP TR 22.261 V15.7.0.21. 3GPP. Technical Speciﬁcation Group Radio Access Network; NR; Physical Channels and Modulation; 3GPP:Sophia Antipolis, France, 2018; Release 15, 3GPP TR 38.211 V15.4.0.22. 3GPP. Technical Speciﬁcation Group Radio Access Network; Study on Scenarios and Requirements for NextGeneration Access Technologies; 3GPP: Sophia Antipolis, France, 2018; Release 15, 3GPP TR 38.913 V15.0.0.23. Bennis, M.; Debbah, M.; Poor, H.V. Ultrareliable and Low-Latency Wireless Communication: Tail, Risk,and Scale. Proc. IEEE 2018, 106, 1834–1853. [CrossRef]24. 3GPP. Evaluation of Latency in LTE; 3GPP: Sophia Antipolis, France, 2017; 3GPP TSG-RAN, R1-1720535.© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open accessarticle distributed under the terms and conditions of the Creative Commons Attribution(CC BY) license (http://creativecommons.org/licenses/by/4.0/).178sensorsArticleNoninvasive Suspicious Liquid Detection UsingWireless SignalsJiewen Deng 1, Wanrong Sun 1, Lei Guan 2, Nan Zhao 1, Muhammad Bilal Khan 1, Aifeng Ren 1,Jianxun Zhao 1, Xiaodong Yang 1,* and Qammer H. Abbasi 31 School of Electronic Engineering, Xidian University, Xi’an 710071, China; djw15529256085@163.com (J.D.);sunwanrong@xidian.edu.cn (W.S.); nan_zhao_@hotmail.com (N.Z.); engrmbkhan1986@gmail.com (M.B.K.);afren@mail.xidian.edu.cn (A.R.); jxzhao@xidian.edu.cn (J.Z.)2 School of Life Sciences and Technology, Xidian University, Xi’an 710126, China; 15926395470@163.com3 School of Engineering, University of Glasgow, Glasgow G12 8QQ, UK; Qammer.Abbasi@glasgow.ac.uk* Correspondence: xdyang@xidian.edu.cnReceived: 20 July 2019; Accepted: 15 September 2019; Published: 21 September 2019Abstract: Conventional liquid detection instruments are very expensive and not conducive tolarge-scale deployment. In this work, we propose a method for detecting and identifying suspiciousliquids based on the dielectric constant by utilizing the radio signals at a 5G frequency band. There arethree major experiments: ﬁrst, we use wireless channel information (WCI) to distinguish betweensuspicious and nonsuspicious liquids; then we identify the type of suspicious liquids; and ﬁnally,we distinguish the diﬀerent concentrations of alcohol. The K-Nearest Neighbor (KNN) algorithm isused to classify the amplitude information extracted from theWCImatrix to detect and identify liquids,which is suitable for multimodal problems and easy to implement without training. The experimentalresult analysis showed that our method could detect more than 98% of the suspicious liquids, identifymore than 97% of the suspicious liquid types, and distinguish up to 94% of the diﬀerent concentrationsof alcohol.Keywords: 5G; liquid detection; radio propagation; dielectric constant; WCI1. IntroductionThe illegal carrying and transportation of ﬂammable and explosive liquids seriously aﬀects publicsafety. Flammable and explosive liquids such as gasoline and alcohol are also commonly used invarious terrorist activities. Therefore, the safety inspection of ﬂammable and explosive liquids is ofgreat signiﬁcance for ensuring public safety. It has broad application prospects in the ﬁelds of publicsecurity, civil aviation, and customs. Liquids cannot be identiﬁed by the naked eye, and it is diﬃcultto carry out dangerous liquid inspection in crowded places, which is a great challenge for securitypersonnel. Moreover, there are still some places where the “taste liquid” method is used to determinewhether the liquid is safe [1]. Forbid passengers to carry large amounts of liquid has become themain method to prevent terrorist attacks. For example, international civil aviation regulations prohibitcarrying more than 100 mL of liquid, and trains and high-speed trains prohibit carrying ﬂammableliquids and more than 120 mL of compressed spray. The traditional detection of suspicious items iseither manual inspection (such as setting checkpoints at each entrance) or special equipment (such assurveillance cameras, X-ray machines, and ultra-wideband scanners), which is costly, expensive todeploy, and diﬃcult to implement on a large scale [2]. It is necessary to introduce a new suspiciousliquid detection scheme which is more economical and covers a wider range.There are several mature liquid detection technologies. A traditional approache, the Ramanspectrum analysis, uses the molecular structure to identify liquids according to their scatteringspectroscopy. Raman spectroscopy has many unique advantages, such as wide detection range, sharpSensors 2019, 19, 4086; doi:10.3390/s19194086 www.mdpi.com/journal/sensors179Sensors 2019, 19, 4086spectral peaks, and high resolution [3]. That is, when monochromatic light radiates on an object,the molecules of the substance will scatter, and the spectrum reﬂected by diﬀerent substances will bediﬀerent. Moreover, the method, based on X-ray image technology, is used to identify liquid substances,and can obtain the atomic number of the liquid through its X-ray [4]. However, this method has acertain error rate and the equipment is very expensive. Another method is to detect liquids basedon the diﬀerent absorption and attenuation characteristics of diﬀerent substances via electromagneticwaves [5,6]. Microwave detection can achieve noncontact detection with a certain distance betweenthe detected objects, and has high detection sensitivity. However, these approaches rely on expensiveand specialized equipment, which does not facilitate wide deployment in practice.Recently, Radio Frequency (RF) based sensing has drawn considerable attention. A couple ofstudies have explored the feasibility of using RF signals for remote monitoring and controlling duringinfusion. For instance, a wireless intelligent monitoring system, based on wireless communicationand network technology, is put forward, which can monitor drip speed in real time and automaticallyalarm in abnormal conditions [7]. The ZigBee wireless sensor is used to detect the velocity of liquiddroplets [8]. While these approaches mainly focus on exploiting the diﬀerences of wireless signalmeasurements to sense physical morphological changes of liquids, using ﬁne-grained wireless channelinformation (WCI) to identify types of liquid remains an option.5G communication technology is a data and information transmission technology developedby technicians based on 4G technology. Its advantages include sound transmission performance,fast transmission speed, high utilization rate of resources, and wide range of coverage. As such,it is favored in modern data and information transmission [9]. Based on ubiquitous wireless signals,the wireless sensing system will provide a variety of high precision, high reliability, high security,and convenient application services, among which the human behavior identiﬁcation technology is atthe core for public use [10]. As the information of the physical layer, the ﬁne-grained WCI contains alot of channel information which is invisible to the Medium Access Control (MAC) layer. The WCI canmeasure the frequency response of multiple subcarriers at the same time from a single packet, ratherthan the overall amplitude response superimposed by all subcarriers. Information about the WCI andfrequency-selective channels is described [11].In the ﬁeld of wireless sensing, WCI has gradually become a popular area of research [12], such asindoor positioning [13–15], respiration detection [16–18], and behavior recognition [19,20]. Zhou et al.proposed using the WCI to detect the presence of people in the environment [21]. Although wirelesssignals have good applications in indoor positioning and fall detection, there are still relatively fewstudies on detecting the types and concentrations of liquids by using them. Liquid interferes withthe path of radio signals. Diﬀerent liquids have diﬀerent degrees of interference to the radio signalpropagation path due to their diﬀerences in dielectric constants, which leads to diﬀerent changes in theWCI. These changes can be eﬀectively observed at the signal receiver, so as to realize the detection andidentiﬁcation of various liquids. Table 1 shows the relative dielectric constants of common objects.Table 1. The relative dielectric constants of common objects.Object Dielectric ConstantWater 80Alcohol 24Oil 2Glycerol 37Methanol 32Sulfuric Acid 84Following this introduction, the paper is organized into four sections. Section 2 describes thepreparatory work. Section 3 details the band selection and method design. Results are analyzed inSection 4 and conclusions are drawn in Section 5.180Sensors 2019, 19, 40862. Preparatory Work2.1. Data AcquisitionThe WCI represents the channel state of a communication link on Orthogonal Frequency DivisionMultiplexing (OFDM) technology. The WCI describes how links are transmitted from a transmitterto a receiver. It also combines the inﬂuence of scattering, fading, power attenuation, and otherfactors. The WCI reﬂects the performance of a link and the interference caused by other factors to agreat extent [22]. The WCI is composed of an OFDM matrix containing 30 subcarriers. Data can betransmitted simultaneously on multiple subcarriers, greatly improving the eﬃciency and accuracy ofthe system [23]. The frequency domain of the wireless channel can be expressed as:→Y = H · →X +→N, (1)where→X and→Y are the transmitted and received vectors while→N is Gaussian noise (AWGN) vectorand H represents the frequency response of the channel.The receivers receive packets for each subcarrier from a channel. The packet carrying the originalamplitude and random phase information complex frequency domain can be expressed as:H(n) =∣∣∣ H(n)∣∣∣ e∠H(n). (2)In Equation (2), H(n) is the data for subcarrier number n, where n ∈ [1 to 30]. |H(n)| is the rawamplitude information and ∠H(n) denotes the random phase data.2.2. Data PreprocessingOur system uses a Hampel ﬁlter to eliminate the singular value in the data and construct a scalesequence with the median. Assuming that the median of the sequence is Z,{d(k)}= {|x0 (k) −Z∣∣∣, . . . , ∣∣∣xm−1(k)−Z|}. (3)The deviation scale of each data from the reference value is given. Suppose the median of {d(k)} isD. The median has an absolute deviation of:MAD = 1.4826 × D. (4)MAD can replace the standard deviation σ. The Hampel ﬁlter uses m data in a mobile window todetermine the validity of current data. If the data is valid, process it; otherwise, replace it with themedian. The Hampel ﬁlter can protect the detailed information while ﬁltering the singular value.2.3. Classiﬁcation MethodWe used the K-Nearest Neighbor (KNN) algorithm to classify the amplitude information extractedfrom the WCI matrix to detect liquids. The KNN algorithm mainly relies on the surrounding adjacentsamples to determine the category. If the k closest neighbors of a sample belong to a certain categoryin the feature space, the sample also belongs to the same category. In the KNN algorithm, the selectedneighbors are all objects that have been correctly classiﬁed. Therefore, the KNN method is moresuitable than other algorithms for the sample sets to be divided with a lot of crossover or overlap ofthe class domain.The KNN algorithm takes the distance between objects as the nonsimilarity index to avoidthe matching problem between objects. Using diﬀerent distance calculation methods, there may besigniﬁcant diﬀerences between the “neighbors” identiﬁed. We used the Euclidean distance method:181Sensors 2019, 19, 4086d (x, y) =√∑nk=1(xk − yk)2, (5)In the KNN algorithm, the choice of the K value will have a signiﬁcant impact on the classiﬁcationresults. Generally, we can take a relatively small value of K, and cross validation is used to select thebest value of K. Usually, K is an integer less than 20.3. Method Design3.1. Band SelectionThe C-band is a frequency band from 4.0 to 8.0 GHz, which is used as the frequency band fordownlink transmission of communication satellite signals. In the application of satellite televisionbroadcasting and various small satellite ground stations, the band was ﬁrst adopted and has beenwidely used. The Ministry of Industry and Information Technology (MITT) has announced thefrequency band division of China’s ﬁfth generation of mobile communications in its latest oﬃcialdocument [24]. The 4.8–5 GHz frequency band in China’s 5G is located in the C-band, which is animportant part of 5G communication in China.The S-band refers to the electromagnetic wave band with a frequency range of 2–4 GHz, which ismainly used in relaying, satellite communication, radar, and so on. Now widely used in Bluetooth,ZigBee, wireless routing, and wireless mouse devices also use S-band electromagnetic waves.Many researches have indicated that the longer the wavelength of the electromagnetic wave, thestronger its ability to diﬀract. For example, radio waves can be transmitted around tall buildings,and red light can travel far in fog to remind drivers, which is more eﬀective than green light andyellow light. The shorter the wavelength of the electromagnetic wave, the greater the energy of thewave, and the stronger the penetrating capacity. As such, X-ray can penetrate through skin and bones,ultraviolet rays can kill bacteria, and strong ultraviolet rays can cause skin cancer.Therefore, we selected electromagnetic waves of two frequency bands to study the inﬂuence ofelectromagnetic waves of diﬀerent bands on liquid detection. We chose a 2.4 GHz signal located in theS-Band and a 4.8 GHz signal located in the C-band (5G frequency band) for comparison.3.2. Method DesignThe experimental scenario used to facilitate the detection and identiﬁcation of suspicious liquidsis shown in Figure 1. Our experiment was conducted in a conference room and used two sets ofequipment placed on a desk to collect data. The transmitter and receiver were one meter apart.The liquid to be detected was placed statically between the transmitter and receiver at the same height.One set of equipment worked at 2.4 GHz, and the transmission and reception of signals were completedby the wireless network adapter of the computer and three omnidirectional antennas. The bandwidthwas 20 MHz, and the output power of the transmitter was set at −5 dBm. The other worked at 4.8 GHz,which is consistent with the 5G standard in China, with the RF signal generator as the transmitterand spectrum analyzer as the receiver. The bandwidth was 100 MHz, and the output power of thetransmitter was set at −5 dBm. The receiver collected the RF signal at the frequency of 4.8 GHz and2.4 GHz, corresponding to the C-band (5G) and the S-band, respectively.To facilitate the liquid detection and identiﬁcation leveraging Wi-Fi signal, we exploited wirelesschannel information (WCI), the ﬁne-grained description of the wireless channel, to capture the minutediﬀerences of the channel state change introduced by diﬀerent liquids. Our method employed theamplitude information of WCI. First, we extracted the WSI from a pair of transmitters and receivers.Second, we preprocessed the data to remove the environmental noise and eliminate the singular value.Then the data were feature selected by a Principal Components Analysis (PCA) algorithm and classiﬁedby a K-Nearest Neighbor (KNN) algorithm. The ﬂow chart of the method is shown in Figure 2.182Sensors 2019, 19, 4086? ????? ????????????????????????????????????????????????????????????????????????????????????????????????????Figure 1. (a) Experimental scenario; (b) The actual scene.Figure 2. Method ﬂow chart. KNN, K-Nearest Neighbor.Using this method, we carried out three experiments. Experiment 1 was to distinguish betweensuspicious and nonsuspicious liquids, Experiment 2 was to identify the type of suspicious liquids,and Experiment 3 was to distinguish the three diﬀerent concentrations of alcohol.4. Evaluation and AnalysesIn this work, containers made of three common materials were selected. Figure 3a shows ourselection of containers, from left to right a paper cup, a plastic bottle, and a glass bottle. We used 50%alcohol, 75% alcohol, 95% alcohol, oil, and a compressed spray as the representative of suspiciousliquid. We chose water as the representative of nonsuspicious liquid. Figure 3b shows the suspiciousliquids selected for the experiment.? ????? ????Figure 3. (a) The selection of containers; (b) Suspicious liquids selected from the experiment.183Sensors 2019, 19, 40864.1. Detection of Suspicious and Nonsuspicious LiquidsIn this section, we ﬁrst analyze the detection of suspicious and nonsuspicious liquids.Figure 4 shows the raw WCI amplitude information on 30 subcarriers when using C-bandelectromagnetic wave signals. We can see the diﬀerence of amplitude change between suspiciousliquids and nonsuspicious liquid intuitively.???? ???? ????????????????????? ?? ?? ????????????????????????????????????????????????????????????????????????????????? ?? ?? ?????????????????????????????????????????????????????????????????????????????????????? ?? ?? ???????????????????????????????????????????????????????????????????Figure 4. Amplitude information of 30 subcarriers of Step 1 at the C-band. (a) Using the paper cup;(b) Using the plastic bottle; (c) Using the glass bottle.The conclusion of the S-band was the same, as shown in Figure 5.? ????? ???? ????????????????????? ?? ?? ????????????????????????????????????????????????????????????????????????????????????? ?? ?? ?????????????????????????????????????????????????????????????????????????????????????????? ?? ?? ???????????????????????????????????????????????????????????????????????Figure 5. Amplitude information of 30 subcarriers of Step 1 at the S-band. (a) Using the paper cup;(b) Using the plastic bottle; (c) Using the glass bottle.Figure 6. The KNN algorithm Classiﬁcation results of Step 1 at the C-band and the S-band.184Sensors 2019, 19, 4086The KNN classiﬁcation algorithm was used to further detect suspicious and nonsuspicious liquids.The classiﬁcation results of the KNN algorithm for two bands of data are shown in Figure 6. Blue,orange, and grey represent the paper cup, the plastic bottle, and the glass bottle, respectively. In theC-band environment, the detection accuracy of Step 1 was over 98%, and that for S-band was 99%.4.2. Identiﬁcation of Suspicious LiquidsIn this section, we will analyze the identiﬁcation of suspected liquids.Figure 7 shows theWCI amplitude information of 30 subcarriers. Blue, orange, and green representalcohol, oil, and compressed spray, respectively. It can be seen that in the C-band environment, the WCIamplitude ranges of alcohol and oil were not very diﬀerent, but the amplitude ﬂuctuation trends wereobviously diﬀerent, and the amplitude ranges of these two categories were greatly diﬀerent from thoseof oil.? ????? ???? ????????????????????? ?? ?? ???????????????????????????????????????????????????????????????????????????? ?? ?? ????????????????????????????????????????????????????????????????????????????? ?? ?? ??????????????????????????????????????????????????????????Figure 7. Amplitude information of 30 subcarriers of Step 2 at the C-band. (a) Using the paper cup;(b) Using the plastic bottle; (c) Using the glass bottle.In the S-band environment, the WCI amplitude ranges of the three kinds of suspected liquidsdiﬀered greatly, as shown in Figure 8.???? ???? ????????????????????? ?? ?? ?????????????????????????????????????????????????????????????????????????????? ?? ?? ??????????????????????????????????????????????????????????????????????????????????? ?? ?? ??????????????????????????????????????????????????????????????Figure 8. Amplitude information of 30 subcarriers of Step 2 at the S-band. (a) Using the paper cup;(b) Using the plastic bottle; (c) Using the glass bottle.Figure 9 is the result of the KNN classiﬁcation algorithm in identifying the types of suspiciousliquids. In the C-band environment, the system can achieve more than 97% accuracy in identifyingtypes of suspicious liquid (Step 2), and that for the S-band environment is 99%.185Sensors 2019, 19, 4086Figure 9. The KNN algorithm Classiﬁcation results of Step 2 at the C-band and the S-band.4.3. Detection of Diﬀerent Concentrations of AlcoholThe dielectric constant of diﬀerent liquids was quite diﬀerent. Even for the same liquid, diﬀerentconcentrations had a certain eﬀect on the dielectric constant. To verify this, we selected 50% alcohol,75% alcohol, and 95% alcohol to carry out the experiment. Due to the diﬀerent physical materials ofthe container, the WCI amplitude of each subcarrier was also aﬀected by diﬀerent containers in thesame band environment. In addition, the amplitude of the WCI varied with the same kind of containerat diﬀerent wavelengths.Figure 10 shows the WCI amplitudes of 30 subcarriers corresponding to diﬀerent concentrationsof alcohol in the C-band where the paper cup, plastic bottle, and glass bottle were detected. We cansee that when the object is in the C-band environment, various containers performed diﬀerentlywhen identifying diﬀerent concentrations of alcohol, but they still accurately identiﬁed diﬀerentconcentrations of alcohol. This experiment veriﬁes that the types of containers will not aﬀect thesystem’s identiﬁcation of liquids, and further illustrates the reliability of our system in detecting andidentifying diﬀerent liquids. The conclusion is also applicable to the experimental measurement in theS-band environment.? ????? ???? ????????????????????? ?? ?? ?????????????????????????????????????????????????????????????????????????????????????????????????????????? ?? ?? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ?? ?? ??????????????????????????????????????????????????????????????????????????????????????????????????Figure 10. The wireless channel information (WCI) amplitudes of 30 subcarriers corresponding todiﬀerent concentrations of alcohol at the C-band by using diﬀerent containers. (a) Using the paper cup;(b) Using the plastic bottle; (c) Using the glass bottle.Figure 11 shows the detection results of diﬀerent concentrations of alcohol in C-band and S-bandenvironments. From the ﬁgure, we can see that the detection accuracy of the C-band is higher than thatof the S-band for the detection of alcohol with diﬀerent concentrations. In the C-band frequency analysisexperiment, the accuracy of the system for the detection of diﬀerent concentrations of alcohol reachedmore than 91%, and that of S-band electromagnetic waves was up to 89%. Therefore, the C-bandelectromagnetic wave is superior to the S-band electromagnetic wave in the accurate detection ofdiﬀerent concentrations of the same liquid.186Sensors 2019, 19, 4086Figure 11. Detection results of diﬀerent concentrations of alcohol at the C-band and the S-band.As can be seen from Table 2, wireless sensing corresponding the 5G frequency band hadexcellent detection results for the detection and identiﬁcation of suspicious liquids, no matter whichcontainer was selected. Moreover, our method had better performance and robustness in detectingdiﬀerent concentrations of alcohol, and had more subtle diﬀerences in dielectric constant than existingWi-Fi technologies.Table 2. The KNN algorithm classiﬁcation results of the system.Band Selection C-Band S-BandContainer Paper Plastic Glass Paper Plastic GlassExperiment 1 0.98 0.99 0.98 0.99 0.99 0.99Experiment 2 0.99 0.97 0.99 0.99 0.99 0.99Experiment 3 0.91 0.91 0.94 0.81 0.80 0.895. ConclusionsIn this work, we explored the feasibility of using a wireless signal in a multi-band environmentto detect suspicious liquids. Our work is novel because it demonstrates that it is possible to detectsuspicious liquids accurately using radio signals without installing expensive liquid detectionmachines.Our system can not only detect whether the liquid is suspicious, but also further identify the types ofsuspicious liquids. In addition, we conﬁrmed the feasibility of liquid concentration detection by usingthe WCI at a 5G frequency band.The results analysis shows that our method can accurately detect suspicious and nonsuspiciousliquids (Experiment 1) with more than 98% accuracy, regardless of the type and size of containers,and can identify the type of suspicious liquids (Experiment 2) with more than 97% accuracy. For thedetection of alcohol with diﬀerent concentrations, the accuracy can reach up to 94%. This providesbetter performance and robustness than existing Wi-Fi technology.However, our method has limitations for liquids stored in metal containers. In that case,we recommend that security personnel intervene. Our method could be further improved by increasingthe number of suspicious liquids prohibited in public and looking for ways to reduce the impact ofmetal containers on liquid detection.Author Contributions: Conceptualization, J.D. and N.Z.; methodology, L.G.; software, J.D.; validation, L.G.;formal analysis, J.D.; writing—original draft preparation, J.D.; writing—review and editing, M.B.K.; supervision,W.S., A.R., J.Z., X.Y. and Q.H.A.; project administration, X.Y.; funding acquisition, X.Y.Funding: The work was supported in part by the Fundamental Research Funds for the Central Universities(No. JB180205).Conﬂicts of Interest: The authors declare no conﬂict of interest.187Sensors 2019, 19, 4086References1. Hu, L. Research on Liquid Classiﬁcation by CNN-SVM Model and WIFI Channel State Information.Master’s Thesis, Hunan University, Changsha, China, 2018.2. Wang, C.; Liu, J.; Chen, Y.; Liu, H.; Wang, Y. Towards in-baggage Suspicious Object Detection UsingCommodity WiFi. In Proceedings of the 2018 IEEE Conference on Communications and Network Security(CNS), Beijing, China, 30 May–1 June 2018.3. Jiang, X.; Qian, Y.T. Research on The Improvement of Raman Spectrum Qualitative Technique and ItsApplication in Airport Inspection. Shandong Inoustrial Technol. 2017, 351, 220–222.4. Wang, Y.S.; Li, B.L. Recognition of Liquid Substances in Containers Based on X-Ray Images. In Proceedingsof the 15th National Conference on Image and Graphics, Guangzhou, China, 10 December 2010; p. 5.5. Xu, X. Study on Microwave Detection Method for Quality of Liquid Food. Master’s Thesis, University ofElectronic Science and Technology of China, Sichuang, China, 2015.6. Ma, Y.MicrowaveDetection ofMixedLiquidComponents. Master’s Thesis, TaiyuanUniversity of Technology,Taiyuan, China, 2013.7. Xiao, W.; Tu, Y.; Wang, J.; Mao, Y.W. Design and Implementation of Wireless Intelligent Monitoring Systemfor Liquid Droplets. Electron. Meas. Technol. 2008, 1, 133–136.8. Yu, H.; Ding, B.; Sun, X.; Ding, Z. Design and Implementation of Infusion Monitoring System Based onZigBee Technology. Piezoelectrics Acoustooptics 2013, 35, 756–762.9. Chengming, Z. Practical Thinking of 5G Communication Technology Based on Internet of Things.Telecom World 2019, 26, 95–96.10. Wang, C.; Chen, S.; Yang, Y.; Hu, F.; Liu, F.; Wu, J. Literature Review on Wireless Sensing Wi-Fi Signal-BasedRecognition of Human Activities. Tsinghua Sci. Technol. 2018, 23, 203–222. [CrossRef]11. Yang, Z.; Liu, Y. Wi-Fi Radar: From RSSI to CSI. Commun. CCF 2014, 10, 11.12. Li, X.; Zhang, D.; Xiong, J.; Zhang, Y.; Li, S.; Wang, Y.; Mei, H. Training-Free Human Vitality MonitoringUsing Commodity Wi-Fi. ACM Interact. Mob. Wearable Ubiquitous Technol. 2018, 2, 121. [CrossRef]13. Xiang, L.; Li, S.; Zhang, D.; Jie, X.; Hong, M. Dynamic-MUSIC: Accurate device-free indoor localization.In Proceedings of theACM International Joint Conference on Pervasive&Ubiquitous Computing, Heidelberg,Germany, 12–16 September 2016.14. Wang, J.; Jiang, H.; Xiong, J.; Jamieson, K.; Xie, B. LiFS: Low Human-Eﬀort, Device-Free Localization withFine-Grained Subcarrier Information. In Proceedings of the International Conference on Mobile Computing& Networking, New York, NY, USA, 3–7 October 2016.15. Xiong, J.; Sundaresan, K.; Jamieson, K. ToneTrack: Leveraging Frequency-Agile Radios for Time-Based IndoorWireless Localization. In Proceedings of the 21st Annual International Conference on Mobile Computingand Networking, Paris, France, 7–11 September 2015.16. Liu, X.; Cao, J.; Tang, S.; Wen, J.; Guo, P. Contactless Respiration Monitoring Via Oﬀ-the-Shelf WiFi Devices.IEEE Trans. Mob. Comput. 2016, 15, 2466–2479. [CrossRef]17. Wang, H.; Zhang, D.; Ma, J.; Wang, Y.; Wang, Y.; Wu, D.; Gu, T.; Xie, B. Human respiration detectionwith commodity WiFi devices: Do user location and body orientation matter? In Proceedings of the ACMInternational JointConference onPervasive&UbiquitousComputing,Heidelberg, Germany, 12–16 September2016.18. Zhang, F.; Zhang, D.; Xiong, J.; Wang, H.; Niu, K.; Jin, B.; Wang, Y. From Fresnel Diﬀraction Model toFine-Grained Human Respiration Sensing with Commodity Wi-Fi Devices. ACM Interact. Mob. WearableUbiquitous Technol. 2018, 21, 53. [CrossRef]19. Wei, W.; Liu, A.X.; Shahzad, M.; Kang, L.; Lu, S. Understanding and Modeling of WiFi Signal Based HumanActivity Recognition. In Proceedings of the International Conference on Mobile Computing & Networking,Paris, France, 7–11 September 2015.20. Yan, W.; Jian, L.; Chen, Y.; Gruteser, M.; Liu, H. E-eyes: Device-free location-oriented activity identiﬁcationusing ﬁne-grained WiFi signatures. In Proceedings of the 20th Annual International Conference on MobileComputing and Networking, Maui, HI, USA, 7–11 September 2014.21. Zhou, Z.; Zheng, Y.; Wu, C.; Shang, G.L.; Liu, Y. Towards Omnidirectional Passive Human Detection.In Proceedings of the 2013 Proceedings IEEE INFOCOM, Turin, Italy, 14–19 April 2013.188Sensors 2019, 19, 408622. Wu, K. Wi-metal: Detecting metal by using wireless networks. In Proceedings of the IEEE InternationalConference on Communications, Kuala Lumpur, Malaysia, 22–27 May 2016.23. Zhong, S.; Huang, Y.; Ruby, R.; Lu,W.; Qiu, Y.X.; Wu, K.Wi-ﬁre: Device-free ﬁre detection usingWiFi networks.In Proceedings of the IEEE International Conference on Communications, Paris, France, 21–25 May 2017.24. Technology, M. The MIIT has Issued a Notice Concerning the Use of the 3300–3600 MHz and 4800–5000MHz Frequency Bands in The Fifth-generation Mobile Communication System. 2017. Available online: http://www.miit.gov.cn/n1146295/n1652858/n1652930/n3757020/c5907905/content.html (accessed on 15 July 2018).© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open accessarticle distributed under the terms and conditions of the Creative Commons Attribution(CC BY) license (http://creativecommons.org/licenses/by/4.0/).189sensorsArticleA Distributed Testbed for 5G Scenarios:An Experimental StudyMohammad Kazem Chamran 1, Kok-Lim Alvin Yau 1,*, Raﬁdah M. D. Noor 2 and Richard Wong 11 School of Science and Technology, Sunway University, Subang Jaya 47500, Malaysia;mohamma.c@imail.sunway.edu.my (M.K.C.); richardwtk@sunway.edu.my (R.W.)2 Department of Computer System and Technology, University of Malaya, Kuala Lumpur 50603, Malaysia;ﬁdah@um.edu.my* Correspondence: koklimy@sunway.edu.myReceived:16 September 2019; Accepted: 29 October 2019; Published: 19 December 2019Abstract: This paper demonstrates the use of Universal Software Radio Peripheral (USRP), togetherwith Raspberry Pi3 B+ (RP3) as the brain (or the decision making engine), to develop a distributedwireless network in which nodes can communicate with other nodes independently and makedecision autonomously. In other words, each USRP node (i.e., sensor) is embedded with separateprocessing units (i.e., RP3), which has not been investigated in the literature, so that each nodecan make independent decisions in a distributed manner. The proposed testbed in this paper iscompared with the traditional distributed testbed, which has been widely used in the literature. In thetraditional distributed testbed, there is a single processing unit (i.e., a personal computer) that makesdecisions in a centralized manner, and each node (i.e., USRP) is connected to the processing unit via aswitch. The single processing unit exchanges control messages with nodes via the switch, while thenodes exchange data packets among themselves using a wireless medium in a distributed manner.The main disadvantage of the traditional testbed is that, despite the network being distributed innature, decisions are made in a centralized manner. Hence, the response delay of the control messageexchange is always neglected. The use of such testbed is mainly due to the limited hardware andmonetary cost to acquire a separate processing unit for each node. The experiment in our testbed hasshown the increase of end-to-end delay and decrease of packet delivery ratio due to software andhardware delays. The observed multihop transmission is performed using device-to-device (D2D)communication, which has been enabled in 5G. Therefore, nodes can either communicate with othernodes via: (a) a direct communication with the base station at the macrocell, which helps to improvenetwork performance; or (b) D2D that improve spectrum efﬁciency, whereby trafﬁc is ofﬂoadedfrom macrocell to small cells. Our testbed is the ﬁrst of its kind in this scale, and it uses RP3 as thedistributed decision-making engine incorporated into the USRP/GNU radio platform. This workprovides an insight to the development of a 5G network.Keywords: D2D communication; 5G; sensor network; sensor; end-to-end delay; USRP; distributedmechanism; Raspberry Pi1. IntroductionFifth generation (5G) is a promising next-generation cellular network armed with new features,particularly device-to-device (D2D) communication that enables direct communication betweendevices without going through base stations (BSs). This helps to ofﬂoad trafﬁc from macrocell (MC)BSs to small cell (SC) (i.e., femtocell) BSs, as well as user equipment and devices, including sensors,while increasing network cell coverage via multihop transmission [1–3]. In 5G, a node can operateeither as a licensed user (or a primary user, PU) to utilize its licensed channels (or cellular channels),or as an unlicensed user (or a secondary user, SU) to explore and utilize white spaces, which are theSensors 2020, 20, 18; doi:10.3390/s20010018 www.mdpi.com/journal/sensors190Sensors 2020, 20, 18underutilized licensed channels (or cognitive channels) [4]. D2D enables nodes to access both cellularand cognitive channels to improve spectrum efﬁciency in order to improve data transmission rate andquality of service (QoS) [5–7].1.1. Our ContributionsAt present, the majority of the research related to 5G presents theoretical analysis [8–12] andsimulation studies [11–16]. In general, various theoretical state of the art and open issues are presentedin [8], the effects of ultra-densiﬁcation are investigated in [9], various network architectures, mediumaccess mechanisms, and open issues are presented in [10], as well as routing algorithms to achievelower interference and a balanced trafﬁc load amoung routes in 5G environment are investigatedin [11,12], respectively. In addition, trafﬁc ofﬂoading from backbone routes and the central controller todistributed nodes is investigated in [13,14], the transmission delay is predicted based on channel statesin [15], and the feasiblity of D2D in 5G environment is investigated in [16]. Some researchers conductproof of concept experiments; however, the focus is primarily on the physical layer, particularlyspectrum management in [2], interference mitigation in [17], channel sensing in [18], as well as on thedata link layer, particularly channel hopping (or switches) in [19].This study focuses on the networking aspect over a 5G-based platform using universal softwareradio peripheral with GNU radio (USRP/GNU radio) units and Raspberry Pi3 B+ (RP3) processors [20].GNU radio is an open source software that serves as development toolkit [21] in the platform (formore details see Section 2.3). There are two types of testbeds under investigation in this paper: (a) thetraditional testbed comprised of BSs or nodes connected to a single traditional processing unit using awired medium via a switch [2]; and (b) our distributed testbed in which each BS and node is embeddedwith a separate processing unit, namely Raspberry Pi3 B+ (RP3). We consider a 5G scenario with:(a) D2D communication; and (b) heterogeneous MC and small cell BSs with different sensing andtransmission capabilities, as well as processing capabilities (i.e., using operating systems with differentcapabilities). Comparison is made on the performance measures of routes via D2D and MC BS. The MCBS selects a route, and informs FC BSs and nodes about the route; subsequently, the FC BSs and nodessetup the route accordingly. Therefore, FC BSs and nodes can be relaxed from performing routeselection and channel sensing. Our testbeds are sufﬁcient for the investigation of our contributions,although further extension is suggested in Section 7.Our contributions are twofold:• Performance comparison achieved in our distributed testbed based on proof of conceptexperiments involving multihop transmission, which is necessary in next-generation wirelesssensor networks. The BSs and nodes are heterogeneous from MC and SCs with different sensingand transmission capabilities, as well as processing capabilities (i.e., using operating systems withdifferent capabilities).• Performance analysis of the software and hardware processing delays for communication viaD2D and going through BSs over the testbed, which is required for route selection.1.2. Signiﬁcance of This PaperThere are two main investigations in this paper. Firstly, comparison is made of the performancemeasures achieved by the traditional testbed and our distributed testbed. Our proposed testbedis distributed in nature and it has a closer resemblance to a real deployed network. The softwareand hardware processing delays, which are generally ignored in theoretical analysis and simulation,are investigated. Secondly, using the testbeds, comparison is made on the performance measuresachieved by: (a) the traditional direct communication with MC BS; and (b) the D2D communication.This comparison is useful for the MC BS to make decision on route selection. This is because, while D2Dcommunication can ofﬂoad trafﬁc from MC BSs to SC BSs, the end-to-end delay over a multihoptransmission increases, and so the direct communication with MC BS can be favorable. The end-to-end191Sensors 2020, 20, 18delay of a route changes with its operating environment (e.g., the processing capability) and itsoperation (e.g., the lower read and write rates of RP3 contribute to a higher end-to-end delay and lowerpacket delivery ratio in our distributed testbed compared to the traditional testbed). Lower end-to-enddelay is favorable to support real-time applications integrated with sensors, such as driverless vehicles.To the best of our knowledge, this is the ﬁrst USRP/GNU radio platform incorporated with RP3implementation with this scale and functionality.1.3. Organization of This PaperThe rest of this paper is organized as follows. Section 2 presents research background. Section 3presents related work. Section 4 presents system model and delay measurement. Section 5 presentsexperimental setup. Section 6 presents experimental results. Section 7 presents our conclusion andfuture work.2. BackgroundThis section presents an overview of 5G, USRP, GNU radio, and RP3.2.1. 5GThe 5G network is a heterogeneous network that consists of different kinds of network cells,including MC and femtocell (FC). The transmission of the BSs and nodes are characterized by differentfrequency bands and transmission power levels. In Figure 1, a 5G network consists of two mainplanes: (a) control plane consists of MC BSs, which use higher transmission power levels at lowerfrequency bands, contributing to larger transmission ranges; and (b) data plane consists of FC BSsand nodes, which use lower transmission power levels at higher frequency bands, contributing tosmaller transmission ranges [22]. The control plane communicates with the cloud, which consistsof a central controller (CC) that manages and coordinates global functions, such as route selection.MC BSs can coordinate and communicate among themselves via the cloud [12,23], and this helps themto determine the nodes that each of them must cover. The FC BSs can coordinate and communicateamong themselves via D2D if they are within each other’s transmission range, and this helps them to:(a) use a route established from a FC source node FCs to a FC destination node FCd by the CC; and (b)ofﬂoad trafﬁc from MC BSs. Both MC and FC overlap, and FC BSs can communicate with each otherdirectly. Hence, the MC BS in the control plane can select a route, and inform FC BSs and nodes inthe data plane about the route; subsequently, the FC BSs and nodes setup the D2D route accordingly.Therefore, FC BSs and nodes can be relaxed from performing complex tasks, such as route selection.192Sensors 2020, 20, 18Figure 1. A 5G network that consists of a single MC BS and a number of FC BSs.2.2. Universal Software Radio PeripheralUSRP is an off-the-shelf wireless device that can be conﬁgured with a wide range of operatingparameters, such as the types of modulation schemes and the channel frequency bands. Our testbeduses USRP N200 series that provides high processing capability. Figure 2 shows a USRP unit that hasa set of two omni-directional VERT900 antennas—one for transmission and one for reception—forsimultaneous transmissions in two different operating channels within channel frequency bands850–890 MHz and 2.3–2.4 GHz. The selected channel frequency bands include the television and globalsystem for mobile communication (GSM) bands. The antennas are connected to a daughterboard.There are two types of paths: (a) receive path in which analogue signals are received and moved fromthe radio frequency (RF) front end towards RP3 for reception; and (b) transmit path in which digitalsignals move from RP3 towards the RF front end for transmission.Figure 2. Transmit and receive paths between an antenna and a RP3 via a USRP/GNU radio unit. Solidarrow line is part of a receive path, and dotted arrow line is part of a transmit path.The USRP consists of three main sections as follows:• Wide bandwidth transceive (WBX) is the RF front end that provides access to different operatingchannels within a range of 50 MHz of RF bandwidth with 8 bit samples, or 25 MHz of RFbandwidth with 16 bit samples. The maximum transmission power is 100 mW (or 20 dBm) with anoise ﬁgure of 5 dB.193Sensors 2020, 20, 18• Converter consists of: (a) an analogue-to-digital converter (ADC) and a digital down converter(DDC) in the receive path; and (b) a digital-to-analogue converter (DAC) and a digital up converter(DUC) in the transmit path. DDC selects desired signals from an array of signals captured by ADC,while DUC increases the bandwidth of baseband signals so that they are compatible with DAC.• Field-programmable gate array (FPGA), speciﬁcally the Xilinx Spartan 3A-DSP 1800 board [24]used in this platform, consists of: (a) a decimation ﬁlter for achieving the required interfacebandwidth in the receive path, and an interpolation ﬁlter for achieving the opposite in thetransmit path; (b) a USRP hardware driver (UHD) block with a software interface that enablesvarious components to communicate among themselves; and (c) a processor block that performsencoding/decoding, modulation/demodulation, timing synchronization, and other signalprocesses required for software deﬁned radio (SDR) operations. The FPGA communicates withRP3 via power over Ethernet (PoE). It provides connection between: (a) gigabit Ethernet CAT5E-350 MHz cables, which provide a maximum data rate of 1000 megabits per second (Mbps)connected to a Gigabit switch; and (b) USB3, which provides a maximum data rate of 1600 Mbps.During system initialization, the kernel, which is the fundamental part of an operating system,of GNU radio controls and monitors programs and systems, as well as performs default functions,such as checking and assigning memory space to FPGA [24].2.3. GNU RadioGNU radio, together with its extended version called GNU radio companion (GRC), is an opensource SDR that enables users to design: (a) conﬁgurable blocks to perform communication tasks usingthe C++ language; and (b) ﬂow graphs to connect the blocks using the Python language. As shownin Figure 3, the blocks and ﬂow graphs deﬁne the roles of the source, intermediate, and destinationnodes as follows:• Source node, which is a RP3 unit with an Internet protocol (IP) address (e.g., 192.168.10.2) anda port number (e.g., 1234), generates and sends a data or video stream in the form of framesencapsulated in user datagram protocol (UDP). In GRC, the frames pass through three maincomponents: (a) an encoder that converts the frames into packets with a predeﬁned payload length(e.g., 1472 bytes); (b) a Gaussian minimum shift keying (GMSK) modulator that converts the packetsinto modulated signals at baseband (e.g., the minimum non-zero frequencies); and (c) a USRPsink block that sets the center frequency (e.g., 850 MHz), channel gain (e.g., 1dB), and sample rate(e.g., 1 MHz). Finally, the signals are broadcasted.• Intermediate node receives signals from a transmitter, which can be a source node or an upstreamintermediate node, and transmits them to the next-hop node, which can be a destination ora downstream intermediate node. There are two processes that help to improve the qualityof packets before forwarding them in order to reduce interference and address poor channelquality [25]: (a) to demodulate signals to packets, and then to decode packets to frames; and(b) to encode frames to packets, and then to modulate packets to signals. The demodulationand decoding processes are performed at the receiver unit, and then modulation and encodingprocesses at the transmitter unit.• Destination node, which is a RP3, receives and demodulates signals to packets, and then decodespackets to frames. Then, a UDP sink block sends the frames to an application (e.g., a VLC mediaplayer) through a port (e.g., port number 1236 or udp://@:1236).194Sensors 2020, 20, 18Figure 3. GNU radio ﬂow graph that consists of source, intermediate and destination nodes.2.4. Raspberry Pi3 B+Conventionally, a testbed consists of BSs or nodes connected to a single traditional processingunit (e.g., a personal computer or a laptop) using a wired medium via a switch [2] (see Section 5).The BSs and nodes exchange control messages and data packets over the wired and wireless media,respectively. In this paper, each BS and node is embedded with a separate processing unit, namely theRP3 unit. Both control messages and data packets are exchanged over the wireless medium.There are three main advantages. Firstly, ease of implementation because nodes can be placed furtherapart from each other rather than being constrained by physical cables and connections to a singleswitch. Secondly, higher cost efﬁciency (or lower overhead) because nodes do not communicate with atraditional processing unit. Thirdly, lower energy consumption with the use of RP3 compared to personalcomputers, laptops, and a switch.However, there are three main disadvantages. Firstly, lower processing capability. The RP3 processor(e.g., 1.4 GHz 64-bit quad-core processor with 1 GB non-expandable on-board RAM) is sufﬁce toperform basic tasks and support simple applications (e.g., running GRC in the Linux environment).Secondly, lower data rate. The network interface of RP3 has approximately 324 Mbps data rate only,which is low compared to 761 Mbps offered by the gigabit Ethernet of a CORE i7 personal computer.This increases the end-to-end delay of the communication between a RP3 and a USRP. In addition,RP3 is embedded with an SD card, such as a high capacity HC-I class 10 SD card that offers a datarate of 10 megabytes per second (MBps), which is low compared to 550 MBps offered by a solid-statehard drive. This increases the internal delay of the communication between a RP3 and an SD card.This is signiﬁcant because such communication is commonplace as an operating system (e.g., Ubuntu)is stored as an image in the SD card. It is worth mentioning that the speed (or rate) of read and writeon SD cards reduces with the increase of occupied space, and the read rate is generally higher thanthe write rate as shown in Section 5.3.2. Thirdly, lower storage space. The SD card provides low storagespace (e.g., 32 GB) for operating systems and software applications.3. Related WorkThis section presents related works on testbeds, particularly USRP/GNU radio platforms,for investigating the networking aspect of 5G. It covers two main topics. Firstly, the communicationdelay between nodes along a route. The routes are assumed to be readily available, and they areselected and provided by the central controller. Secondly, the testbeds, particularly USRP/GNU radioand RP3 platforms.3.1. Communication Delay between NodesIn [26], the hardware and software processing delays are investigated on a testbed comprised often USRP/GNU radio nodes connected to a single traditional processing unit (i.e., a personal computer)195Sensors 2020, 20, 18via a switch. The end-to-end delay has shown to reduce since the personal computer can pre-processroute selection prior to data transmission.In [27], the response delay (or the round-trip time) is investigated on a testbed comprised oftwo USRP/GNU radio nodes connected to a personal computer. The response delay is the durationbetween the moment the ﬁrst byte of a packet passes the digital signal processing block of a sender andthe moment the ﬁrst byte of an acknowledgement packet arrives at the sender. The delay is incurredin: (a) the initiation process that includes modulation, sampling, encoding, as well as packet transferbetween GNU radio and kernel (or the operating system); (b) the buffering process that collects andstores packets in a buffer (e.g., the buffer of a VLC media player); and (c) the transmission processthat receives and sends packets to the FPGA unit of USRP so that they are interpolated before beingtransmitted via antenna. Measurement shows that the initiation process has the highest time delay,and the transmission process has the lowest time delay.In [28], the hardware and software processing delays, as well as the response delay,are investigated on a testbed comprised of two USRP/SDR nodes, which serve as the source anddestination nodes, connected to a single traditional processing unit (i.e., a personal computer).The source node transmits a data packet to the destination node; and subsequently, the destinationnode returns a response packet to the source node. The delays are incurred in: (a) processes run in aUSRP/SDR node (e.g., operating system and the modulation process); and (b) communication betweenthe two nodes. Measurement shows that the software processing delay incurred in SDR is signiﬁcantlyhigher than the hardware processing delay incurred in USRP and the communication delay incurredbetween the two nodes.In [29], the response delay, which includes the waiting time of a packet in a queue, is investigatedon a testbed comprised of four USRP/GNU radio nodes connected to a single traditional processingunit (i.e., another USRP/GNU radio unit). There are a pair of PU transmitter and receiver, and anotherpair of SU transmitter and receiver. The SU transmitter must sense the operating channels beforetransmission so as not to interfere with the PUs. Up to 30% of the delay incurred in the SU transmitteris attributed to channel sensing, which can be reduced to increase throughput at the expense of higherinterference level to PUs. Hence, there is a tradeoff between the delay and throughput performances.In [30], the hardware and software processing delays of different processes are investigatedon a testbed comprised of two USRP/GNU radio nodes embedded with separate processing units(i.e., personal computers). The nodes are connected to each other via Ethernet. Examples of the USRPprocesses are the operating system processes in the kernel, and the decimation ﬁltering in FPGA; andan example of the GNU radio process is the modulation process. During measurement, a 1 μs guardtime is included between the processes. A node transmits a ping packet to another node. The packetmoves through the transmit and receive paths, and the timestamps for different processes in theUSRP/GNU radio node are recorded. Measurement shows that the hardware and software processingdelays are highest for processes running in the Kernel. This indicates that the USRP/GNU radioplatform has low efﬁciency providing low network performance, particularly high end-to-end delay.In this paper, the testbed is comprised of ﬁve USRP/GNU radio nodes embedded with separateprocessing units. Investigation is conducted on multihop transmissions in the network layer.3.2. Testbed of USRP/GNU Radio and Raspberry PiThis section presents related works on USRP/GNU radio with and without Raspberry Pi.3.2.1. USRP/GNU Radio without Raspberry PiIn [2,31–36], a testbed consists of BSs or nodes connected to a single traditional processing unit(e.g., a personal computer or a laptop) via a switch (i.e., Gigabit D-link) [2] (see Figure 4a). This allowsthe central controller to exchange control messages with BSs and nodes via a switch in a centralizedmanner, while the BSs and nodes can exchange data packets using the wireless medium [37] in adistributed manner. Examples of control messages include those that carry information about channel196Sensors 2020, 20, 18sensing and selection, route discovery and selection, and handshaking (e.g., request, acknowledgement,and responsemessages). Therefore, the response delay of a D2D type of communication in a real testbedis a cause of concern because of the sensitivity of wireless communication and the delay incurreddue to the distance between a node pair. The response delay is important in D2D communicationbecause if it may not fulﬁll the delay requirement (or higher than a threshold), MC BS must be used.This paper focuses on the response delay, which is end-to-end in nature, between a source node fromﬁrst transmitted packet up to last received one.Figure 4. Two scenarios in experimental setup.In this paper, each BS or node is embedded with a separate processing unit, particularly RP3as the core processing unit, to provide a more realistic wireless testbed, and so a single traditionalprocessing unit is not needed.3.2.2. USRP/GNU Radio with Raspberry PiIn [38], a single USRP/GNU radio embedded with Raspberry Pi3 is used to generate signalsin the range of FM radio frequency bands. In addition, a radio station also generates signals in thefrequency bands. Subsequently, the signals generated by the Raspberry Pi3 and radio station aremeasured using a spectrum analyzer, and a comparison is made. The quality of signals generated bythe USRP/GNU radio, despite using a lower transmission power at lower frequency bands, has shownto be close to that from a radio station. In general, the received signals from USRP/GNU radio has alower throughput and energy consumption.In [39], a single USRP/GNU radio node embedded with Raspberry Pi3 is used to performenergy-based channel sensing in order to detect PUs activities. There are two main sources of energyconsumption in Raspberry Pi3: (a) software initiation; and (b) the calculation of the available memoryof the kernel running GNU radio libraries. Channel sensing has shown to incur the highest delay.Similar investigation is performed in [20]. Energy consumption in Raspberry Pi3 has shown to beconsiderably lower compared to that in personal computer.197Sensors 2020, 20, 18In [40], a single register transfer level (RTL) dongle embedded with a Raspberry Pi transmitsand receives in the frequency bands 24–1850 MHz. The general-purpose input/output (GPIO) pinsof the Raspberry Pi is used to generate and transmit pulse width modulation (PWM). The energyconsumption of the RTL dongle embedded with Raspberry Pi has shown to be less than 3 watts, andso the dongle and Raspberry Pi can be powered by portable batteries. In addition, the use of RaspberryPi has shown to enable the detection of a wide range of frequency bands while incurring low energyconsumption.In [41], a single USRP/GNU radio node connected to a personal computer, which serves as thetransmitter, broadcasts signals to a RTL dongle embedded with a Raspberry Pi, which serves as thereceiver. The testbed consists of a low-cost radio community that transmit at two frequency bands,namely 915 MHz (or the ISM band) and 40.68 MHz (or the FM radio frequency band). The testbed hasdemonstrated the capability of Raspberry Pi for transmitting and receiving signals in these frequencybands, and the quality of reception depends on the transmission power and the height of the antennaof the transmitter.In [42], a testbed consists of two USRP/GNU radio nodes: (a) a static node, which is connectedto a personal computer, serves as the ground BS; and (b) a dynamic node, which is embedded withRaspberry Pi3, is installed on an unmanned aerial vehicle (or a drone). The ground BS receives locationinformation from the drones so that it can monitor the location of the drone. The ground BS anddrone exchange messages in the frequency bands 400–4400 MHz. The testbed has demonstrated thecapability of Raspberry Pi3 for setting up communication and processing information with lowerenergy consumption.In [43], a testbed consists of three main USRP/GNU radio nodes: (a) a SU source node, which isconnected to a personal computer; (b) a SU intermediate node, which is embedded with Raspberry Pi3,that performs energy-based channel sensing; and (c) a SU destination node, which is embedded withRaspberry Pi3. The rest of the nodes are PUs. The SU source node transmits data packets to the SUdestination node in multiple hops without interfering with the random PUs’ activities. The channelsensing delay incurred by the SU intermediate node embedded with Raspberry Pi3 has shown to betwice of that incurred by the SU source node connected to a personal computer.In this paper, our testbed consists of ﬁve USRP/GNU radio nodes embedded with Raspberry Pi3B+ (RP3) that constitutes a source node and four intermediate nodes. In addition, a personal computerserves as the destination node. The USRP/GNU radio performs communication, and the RP3 performsprocesses. While existing works in the literature [20,38–43], focus on the capability and compatibilityof USRP/GNU radio and RP3, this paper focuses on end-to-end hardware and software processingdelays between a source node and a destination node, and the use of the delay measurement for routeselection (i.e., either via D2D or MC BS).4. System Model and Delay MeasurementThis study measures network performance, particularly end-to-end delay, packet delivery ratio,and throughput, under different scenarios characterized by the characteristics of 5G, and comparesresults obtained from two types of testbeds, namely: (a) a testbed with a single traditional processingunit via a switch (see Figure 4a), (see Section 5.3.1); and (b) a testbed with separate processingunit (see Figure 4b), particularly RP3, embedded in each node and BS without using a switch(see Section 5.3.2).The rest of this section presents system model in Section 4.1, D2D link delay in Section 4.2.1, andD2D end-to-end delay in Section 4.2.2.4.1. System ModelThe system topology consists of femtocell nodes f c f ∈ { f c1, f c2, ..., f c|F|} located within thetransmission range of a MC BS. Nodes can transmit UDP packets in one of the routes in a route set,speciﬁcally kk ∈ K ={k1, k2, ..., k|K|}. Each link ln ∈ L = {l1, l2, ..., l|L|} uses one of the channels cc ∈198Sensors 2020, 20, 18C ={c1, c2, ..., c|C|}. Each route kk = ∪ ln ∈ L consists of a set of links from a femtocell source nodeto a femtocell destination node. a femtocell source node f cs sends packets to a femtocell destinationnode f cd along a primary route kk = ( f cs, f c1, f c2, f c3, f c4, f cd), which is D2D and multihop in nature.In a testbed with a single traditional processing unit, the femtocell nodes f cs, f c1, f c2, f c3, f c4, f cd areconnected to a MC BS, which serves as the centralized processor, via a switch as shown in Figure 4a(see Section 5.3.1 for more descriptions). On the other hand, in a testbed with separate processingunits, each femtocell node is embedded with a separate processing unit, namely RP3, as shown inFigure 4b (see Section 5.3.2 for more descriptions). In this paper, a primary route has up to ﬁve hops.The primary route uses cognitive channels (or white spaces in licensed channels), and the secondaryroute uses cellular channels (or the licensed channels). The use of primary routes helps to reduce thecongestion level of MC BS [4]. However, when the primary route becomes unavailable or broken,then a secondary route kk = ( f cs,mc1, f cd), which passes through the macrocell BS mc1. The routeselection between primary and secondary routes is shown in the form of a ﬂowchart in Figure 5 andan algorithm in Algorithm 1.Algorithm 1 Route selection between the primary route (via D2D) and the secondary route (via MC BS)1: procedure ROUTE SELECTION2: for k1 = ( f cs, f c1, f c2, f c3, f c4, f cd) and k2 = ( f cs,mc1, f cd) do3: if route k1 is available then4: f cs send packet to f c15: if f c1 is not available then6: packet goes through mc1 (secondary route) to f cd7: end if8: if f c2 is available then9: check the second condition:10: if t f c1to f c2 is ≤ α then11: f c1 send packet to f c212: end if13: end if14: if f c2 is not available then15: packet goes through mc1 to f cd16: end if17: if f c3 is available then18: check the second condition:19: if t f c2to f c3 is ≤ α then20: f c2 send packet to f c321: end if22: end if23: if f c3 is not available then24: packet goes through mc1 to f cd25: end if26: if f c4 is available then27: check the second condition:28: if t f c3to f c4 is ≤ α then29: f c3 send the packet to f c430: f c4 send the packet to destination f cd31: else packet goes through mc1 to f cd32: end if33: end if34: end if35: end for36: end procedureThe end-to-end delay tkk of a primary route kk increases with the number of hops [44], and it mustbe less than a threshold tkk < α, where α = 10 ms is imposed by the IEEE 802.15.4 standard [14,45].The secondary route is selected if the threshold is not fulﬁlled. The threshold α is imposed due to199Sensors 2020, 20, 18the need to reduce end-to-end delay in order to support and deploy real-time applications, includingapplications integrated with sensors such as driverless vehicles, in 5G. Long software and hardwareprocessing delays can increase the queue size of base stations and nodes, and so they affect networkperformance, such as reducing packet delivery ratio [46,47].In this paper, route selection is made by a central controller, and so the underlying routes, as wellas the channels of the links in the routes, are readily available. There is a single MC BS that selectsa route, and informs FC BSs and nodes about the route; subsequently, the FC BSs and nodes setupthe route accordingly. Further extension to the testbed, such as increasing the number of MC BSs,is suggested in Section 7. The investigation takes into account the effects of the characteristics of 5G,including heterogeneity that involves nodes with different features and characteristics (i.e., differenttransmission power, frequency range, and strength of operating system).Figure 5. Conditions for selecting a primary route (via D2D).4.2. Delay MeasurementThis section presents D2D link and end-to-end delay, respectively.4.2.1. D2D Link DelayThe D2D link delay (or per-hop delay) consists of three kinds of delays as shown in Figure 6. Firstly,the software processing delay tsln is incurred in GNU radio running on processing unit, such as personalcomputer and RP3, to process packets, such as encoding and modulating packets (see Section 2.3),over a link ln. Secondly, hardware processing delay thln is incurred in USRP to convert electrical packets toRF signals for transmission in the transmit path, and to convert RF signals to electrical packets uponreception in the receive path, over a link ln. Thirdly, the propagation delay tplnis incurred for the RF200Sensors 2020, 20, 18signal to travel from one USRP/GNU radio node to another over a link ln; however, it is negligiblecompared to software and hardware processing delays [27].The D2D link or per-hop delay tln for link ln ∈ L is as follows:tln = tsln + thln (1)Figure 6. D2D link delay for a single-hop transmission over a link ln. The processing unit can be eithera personal computer or a RP3.4.2.2. D2D End-to-End DelayThe D2D end-to-end delay tkk of a route kk ∈ K = {k1, k2, ..., k|K|} is the accumulation of the D2Dlink delay at each link, which consists of software and hardware processing delays, as follows:tkk = ∑ln∈kktln (2)5. Experimental SetupThis experiment investigates the link (or per hop) and end-to-end delays of a route via D2Dcommunication among heterogeneous BSs and nodes (i.e., MC and FCs) (see Section 4 for more details).5.1. Experiment ParametersThe experiment uses: (a) licensed channels, including the TV frequency bands (i.e., 850–890 MHz)and long term evolution (LTE) frequency bands (i.e., 2.3–2.4 GHz) [48]; and (b) unlicensedchannels, particularly the industrial, scientiﬁc and medical (ISM) frequency bands (i.e., 2.4 GHz).The experimental parameters are summarized in Table 1. In addition, the USRP parameters arepresented in Section 2.2, and the GNU radio parameters and ﬂow graph are presented in Section 2.3and Figure 3, respectively. In GNU radio, the bandwidth can be represented by sample rate (or thenumber of samples per second). The transmission power used for the range of frequency bands within850 and 890 MHz with a 1 dB set gain is 10 mW for 10 dBm.Table 1. Experimental parameters.Category Parameter PCU RPUExperiment Duration 300 s 300 sUSRP Number of channels 6 6Transport layer UDP UDPBandwidth 1.6 Mbps 1.6 MbpsTransmission power 10 dBm 10 dBmAntenna Carrier frequency 850 MHz 850 MHzComputer Operating system Ubuntu -Switch Number of units 1 -Number of inputs 6 -RP3 Operating system - Ubuntu-MatePoE Number of units - 5Number of inputs - 1201Sensors 2020, 20, 185.2. Experiment MeasurementThe software and hardware processing delays are measured using Wireshark [49], which is anopen source packet analyzer (or a packet sniffer) software. For each packet transmission, Wireshark isused to measure the delays incurred by the processing unit (i.e., the time period for a media stream tobe transformed into packets), the USRP (i.e., the time period for a packet to traverse from the USRPsink block to the antenna for transmission), and the propagation from one USRP unit to another. The IPaddress of the source node of the packet can be identiﬁed using Wireshark. The delay incurred byGNU radio (i.e., time period for a packet to traverse from the USRP source block to the USRP sinkblock) can be measured using Python, whereby the delay is given by the time difference between theUSRP source block and the USRP sink block.Meanwhile, there are two types of delays that are negligible: (a) the delay incurred for theinitialization of different software prior to converting frames into packets; and (b) the delay incurredfor signal propagation because the same transmit and receive components are used in both PCUand RPU.5.3. Experiment TestbedsThe testbed has a route with ﬁve hops as shown in Figure 7. A video stream in the form of UDPpackets is generated at a source node, forwarded along a route with four intermediate nodes, andreceived at a destination node. Video stream is chosen in this experiment due to its stringent QoScompared to data packets. Two types of testbeds are considered. Figure 4a shows a testbed with asingle processing unit (PCU), in which a number of heterogeneous MC and FC BSs connect to a singletraditional processing unit (i.e., a personal computer) using a wired medium via a switch. Figure 4bshows a testbed with separate processing units (RPU), in which the MC and FC BSs are embeddedwith separate processing units, namely RP3, and the FC BSs are located within the MC BS proximity.The rest of this section presents the two types of testbeds and their differences.Figure 7. An experimental setup for a D2D route with ﬁve hops using RP3 in the RPU testbed, whichis equivalent to Figure 4b.202Sensors 2020, 20, 185.3.1. Testbed with a Single Processing Unit (PCU)Figure 8a shows a testbed in which BSs or nodes are connected to a single traditional processingunit (PCU) (i.e., a personal computer) via a switch [2]. The personal computer has a CORE i7 processor,a 8 GB RAM, and a 1 terabyte of storage space in hard disk, and it provides a centralized approachto handle D2D communication. In other words, a single GNU radio ﬂow graph (see Figure 3 inSection 2.3) is installed in the personal computer. The BSs and nodes exchange control messages withthe personal computer via Gigabit Ethernet CAT 5E-350 MHz cables connected to a Gigabit switch,and data messages among themselves via wireless transmission. The control message has a packetsize of 826 megabytes and a payload size of 1472 bytes, and it contains the source node IP address,destination node IP address, available channels, acknowledgement, packet types, and timestamp.The source node incurs three types of delays: (a) the software processing delay incurred in the personalcomputer (i.e., to generate and segment video stream into UDP packets); (b) the software processingdelay incurred in GNU radio (see Section 2.3 for the processes); and (c) the hardware processing delayincurred in USRP (see Section 2.2 for the processes) and the propagation delay in the transmissionincurred from the current node to the next-hop node. Subsequently, the intermediate nodes do notincur the software processing delay in the personal computer. Based on Equation (2), the end-to-enddelay of a route kk ∈ K is as follows:tkk = (tPCl1 + tGNUl1 + thl1) + (tGNUl2 + thl2) + ...+ (tGNUln + thln) (3)where tPCl∗ is the single processing unit delay, tGNUl∗ is the GNU radio processing delay, and thl∗ is thehardware processing delay (see Figure 6). The software processing delay is tsln=l1 = tPCl1+ tGNUl1 for theﬁrst hop and tsln =l1 = tGNUl∗ for the subsequent hops. The tGNUl∗ includes the response time incurred fora node of a route to request for next-hop node information (e.g., the next-hop node IP address andoperating channel) from the personal computer.Figure 8. The end-to-end delay of a route in the two testbeds. The delays are shown in same sizedblocks although the time period of each block may be different.5.3.2. Testbed with Separate Processing Units (RPU)Figure 8b shows a testbed in which each BS or node is connected to a separate processing unit(RPU) (i.e., a RP3), and so a switch is not required. RP3 provides a distributed approach to handleD2D communication. In the route, f cs transmits to next-hop f c1 in 850 MHz. Then, f c1 receives in203Sensors 2020, 20, 18850 MHz and transmits in 860 MHz; f c2 receives in 860 MHz and transmits in 870 MHz; f c3 receivesin 870 MHz and transmits in 880 MHz; f c4 receives in 880 MHz and transmits in 890 MHz to thedestination node f cd.The BSs and nodes exchange control and data messages among themselves via wirelesstransmission. Each node incurs three types of delays: (a) the software processing delay incurredin RP3; (b) the software processing delay incurred in GNU radio; and (c) the hardware processing delayincurred in USRP and the propagation delay. In contrast to the testbed with a PCU (see Section 5.3.1),the intermediate nodes incur the software processing delay in the RP3. Unlike PCU, in RPU, everysingle node (either source, intermediate, or destination node) receives decision on the next-hop nodeand the transmission channel, which incurs tRP3ln (see Figure 8), from the personal computer. The nodecan also receive such information from the personal computer. Since RP3 has limited processingcapability, the software processing delay for the software processes is non-negligible at each node inRPU, causing longer end-to-end delay compared to that in PCU. Based on Equation (2), the end-to-enddelay of a route kk ∈ K is as follows:tkk =∑(tRP3ln + tGNUln + thln) (4)where the software processing delay is tsl∗ = tRP3l∗ + tGNUl∗ for all hops.The software processing delay for personal computer tPCl∗ and RP3 tRP3i∗ are different. There aretwo types of access rates: (a) the read rate is the speed in which a node reads information and readingis needed during transmission; and (b) the write rate is the speed in which a node writes informationand writing is needed during reception. For both read and write, the access time deﬁnes their rates.Access time is the average duration for the kernel to access a partition and perform a task, includingread and write. In this evaluation, the kernel performs read and write operations from the bootpartition of the Linux Ubuntu MATE operating system installed in RP3. The read rate is considered ina source node, both read and write rates are considered in an intermediate node, and the write rate isconsidered in a destination node. In general, the average read rate is higher than the average write rate.In personal computer, the read and write rates are much higher; speciﬁcally the hard disk drive (HDD)of a personal computer has a drives spin of 7200 revolutions per minute (RPM) in our testbed, andthe read and write rates are approximately 80 MBps and 50 MBps, respectively. In RP3, the read andwrite rates are approximately 22.6 MBps and 16.6 MBps, respectively, for the memory card (i.e., theSD card) on RP3. Due to the lower write rate, the packet queue size at intermediate and destinationnodes increase with the number of packets. Hence, RP3 has a limited performance. Figure 9 shows theaverage read and write rates for reading and writing 1000 samples from/to the boot partition of RP3with an average access time of 0.48 ms. The average read rate hovers between 22 and 23 MBps, and theaverage write rate varies between 9 and 20 MBps. The average access time refers to the duration ofreading a packet from or writing a packet to the boot partition, and it has an average value of 0.56 ms.The signiﬁcant lower read and write rates of RP3 can contribute to a higher end-to-end delay andlower packet delivery ratio in RPU compared to PCU.For an intermediate node, it receives and processes (i.e., decodes and demodulates) packets beforeit writes them in its memory, then it reads them from the memory and processes (i.e., encodes andmodulates) the packets for transmission. The intermediate node must write an entire packet beforeit can read the packet again for transmission, which is a phenomenon called buffering that occursduring initiation. This allows packets to be fully converted (or digitized) before transmission; however,it causes a higher end-to-end delay as the number of hops increases.204Sensors 2020, 20, 18Figure 9. Comparison between read and write performance of SD card on RP3.6. Experimental ResultsThis section compares the PCU and RPU performance measures in Section 6.1, and the packetdelivery via primary route (i.e., via D2D) and secondary route (i.e., via MC BS) in Section 6.2.6.1. Performance Comparison between PCU and RPUFigure 10a presents a logarithmic graph that shows the total delay, as well as the software andhardware processing delays, for a route from the source node to the destination node. For each numberof hops, the results are presented using a pair of bars: the left represents PCU, and the right representsRPU. The total delay is higher in RPU, and it increases as the number of hops increases, speciﬁcally thetotal delay increases from 0.024102s for a single hop to 0.10566s for ﬁve hops in PCU, and from 0.02992s for a single hop to 0.18924s for ﬁve hops in RPU. Compared to the hardware processing delay, thesoftware processing delay is signiﬁcantly lower due to the high processing capability of the personalcomputer in PCU; speciﬁcally, the software processing delay is 0.00018s (or 0.7468%) for a single hop,0.00023s (or 0.6189%) for two hops, 0.00038s (or 0.6316%) for three hops, 0.00057s (or 0.6958%) for fourhops, and 0.00079s (or 0.7476%) for ﬁve hops. However, the software processing delay is signiﬁcantlyhigher than that in PCU (i.e., approximately ﬁve times higher); speciﬁcally, the software processingdelay is 0.00141s (or 4.7125%) for a single hop, 0.00433 s (or 6.252%) for two hops, 0.00628s (or 6.8484%)for three hops, 0.00771s (or 6.5844%) for four hops, and 0.01074s (or 5.675%) for ﬁve hops. This isbecause, in RPU, RP3 has a lower processing capability, causing a higher total delay in each hop.Figure 10b shows that the packet delivery ratio reduces as the number of hops in a route from thesource node to the destination node increases because more intermediate nodes are affected by theambient noise in the operating environment. Speciﬁcally, the packet delivery ratio reduces from 99.7%for a single hop to 94.8% for ﬁve hops in PCU, and it reduces from 97.49% for a single hop to 88.73%for ﬁve hops in RPU.205Sensors 2020, 20, 18Figure 10. End-to-end delay and packet delivery ratio for PCU and RPU. (a) Comparison of softwareand hardware processing delays between PCU and RPU. (b) Comparison of packet delivery ratiobetween PCU and RPU.6.2. Comparison of Packet Delivery via Primary and Secondary RoutesThis section compares the performance of a communication that uses: (a) a multihop primaryroute (via D2D); and (b) a secondary route (via MC BS). This is because a source node can communicatewith a destination node via either a multihop D2D route or going through a BS. Nevertheless, thedelay in a multihop D2D route must be less than 10 ms (see Section 4). Figure 11 shows that a packetdelivered via a D2D route has a higher end-to-end delay than its counterpart route using MC BS.Figure 11. Comparison of software and hardware processing delays between Case I and CaseII. In Case I, a two-hop communication is performed via MC BS in PCU. In Case II, a two-hopcommunication is performed via D2D in RPU.206Sensors 2020, 20, 18This section investigates the end-to-end delay of a primary route when nodes are embedded withRP3, and compares it with that of a secondary route, in which only source and destination nodesare embedded with RP3, and the BS is implemented using a personal computer. In Figure 12, a RP3source node f cs can communicate with a RP3 destination node f cd via either: (a) Case I which is adirect communication with MC BS MC1 (i.e., a personal computer), speciﬁcally f cs − MC1 − f cd inPCU; or (b) Case II which is a two-hop route using D2D communication, speciﬁcally f cs − f c1 − f cd inRPU, whereby f c1 is an intermediate node embedded with RP3. The total delay of Cases I and II are0.039736 s and 0.08666 s, respectively, and so the total delay of Case II is more than twice higher thanthat in Case I. In RPU, the total software and hardware processing delay of a D2D communication fromthe source node up to the intermediate node is 0.03384s; while in PCU, the total software and hardwareprocessing delay from the source node up to the MC BS is 0.01655 s. In RPU, the total softwareand hardware processing delay from the intermediate node f c1 to the destination node is 0.05282 s;while in PCU, the total software and hardware processing delay from the MC BS to the destination is0.02318 s. Hence, Case I has a lower total delay as compared to Case II due to its greater processingcapability. Figure 11 shows the total delay, which includes software and hardware processing delays,of a direct communication with MC BS and a multihop D2D route. Figure 13 shows the packet deliveryratio via D2D and MC BS. A source node transmits a packet towards a destination node, the packetgoes through intermediate node as the destination node is beyond the transmission range of thetransmitter. For this packet transmission, the intermediate node is ﬁrst selected (i.e., Case I), and thenthe MC BS is selected (i.e., Case II). From the source node to the intermediate node in Figure 13, the MCBS has a 99.65% packet delivery ratio, while the node with RP3 has 97.32%. From the intermediatenode to the destination node, MC BS delivers 99.59% of the packets, while the node with RP3 delivers96.63% of the packets. Therefore, in Figure 13 the total packet delivery ratio from a source node to adestination node via MC BS is 99.24%. However, the packet delivery ratio from the source node to thedestination node via RP3 is 93.95%, which is about 7% lesser.Figure 12. A D2D route from a FC source node f cs to a FC destination node f cd.207Sensors 2020, 20, 18Figure 13. Comparison of the two-hop packet delivery via D2D and MC BS.In Figure 14, a throughput comparison is made between PCU and RPU. Higher throughput refersindicates a higher successful packets transmission rate [26]. Throughput reduces as the number ofhops increases for both PCU and RPU; however, PCU achieves a higher throughput compared to RPUbecause of higher packet delivery ratio (see Figure 10b).Figure 14. Throughput comparison between RPU and PCU.7. Conclusions and Future WorkThis paper presented an experimental study to compare performance measures in a testbed witha single processing unit (PCU) and a testbed with separate processing units (RPU). The testbed consistsof nodes implemented using universal software radio peripheral with GNU radio. In PCU, basestations and user equipment, including sensors, connect to a single centralized traditional processingunit (e.g., a personal computer or a laptop) via physical cables and a switch. On the other hand, inRPU, each BS or node is embedded with a separate processing unit, particularly Raspberry Pi3 B+.208Sensors 2020, 20, 18While PCU is a widely used testbed in the literature, nodes are constrained to be located at closeproximity to the centralized processing unit. Meanwhile, RPU has a closer resemblance to a realdeployed network, and it has not been investigated in the literature, and so it is the focus of this paper.Our experimental results showed that: (a) the end-to-end delay is lower in PCU as control messagesare exchanged via a switch using gigabit Ethernet; and (b) the per-hop and end-to-end delays increasewith the number of hops in RPU. However, in RPU, device-to-device communication between nodesfrom a source node to a destination node can ofﬂoad trafﬁc from BS, which is one of the promisingfeatures of 5G. Therefore, this paper presents a case study in which the intermediate node of a two-hoproute can be: (a) a node (via D2D); or (b) a macrocell BS. While the preceding case can reduce the trafﬁcamount at a macrocell BS, it can increase end-to-end delay and reduce packet delivery ratio comparedto the latter case due to its lower processing capability.As for future work, we aim to relax the assumptions made in this article to enable a macrocellbase station (MC BS) to receive updates from femtocell base stations and nodes. Examples of suchupdates are the packet delivery ratio and per-hop delay, which allows MC BS to make decision onroute selection based on the updates under unpredictable and dynamic operating environment.Author Contributions: M.K.C. designed and implemented testbed. K.-L.A.Y. contributed to system model andliterature review. R.M.D.N. and R.W. contributed to paper review and formatting. All authors have read andagreed to the published version of the manuscript.Funding: This work was part of the project entitled “A Novel Clustering Algorithm based on ReinforcementLearning for the Optimization of Global and Local Network Performances in Mobile Networks” funded by theMalaysian Ministry of Education under Fundamental Research Grant Scheme FRGS/1/2019/ICT03/SYUC/01/1,as well as the Partnership Grant CR-UM-SST-DCIS-2018-01 and RK004-2017 between Sunway University andUniversity of Malaya.Conﬂicts of Interest: The author declares no conﬂict of interest.AbbreviationsThe following abbreviations are used in this manuscript:USRP Universal software radio partnershipRP3 Raspberry Pi3 B+5G Fifth generationD2D Device to deviceMC MacrocellFC FemtocellPC PicocellCC Central controllerBC Base stationWBX Wide bandwidth transceiverFPGA Field-programmable gate arrayRF Radio frequencyADC Analogue-to-digital converterDAC Digital-to-analogue converterDUD Digital up converterDDC Digital down converterUHD USRP hardware driverSDR Software deﬁned radioPoE Power over EthernetMbps Megabits per secondUSP Universal serial busCAT CategoryGRC GNU radio companionIP Internet protocolUDP User datagram protocol209Sensors 2020, 20, 18GMSK Gausian minimum shift keyingSD Secure digitalRAM Random access memoryRPU Separate processing unitPCU Single processing unitHDD Hard disk driveReferences1. Ashraf, M.I.; Tamoor-Ul-Hassan, S.; Mumtaz, S.; Tsang, K.F.; Rodriquez, J. Device-to-device assisted mobilecloud framework for 5G networks. In Proceedings of the IEEE International Conference on IndustrialInformatics (INDIN), Emden, Germany, 24–26 July 2017; pp. 1020–1023. [CrossRef]2. Raschellà, A.; Umbert, A. Implementation of Cognitive Radio Networks to evaluate spectrum managementstrategies in real-time. Comput. Commun. 2016, 79, 37–52. [CrossRef]3. Okasaka, S.; Weiler, R.J.; Keusgen, W.; Pudeyev, A.; Maltsev, A.; Karls, I.; Sakaguchi, K. Proof-of-conceptof a millimeter-wave integrated heterogeneous network for 5G cellular. Sensors 2016, 16, 1362. [CrossRef][PubMed]4. Asadi, A.; Member, S.; Wang, Q.; Member, S.; Mancuso, V. A Survey on Device-to-Device Communication inCellular Networks. IEEE Commun. Surv. Tutor. 2014, 16, 1801–1819. [CrossRef]5. Aly, A.A.; ELAttar, H.M.; ElBadawy, H.; Abbas, W. Aggregated Throughput Prediction for Collated MassiveMachine-Type Communications in 5G Wireless Networks. Sensors 2019, 19, 3651. [CrossRef]6. Chang, B.; Zhao, G.; Imran, M.A.; Chen, Z.; Li, L. Dynamic Wireless QoS Analysis for Real-Time Control inURLLC. In Proceedings of the 2018 IEEE Globecom Workshops, GC Wkshps 2018—Proceedings, Abu Dhabi,UAE, 9–13 December 2018; pp. 1–5. [CrossRef]7. Zhang, H.; Liao, Y.; Song, L. D2D-U: Device-to-Device Communications in Unlicensed Bands for 5G System.IEEE Trans. Wirel. Commun. 2017, 16, 3507–3519. Available online: http://xxx.lanl.gov/abs/1610.04982(accessed on 1 August 2019). [CrossRef]8. Yau, K.l.A.; Qadir, J.; Wu, C.; Imran, M.A.L.I.; Ling, M.E.E.H. Cognition-inspired 5G cellular networks:A review and the road ahead. IEEE Access 2018, 6, 35072–35090. [CrossRef]9. Ge, X.; Tu, S.; Mao, G.; Wang, C.X.; Han, T. 5G Ultra-Dense Cellular Networks. IEEE Wirel. Commun. 2016,23, 72–79. Available online: http://xxx.lanl.gov/abs/1512.03143 (accessed on 22 December 2018). [CrossRef]10. Agiwal, M.; Roy, A.; Saxena, N. Next generation 5G wireless networks: A comprehensive survey.IEEE Commun. Surv. Tutor. 2016, 18, 1617–1655. [CrossRef]11. Kim, J.; Molisch, A.F. Quality-aware millimeter-wave device-to-device multi-hop routing for 5G cellularnetworks. In Proceedings of the 2014 IEEE International Conference on Communications, ICC 2014, Sydney,Australia, 10–14 June 2014; pp. 5251–5256. [CrossRef]12. He, J.; Song, W. Evolving to 5G: A Fast and Near-optimal Request Routing Protocol for Mobile CoreNetworks. IEEE Wirel. Netw. 2014, 4586–4591. [CrossRef]13. Tran, T.X.; Hajisami, A.; Pompili, D. ULTRA-DENSE HETEROGENEOUS SMALL CELL DEPLOYMENTIN 5G AND BEYOND Cooperative Hierarchical Caching in 5G Cloud Radio Access Networks. IEEE Netw.2017, 31, 35–41. [CrossRef]14. Chávez-Santiago, R.; Szydełko, M.; Kliks, A.; Foukalas, F.; Haddad, Y.; Nolan, K.E.; Kelly, M.Y.; Masonta,M.T.; Balasingham, I. 5G: The Convergence of Wireless Communications. Wirel. Pers. Commun. 2015,83, 1617–1642. [CrossRef] [PubMed]15. Mahmood, A.M.; Al-yasiri, A.; Alani, O.Y. Cognitive Neural Network Delay Predictor for High SpeedMobility in 5G C-RAN Cellular Networks. In Proceedings of the 2018 IEEE 5G World Forum (5GWF), SiliconValley, CA, USA, 9–11 July 2018; pp. 93–98. [CrossRef]16. Abd, S.A.; Manjunath, S.; Abdulhayan, S. Direct Device-to-Device Communication in 5G Networks.In Proceedings of the 2016 International Conference on Computation System and Information Technologyfor Sustainable Solutions (CSITSS), Bangalore, India, 6–8 October 2016; pp. 216–219. [CrossRef]17. Wassie, D.A.; Berardinelli, G.; Tavares, F.M.L. Experimental Veriﬁcation of Interference Mitigation techniquesfor 5G Small Cells. In Proceedings of the 2015 IEEE 81st Vehicular Technology Conference (VTC Spring),Glasgow, UK, 11–14 May 2015; pp. 1–5. [CrossRef]210Sensors 2020, 20, 1818. Quaresma, J.; Ribeiro, C.; Gameiro, A.; Zelenak, M.; Duplicy, J. Distributed RF sensing framework withradio environment emulation. In Proceedings of the 2013 IEEE 14th International Symposium on a Worldof Wireless, Mobile and Multimedia Networks, WoWMoM 2013, Madrid, Spain, 4–7 June 2013; pp. 1–6.[CrossRef]19. Jain, A.; Sharma, V.; Amrutur, B. Soft real time implementation of a Cognitive Radio testbed for frequencyhopping primary satisfying QoS requirements. In Proceedings of the 2014 20th National Conference onCommunications, NCC 2014, Kanpur, Uttar Pradesh, India, 28 February–2 March 2014; pp. 1–6. [CrossRef]20. Barrak, S.E.; Lyhyaoui, A.; Puliaﬁto, A.; Serrano, S. Implementation of a low cost SDR-based SpectrumSensing Prototype using USRP and Raspberry Pi board. Int. Conf. Autom. Control Eng. Comput. Sci. (ACECS)2017, 20, 54–58.21. Briand, A.; Albert, B.B.; Gurjao, E.C. Complete software deﬁned RFID system using GNU radio.In Proceedings of the 2012 IEEE International Conference on RFID-Technologies and Applications, RFID-TA2012, Nice, France, 5–7 November 2012; pp. 287–291. [CrossRef]22. Öhlén, P.; Skubic, B.; Rostami, A.; Ghebretensaé, Z.; Mårtensson, J.; Fiorani, M.; Monti, P.; Wosinska, L. DataPlane and Control Architectures for 5G Transport Networks. J. Light. Technol. 2016, 34, 1501–1508. [CrossRef]23. Maksymyuk, T.; Gazda, J.; Yaremko, O.; Nevinskiy, D. Deep Learning Based Massive MIMO Beamformingfor 5G Mobile Network. In Proceedings of the 2018 IEEE 4th International Symposium on Wireless Systemswithin the International Conferences on Intelligent Data Acquisition and Advanced Computing Systems(IDAACS-SWS), Lviv, Ukraine, 20–21 September 2018; pp. 241–244. [CrossRef]24. Ettus, M.; Braun, M. The universal software radio peripheral (usrp) family of low-cost sdrs. In OpportunisticSpectrum Sharing and White Space Access: The Practical Reality; Wiley Telecom: Hoboken, NJ, USA, 2015;pp. 3–23.25. Ali, S.A.; Umrani, F.A.; Umrani, N.A. Performance Evaluation Of Decode and Forward CooperativeCommunication Protocol. Int. J. Inf. Technol. Electr. Eng. 2012, 1, 321–325.26. Syed, A.; Yau, K.L.; Qadir, J.; Mohamad, H.; Ramli, N.; Keoh, S. Route selection for multi-hop cognitive radionetworks using reinforcement learning: An experimental study. IEEE Access 2016, 4, 6304–6324. [CrossRef]27. Truong, N.B.; Suh, Y.J.; Yu, C. Latency analysis in GNU radio/USRP-based software radio platforms.In Proceedings of the IEEE Military Communications Conference MILCOM, San Diego, CA, USA,18–20 November 2013; pp. 305–310. [CrossRef]28. Van den Bergh, B.; Vermeulen, T.; Verhelst, M.; Pollin, S. CLAWS: Cross-Layer Adaptable Wireless Systemenabling full cross-layer experimentation on real-time software-deﬁned 802.15. 4. EURASIP J. Wirel.Commun. Netw. 2014, 1–13. [CrossRef]29. Khandakar, A.; Mohamed, A.M.S. Understanding probabilistic cognitive relaying communication withexperimental implementation and performance analysis. Sensors 2019, 19, 179. [CrossRef]30. Nychis, G.; Hottelier, T.; Yang, Z.; Seshan, S.; Steenkiste, P. Enabling MAC Protocol Implementations onSoftware-Deﬁned Radios. Netw. Syst. Des. Implement. 2009, 9, 91–105.31. Chen, W.T.; Chang, K.T.; Ko, C.P. Spectrum monitoring for wireless TV and FM broadcast usingsoftware-deﬁned radio. Multimed. Tools Appl. 2016, 75, 9819–9836. [CrossRef]32. Byun, S.S. TCP over scarce transmission opportunity in cognitive radio networks. Comput. Netw. 2016,103, 101–114. [CrossRef]33. Gameiro, A.; Ribeiro, C.; Quaresma, J. Selective reporting—A half signalling load algorithm for distributedsensing. Eurasip J. Wirel. Commun. Netw. 2013, 2013, 1–14. [CrossRef]34. Marko, H.; Korpi, J.; Hiivala, M. Predictive Channel Selection for over-the-Air Video Transmission UsingSoftware-Deﬁned Radio Platforms. In International Conference on Cognitive Radio Oriented Wireless Networks;Springer: Cham, Switzerland, 2016; Volume 172, pp. 569–579. [CrossRef]35. Berardinelli, G.; Buthler, J.L.; Tavares, F.M.L.; Tonelli, O.; Wassie, D.A.; Hakhamaneshi, F.; Sørensen, T.B.;Mogensen, P. Distributed Synchronization of a testbed network with USRP N200 radio boards.In Proceedings of the IEEE International Conference on Communications, Paciﬁc Grove, CA, USA,2–5 November 2014; pp. 563–567. [CrossRef]36. Nagaraju, P.B.; Ding, L.; Melodia, T.; Batalama, S.N.; Pados, D.A.; Matyjas, J.D. Implementation of aDistributed Joint Routing and Dynamic Spectrum Allocation Algorithm on USRP2 Radios. In Proceedingsof the IEEE International Conference on Communications, Boston, MA, USA, 21–25 June 2010; pp. 26–27.[CrossRef]211Sensors 2020, 20, 1837. Chang, B.; Zhao, G.; Chen, Z.; Li, L.; Imran, M.A. Packet-Drop Design in URLLC for Real-Time WirelessControl Systems. IEEE Access 2019. [CrossRef]38. Martínez-quintero, J.C.; Estupiñán-cuesta, E.P.; Rodríguez-ortega, V.D. Raspberry PI 3 RF signal generationsystem. Visión Electrónica 2019, 13, 2.39. Zhang, N.; Sun, W.; Lou, W.; Hou, Y.T.; Trappe, W. ROSTER: Radio context attestation in cognitive radionetwork. In Proceedings of the 2018 IEEE Conference on Communications and Network Security, CNS 2018,Beijing, China, 30 May–1 June 2018; pp. 1–9. [CrossRef]40. Tomar, V.S.; Bhatia, V. Low Cost and Power Software Deﬁned Radio Using Raspberry Pi for Disaster EffectedRegions. Procedia Comput. Sci. 2015, 58, 401–407. [CrossRef]41. Al-saﬁ, A.; Narasimhan, L.; Bazuin, B. Software Deﬁned Community Radio Using Low Cost Hardware andFree Software. In Proceedings of the Universal Technology Management Conference (UTMC), Bemidji, MN,USA, 26–28 May 2016; pp. 51–55.42. Shi, Y.; Wensowitch, J.; Ward, A.; Badi, M.; Camp, J. Building UAV-Based Testbeds for Autonomous Mobilityand Beamforming Experimentation. In Proceedings of the 2018 IEEE International Conference on Sensing,Communication and Networking, SECON Workshops 2018, Hong Kong, China, 11–13 June 2018; pp. 1–5.[CrossRef]43. Park, H.J.; Lee, G.M.; Shin, S.H.; Roh, B.H.; Oh, J.M. Implementation of Multi-Hop Cognitive Radio Testbedusing Raspberry Pi and USRP. Int. J. Interdiscip. Telecommun. Netw. 2017, 9, 37–48. [CrossRef]44. Habib, M.A.; Moh, S. Robust Evolutionary-Game-Based Routing for Wireless Multimedia Sensor Networks.Sensors 2019, 19, 3544. [CrossRef] [PubMed]45. Shariatmadari, H.; Iraji, S.; Anjum, O.; Riku, J.; Li, Z.; Wijting, C. Delay Analysis of Network Architectures forMachine-to-Machine Communications in LTE System. In Proceedings of the 21st International Conferenceon Telecommunications (ICT)—Workshop on M2M Solutions and Services, Lisbon, Portugal, 4–7 May 2014;pp. 502–506. [CrossRef]46. Homaei, M.H.; Salwana, E.; Shamshirband, S. An Enhanced Distributed Data Aggregation Method in theInternet of Things. Sensors 2019, 19, 3173. [CrossRef]47. Jin, Z.; Ma, Y.; Su, Y.; Li, S.; Fu, X. A Q-learning-based delay-aware routing algorithm to extend the lifetimeof underwater sensor networks. Sensors 2017, 17, 1660. [CrossRef]48. Zhang, W.; Wang, C.X.; Ge, X.; Chen, Y. Enhanced 5G Cognitive Radio Networks Based on Spectrum Sharingand Spectrum Aggregation. IEEE Trans. Commun. 2018, PP, 492–496. [CrossRef]49. Orebaugh, A.; Ramirez, G.; Beale, J. Wireshark & Ethereal Network Protocol Analyzer Toolkit; Elsevier:Amsterdam, The Netherlands, 2006.c© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open accessarticle distributed under the terms and conditions of the Creative Commons Attribution(CC BY) license (http://creativecommons.org/licenses/by/4.0/).212MDPISt. Alban-Anlage 664052 BaselSwitzerlandTel. +41 61 683 77 34Fax +41 61 302 89 18www.mdpi.comSensors Editorial OfﬁceE-mail: sensors@mdpi.comwww.mdpi.com/journal/sensorsMDPI St. Alban-Anlage 66 4052 Basel SwitzerlandTel: +41 61 683 77 34 Fax: +41 61 302 89 18www.mdpi.com ISBN 978-3-03928-149-7

Internet of Things and Sensors Networks in 5G Wireless Communications

Abstract

Similar works

Full text

Available Versions

Enlighten