iFogSim2: An Extended iFogSim Simulator for Mobility,Clustering, and Microservice Management in Edge and Fog

Computing Environments

Redowan Mahmud, Samodha Pallewatta, Mohammad Goudarzi, and Rajkumar Buyya

September 17, 2021

Abstract

Internet of Things (IoT) has already proven to be thebuilding block for next-generation Cyber-Physical Systems(CPSs). The considerable amount of data generated bythe IoT devices needs latency-sensitive processing, whichis not feasible by deploying the respective applications inremote Cloud datacentres. Edge/Fog computing, a promis-ing extension of Cloud at the IoT-proximate network, canmeet such requirements for smart CPSs. However, thestructural and operational differences of Edge/Fog infras-tructure resist employing Cloud-based service regulationsdirectly to these environments. As a result, many researchworks have been recently conducted, focusing on efficientapplication and resource management in Edge/Fog com-puting environments. Scalable Edge/Fog infrastructure isa must to validate these policies, which is also challengingto accommodate in the real-world due to high cost andimplementation time. Considering simulation as a key tothis constraint, various software has been developed thatcan imitate the physical behaviour of Edge/Fog computingenvironments. Nevertheless, the existing simulators of-ten fail to support advanced service management featuresbecause of their monolithic architecture, lack of actualdataset, and limited scope for a periodic update. To over-come these issues, we have developed multiple simulationmodels for service migration, dynamic distributed clusterformation, and microservice orchestration for Edge/Fogcomputing in this work and integrated with the existingiFogSim simulation toolkit for launching it as iFogSim2.The performance of iFogSim2 and its built-in policies areevaluated using three use case scenarios and comparedwith the contemporary simulators and benchmark policiesunder different settings. Results indicate that the proposedsolution outperform others in service management time,network usage, ram consumption, and simulation time.

Keywords:Edge/Fog Computing, Mobility, Microservices, Clustering,Simulation, Internet of Things

R. Mahmud is with The School of Computing Technologies,STEM College, RMIT University, Melbourne, Australia

S. Pallawetta, M. Goudarzi, and R. Buyya are with TheCloud Computing and Distributed Systems (CLOUDS) Laboratory,School of Computing and Information Systems, The University ofMelbourne, Australia

Corresponding Author: Mohammad Goudarzi, Email:mgoudarzi@student.unimelb.edu.au

1 Introduction

The Internet of Things (IoT) paradigm has drasticallychanged the convention of interactions between physicalenvironments and digital infrastructures that sets the toneof using numerous IoT devices including fitness trackers,voice controllers, smart locks, and air quality monitors inour daily activities. Currently, IoT devices are contribut-ing 11.5 Zetta bytes to the total data generated aroundthe globe, which is experiencing an exponential rise eachyear [1]. The recent adoption of Edge/Fog computing hasrelaxed the requirements of harnessing Cloud datacentresfor different IoT-driven use cases. This novel computingparadigm spans the computing facilities across the proxi-mate network and enables smart health, smart city, Agtech,Industry 4.0, and other IoT-enabled Cyber-Physical Sys-tems (CPSs) to run the necessary application softwarein the vicinity of the data source [2]. Thus, Edge/Fogcomputing ensures the delivery of application services tothe requesting IoT-enabled CPSs with reduced data trans-mission and propagation delay and lessens the possibilityof congestion within the core network infrastructure byinhibiting the transfer of a large amount of raw data tothe Cloud datacentres.

The realisation of Edge/Fog computing environments pri-marily depends on the integration of computing resourcessuch as processing cores, nano servers, and micro datacen-tres with the traditional networking components, includinggateway routers, switches, hubs, and base stations [3, 4]. Incontrast to Cloud datacentres, such Edge/Fog computingnodes are highly distributed. Similarly, the heterogeneityin terms of resource architecture, communication stan-dards, and operating principles predominantly exist amongthese nodes. Because of such constraints, the centralisedCloud-based resource provisioning and application place-ment techniques have compatibility issues with Edge/Fogcomputing environments and cannot be applied directlyto regulate the respective services [5, 6]. Identifying thispotential research gap, a significant number of initiativeshave been taken to develop efficient service managementpolicies for Edge/Fog computing. The research interest forEdge/Fog computing has increased around 69% in the lastfive years [7]. However, the newly developed service man-agement policies for Edge/Fog computing environmentsrequire extensive validation before enterprise adoption.Real-world deployment is the most effective approach

to evaluate the performance of any service management

1

arX

iv:2

109.

0563

6v2

[cs

.DC

] 1

5 Se

p 20

21

policy. However, since Edge/Fog computing environmentsincorporate numerous IoT devices and computing nodes,both in tiered and flatted order with vast amounts of batchor streaming data and distributed applications, their real-world implementation with such a scale is challenging. Thelack of global Edge/Fog service providers offering infras-tructure on pay-as-you-go models like commercial Cloudplatforms such as Microsoft Azure and Amazon AWS fur-ther forces researchers to set up real Edge/Fog computingenvironments by themselves for costly policy evaluation.Additionally, the implementation time for a real-world en-vironment is significantly high, and the modification andtuning of any entity or system parameters during the exper-iments are tedious [8]. These constraints can be addressedby simulating Edge/Fog computing environments. Notonly does simulation provide support for designing a cus-tomized and scalable experiment environment, but it alsoassists in the repeatable evaluation under different settings.Although there exist several simulators for Edge/Fog

computing environments, a majority of them lack bench-marks to validate other service management policies. Theymerely use synthetic data without any functional ground,which often direct to biased and erroneous performanceevaluation. Their monolithic architecture also refuses peri-odic updates, resisting them to cope up with the advancedfeatures of genuine Edge/Fog computing nodes. Conse-quently, they fail in imitating various complex scenariostriggered by uncertain device mobility, resource constraints,and heavy-weight computations. To meet such shortcom-ings, a set of simulation models for mobility-aware applica-tion migration, dynamic distributed cluster formation, andmicroservice orchestration has been developed in this work.The proposed models exploit real-world dataset and arecomprehensively mimics the capabilities of the state-of-the-art Edge/Fog computing nodes and IoT devices. Thesemodels are integrated with the existing iFogSim simulator[9] and launched as iFogSim2 for widespread adoption asbenchmarks. The major contributions of our work arelisted below.

• A service migration simulation model that can operateacross multi-tier infrastructure and support simpli-fied integration of real-world dataset. The launchingversion uses EUA Dataset as the default and accom-modates different device mobility models, includingpathway and random waypoint.

• A dynamic distributed cluster formation among multi-tier infrastructure is proposed, where Edge/Fog nodesin different tiers can provide services with a higherquality of service. The cluster management is per-formed in a distributed manner through which differ-ent cluster formation policies can be simultaneouslyintegrated.

• An orchestration model for microservices deployedacross multi-tier infrastructure, which enables place-ment policies to dynamically scale microservicesamong federated Edge/Fog nodes to improve resourceutilisation. Different service discovery and load balanc-ing policies can be integrated to simulate the dynamicmicroservice behavior.

The rest of the paper is organized as follows. In Section2, related researches are reviewed. Section 3 denotes howthe proposed simulation models are integrated with theiFogSim simulator. The performance of proposed iFogSim2simulator is evaluated in Section 4. Finally, Section 5concludes the paper with future works.

2 Related work

Among the existing simulators for Edge/Fog computing,the EdgeCloudSim software supports the nomadic move-ments of the IoT devices [10]. Additionally, it considers thestatic deployment and coverage area for the gateway nodesand assumes the link quality between IoT and gatewaynodes remains always the same despite their distance. Sim-ilarly, the FogNetSim++ simulator developed by Qayyumet al. [11] can imitate different mobility models for IoTdevices, including random waypoint, mass mobility, andlinear mobility. It also provides the facilities to developcustomized mobility models as per the operating environ-ment. The mobility support system of FogNetSim++ isloosely coupled with the core simulation engine, and thusits extension requires the least modifications of the primarylibraries. However, both EdgeCloudSim and FogNetSim++lack abstractions for implementing microservice orchestra-tion and dynamic clusterisation among multiple Edge/Fognodes. In [12], Jha et al. proposed another simulatornamed IoTSim-Edge for modeling the characteristics ofIoT devices in the Fog computing environment. It rep-resents IoT applications as a collection of microservices,and the mobility model associated with IoTSim-Edge in-corporates different attributes of IoT devices, including itsrange, velocity, location, and time interval. This simulatoralso facilitates users to implement their mobility model byextending the core simulator programming interfaces butbarely highlights the clustering of the computing nodes.

Furthermore, Puliafito et al. [13] have recently developedMobFogSim for simulating device mobility and applicationmigration in Fog computing environments. It is an ex-tension of iFogSim that modifies the basic functionalitiesof different iFogSim components with mobility features.However, the mobility support system of MobFogSim onlydeals with the IoT gateways and Cloud datacentres insteadof tiered Edge/Fog infrastructure and limits the scope forcreating clusters in Edge/Fog computing environments.Mechalikh et al. [14] developed another simulator calledPureEdgeSim to evaluate the performance of Fog and Cloudcomputing environments for different IoT-driven use cases.The mobility support system of PureEdgeSim includes alocation manager, which is loosely coupled with the coresimulation engine. However, the default mobility-awareapplication management policy of PureEdgeSim is complexand difficult to customize. It also has limitations in formingnode clusters and augmenting microservice managementtechniques. Conversely, Mass et al. [15] developed theSTEP-ONE simulator to imitate the operations of Fog-based opportunistic network environments. STEP-ONEextends the conventional ONE simulator with advancedmobility and messaging interfaces and primarily focuses onmodeling simple business processes. Although STEP-ONE

2

incorporates support for the real-world dataset, it lacksdefault policies for mobility management, node clustering,and microservice orchestration. Likewise, in [16], Lera etal. proposed YAFS simulator for Fog computing to designand deploy various IoT applications with customized re-source management policies. The mobility support systemof YAFS operates based on the sender-receiver relationshipbetween the Fog nodes that identifies the shortest pathduring device movements. YAFS also defines logical rela-tions among microservices through graphs and providesinterfaces for node clustering.

Furthermore, the IoTNetSim [17] simulator for Edge/Fogcomputing environments developed by Salama et al. canmodel different IoT devices and their granular details, in-cluding energy profile. It supports the mobility of IoTdevices in three-dimensional space. Although IoTNetSimis highly modular, it lacks benchmark policies for mobility-driven service management and dynamic cluster formation.Wei et al. proposed another simulator named SatEdgeSimfor evaluating the performance of service management poli-cies in three-tier satellite edge computing environments[18]. Considering the high mobility of satellite nodes, Itsupports the dynamic alteration in network topology andimitates the impact of communication distance on ser-vice offloading delay. Although SatEdgeSim is modular,it barely exploits the concept of microservice. Conversely,the IoTSim-Osmosis simulator, developed by Alwasel et al.targets the migration of workload to edge nodes based onperformance and security requirements. It considers theIoT environment as a four-tier architecture and models theapplications in form of microservices. However, the simu-lation components of IoTSim-Osmosis are tightly coupledand constrained in imitating device mobility. ECSNeT++[19] is another simulator developed by Amarasinghe et al.that mimics the execution of distributed stream processing(DSP) applications in Edge/Fog computing environments.It extends OMNeT++/INET and provides multiple config-urations for two real DSP applications with calibration anddeployment management policy. Nevertheless, ECSNeT++lacks interfaces for supporting customized mobility of IoTdevices and forming dynamic clusters.In comparison to most of the existing solutions, the

proposed iFogSim2 simulator simultaneously supports theintegration of real dataset for evaluating the performanceof different service management policies in Edge/Fog com-puting environments and provides default techniques formobility management, node clustering, and microserviceorchestration, which can be adopted as benchmarks dur-ing performance comparison. Additionally, the simulationcomponents of iFogSim2 are highly modular that eases itscustomisation for imitating a wide range of service man-agement scenarios in Edge/Fog computing environments.

3 iFogSim2 components

To address the prevailing limitations of the iFogSim sim-ulator in supporting the migration of application, logicalgrouping of Fog nodes, and orchestration of loosely-coupledapplication services, three new components, namely Mobil-ity, Clustering and Microservices have been implemented

and included in iFogSim2 (as shown in Fig. 1).While simulating any use case through basic iFogSim,

its Controller class contains the object references of alliFogSim core classes such as FogDevice, Sensor, Actuatorand AppModule. Through an Application object, the Con-troller class can also access the Tuple class of iFogSim.Therefore, in the newly developed three components, wehave inherited the Controller class separately, so that theycan be easily integrated with the core iFogSim simulator.Additionally, the Controller class of iFogSim itself is asubclass of SimEntity, which also helps bridge iFogSim2with the core CloudSim 5.0 simulator. Besides, FogDeviceis updated with several parameters to support the integra-tion of these new components. In the following subsections,different iFogSim2 components are discussed in detail.

3.1 Mobility

The Mobility of IoT devices can affect the performance ofEdge/Fog computing, especially when they change accesspoints very frequently [21]. This event urges to migratethe requested application service of the IoT devices fromone computing node (migration source) to another (mi-gration destination) for ensuring the committed QoS. In alogical multi-tier computing infrastructure like Edge/Fog,such service migration operations depend on the followingaspects.

• The location of IoT devices.

• The timeline of movement or the mobility directionand speed of the device.

• The identification of an intermediate node to whichthe migration source can upload the correspondingapplication service and the migration destination candownload; provided that there is no direct link betweenthe respective migration points.

Based on them, the performance indicators of the Edge/Fogenvironment such as network delay, energy consumption,and service delivery time can also vary significantly. There-fore, taking these facts into account and aiming to as-sist users in customizing them, we have developed severalclasses in the Mobility component of iFogSim2. A detaileddescription of these classes is given below.DataParser: This class works as the interface for extend-

ing data from external sources to the proposed iFogSim2components. Currently, it incorporates abstractions forreading data from .csv files, which can be further extendedfor other formats. However, while simulating mobility-driven cases, iFogSim2 adopts the EUA Datasets whichcontains the location information of a notable number ofEdge/Fog nodes deployed across Central Business District(CBD) regions of major cities in Australia, including Mel-bourne and Sydney.

To ensure granularity, we further customised the datasetby segmenting the respective regions in multiple blocksand selecting a particular node at the middle of each blockas the proxy server. All nodes but the proxy server withina block act as the gateway for the IoT devices. As a meansof notations, proxy servers are specified as the tier-1 nodes

3

Table 1: A Summary of related work and their comparisonSimulators Real

datasetBenchmark pol-icy

Supports Modular architec-ture

Customised mobil-ity

Cluster forma-tion

Microservices

EdgeCloudSim [10] XFogNetSim++ [11] X XIoTSim-Edge [12] X X XMobFogSim [13] X X X XPureEdgeSim [14] X XSTEP-ONE [15] X XYAFS [16] X X X XIoTNetSim [17] X X XSatEdgeSim [18] X X XECSNeT++ [19] X X X XIoTSim-Osmosis[20]

X X

iFogSim2 X X X X X X

SimEvent

CloudSim 5.0

VM Datacenter

Core Components

Mobility

MobilityController

Location

Reference

RandomMobility

Generator

Actuator

AppModule

DataParser

Clustering

Controller

Clustering

MicroservicesMobility

ClusteringController

Microservices

MicroserviceFogDevice

PowerVMSimEntity PowerDatacenter

Tuple

ClusteringFogDevice

Controller

Microservices

Controller

LocationHandler

LoadBalancer

DeviceController

Component

ServiceDiscoveryMicroservice

PlacementLogic ModulePlacement

MobileEdgewards

Sensor

Object reference

Inheritence

Application

Figure 1: Overview of iFogSim2 simulator

in iFogSim2 and assumed to be the immediate upper tiercontact for the gateways residing at the same block. Inthis case, the gateways are referred to as tier-2 nodes. Forexample, Fig. 2.a presents the location of tier-1 (marked inblue) and tier-2 (marked in red) nodes deployed in the Mel-bourne CBD. Finally, such a logical hierarchy of Edge/Fognodes has been ended by connecting the proxy servers ofall blocks to a Cloud datacentre serving as the tier-0 node.To align with the characteristics of conventional networktopology, the location of these computing nodes is set tobe static. Furthermore, using a config.properties file, thesecustomised information are injected to the DataParserclass, which is easily modifiable as per the simulation usecases.

Additionally, the DataParser class provides scopefor assimilating location information of multiple mobileusers/IoT devices individually so that respective appli-cation services can be managed based on their distinc-

tive mobility pattern without affecting others. Currently,two different types of mobility patterns namely DIREC-TIONAL_MOBILITY and RANDOM_MOBILITY areassociated with the DataParser class through an object ofReference class.

• DIRECTIONAL_MOBILITY : This model refers tothe fixed speed acyclic movement of users/IoT devices.To realise the DIRECTIONAL_MOBILITY model,we have at first identified a considerable number ofsequential coordinates lying at the same distancesacross the Melbourne CBD for a user/IoT device (asshown in Fig. 2.b). Later, based on those coordinates,SimEvents using CloudSim 5.0 are created to mimicthe movement of the respective user/IoT device. Dur-ing simulations, the time interval between any two ofsuch movements is set to be equal for ensuring thefixed speed of the user/IoT device. iFogSim2 pro-vides a scope to tune this time interval as per the

4

test case requirements. Although this mobility modelprovides the pedestrial presentation of users’/IoT de-vices’ movement, it is difficult and time-consumingto generate for each individual. Therefore, iFogSim2also incorporates the RANDOM_MOBILITY modelfor faster generation of users’/IoT devices’ movementdata.

• RANDOM_MOBILITY : There are several randommobility patterns to model the mobility behaviourof users. The RandomMobilityGenerator class con-tains requirements to generate and extend differentrandom mobility models according to various mobilitycharacteristics, such as users’ direction, speed, stop-ping time in each position, and users’ sojourn timein the communication range of each Edge/Fog node.Currently, the RandomMobilityGenerator class imple-ments two well-known random mobility models, calledrandom_waypoint and random_walk that can be usedto represent the mobility model of either users or evenEdge/Fog nodes, if required. Besides, in multi-userscenarios, where multiple different random mobilitydatasets are required, the RandomMobilityGeneratorclass can be configured to generate different mobilitydatasets for users. Furthermore, iFogSim2 users canuse the functions embedded in RandomMobilityGen-erator class to generate mobility positions in theirdesired Region of Interest (RoI). Fig. 2.c depicts a ran-dom mobility pattern where the RoI is in MelbourneCBD.

However, while parsing the location information of bothEdge/Fog nodes and users/IoT devices, the DataParserclass creates separate Location objects for each coordi-nate. The block-wise information of the respective entities(servers and mobile objects) can also be included in aLocation object. Furthermore, using these Location ob-jects, DataParser can refer to the LocationHandler classfor sequencing the movement events of all mobile entities.MobilityController : Conventionally, iFogSim requires a

test script where the overall simulation environment, includ-ing the specifications of sensors, actuators, Fog nodes, andapplications are defined. These classes are also set to belinked with the iFogSim simulation engine through a Con-troller object. However, for simulating any mobility-drivenuse cases, a synthesis of such iFogSim core classes andthe newly created mobility-specialised classes, includingDataParser and LocationHandler is required. Therefore,in iFogSim2, we have created a subclass of Controller classnamed MobilityController and encapsulated the object ref-erences of those specialised classes within it so that theycan collectively address the mobility-driven issues. Sincethe Controller class itself is a subclass of SimEntity, Mo-bilityController poses direct access to the SimEvent classof CloudSim 5.0. As a result, it allows the flexibility todynamically initiate the required sequential or parallelevents on different referenced objects of FogDevice andAppModule for mobility management.

In iFogSim2, for initial placement of AppModules, theMobilityController refers to an object of MobilityPlace-mentMobileEdgewards class. This class replicates Module-PlacementEdgewards of core iFogSim with an additional

feature that tracks the Fog node-wise deployment of ap-plication modules/microservices for each mobile entity inthe simulation environment. Once the simulation starts,MobilityController approaches to execute events, such aslaunching of modules and management of resources as perthe initial placement. However, when it encounters anymobility-driven event (e.g., changing of locations) as se-quenced by the LocationHandler class, MobilityControllertriggers a module migration operation. In this case, Mo-bilityController executes the built-in ManageMobilityprocedure as noted in Algorithm 1. This procedure takesthe object reference mobile entitym and the SimEvent trig-gered timestamp t as the arguments (line 1) and consistsof the following five phases. For a better understandingof the algorithm, Java-based object-oriented notations areused in the description.• Initialisation: In this phase, the required initialisation

for the mobility management operation is performed. Atfirst, using the Getter methods of MobilityController class,the object referrals of DataParser, Reference and Location-Handler are extended to the procedure (lines 2-4). Later,the location Ltm of the mobile entity m at timestamp tis determined through the mobileLocation method withinthe DataParser class (line 5). This procedure also iden-tifies the current upper tier contact ρ′ (noted as parentFog node) of m and its operating tier η using the Gettermethods within the mobile entity (lines 6-7). An additionalvariable ρ is also formatted in the procedure to hold thereference of m’s prospective new upper tier contact (dueto location change) residing at the closest distance (line8). For realising this case, another variable δ is set with

Algorithm 1 Mobility Management Logic1: procedure ManageMobility(m, t)2: dP ← getDataParserObject()3: rF ← getReferenceObject()4: lH ← getLocationHandlerObject()5: Ltm ← dP.mobileLoction(m)6: ρ′ ← m.getParent()7: η ← m.getLevel()8: ρ← null9: δ ← rf.getMaxDistance()

10: for fu := Fη−1 do11: Lfu ← dP.fogLoction(fu)12: δ′ ← lH.calculateDistance(Lfu , L

tm)

13: if δ′ ≤ δ then14: δ ← δ′

15: ρ← fu

16: if ρ′ 6= ρ then17: Λ← ρ′.getP lacedModulesOf(m)18: if ρ.inSameClusterOf(ρ′) then19: pushModules(ρ′, ρ,Λ)20: else21: κ← null22: Φ← getNodesInPath(ρ, Cloud )23: Φ′ ← getNodesInPath(ρ′, Cloud )24: for ϕ := Φ do25: for ϕ′ := Φ′ do26: if ϕ = ϕ′ then27: κ← ϕ28: break29: pushModules(ρ′, κ,Λ)30: pushModules(κ, ρ,Λ)

31: m.setParent(ρ)32: ρ.placeModules(m,Λ)33: ρ′.terminateModules(m,Λ)

5

(a) (b) (c)

Figure 2: (a) Block-wise Edge/Fog computing nodes, (b) Directional Movement of an user, and (c) Random Movementof an user in Melbourne Central Business District

a maximum distant value denoted in the Reference class(line 9).• New parent selection: Driven by the changing of lo-

cations, this phase determines the minimum-distant newupper tier contact for the mobile entity m. For such anoperation, Algorithm 1 primarily considers Fη−1 as theset of all upper tier Fog nodes corresponding to m andidentifies the location Lfu of each fu ∈ Fη−1 with the helpof DataParser object line (lines 10-11). Later, the distanceδ′ from m and a candidate fu is calculated. In this case,the calculateDistance method of LocationHandler class isexploited that takes two location objects as arguments andreturns their haversine distance (line 12). Such explorationalso continues for other upper tier nodes, and the Fog nodethat resides at a minimum distance from the mobile entitym is marked as its new upper tier contact or parent nodeρ (lines 13-15).• Intra-cluster module migration: According to Algo-

rithm 1, the migration of application modules occurs whenthe current and new upper tier contact (ρ′ and ρ, respec-tively) of the mobile entity m differs (line 16). To deal withsuch a scenario, firstly, the application modules Λ deployedin ρ′ corresponding to m are identified (line 17). Later,it is checked whether both ρ′ and ρ belong to the sameFog node cluster (line 18). If this condition is satisfied, ρ′simply push the modules Λ to ρ using their shared clustercommunication links (line 19). Such a shifting of mod-ules from one cluster node to another is referred to as theintra-cluster module migration. Conversely, when ρ′ and ρdo not share a common cluster, inter-cluster migration isperformed (line 20).• Inter-cluster module migration: To start this approach,

Algorithm 1 firstly initialises a variable κ which is ulti-mately used to refer a common accessible point for bothρ and ρ′ (line 21). For defining κ, it also includes all Fognodes from ρ and ρ′ to Cloud in separate sets Φ and Φ′,respectively (lines 22-23). Later, each Fog node ϕ ∈ Φand ϕ′ ∈ Φ′ are explored and mutually compared (lines24-25). During the exploration, if any candidate ϕ and ϕ′indicates to the same Fog node, that node is defined tobe κ, and the exploration is immediately terminated (lines26-28). Subsequently, the application modules are pushedfrom ρ′ to κ so that they can be further pushed to ρ′ andmigrated across clusters (lines 29-30).• Update: In the last phase of Algorithm 1, necessary

updates in the simulation environment based on eitherintra- or inter-cluster module migration are made. For

example, the current upper tier contact of mobile entitym is set as ρ (line 31). Finally, the application modules Λcorresponding to m start execution in ρ and ρ′ terminatesthem.

Algorithm 1 is a sample illustration of managing mobil-ity in iFogSim2. From line 10 to 15 of Algorithm 1, thereare O(|Fη−1|) iterations, where |Fη−1| denotes the numberupper tier Fog nodes to the mobile entity m. Additionally,it has O(|Φ| · |Φ′|) iterations from line 24 to 28, where |Φ|and |Φ′| define the number of nodes residing in the pathfrom ρ and ρ′, respectively. Such iterations helps Man-ageMobility procedure to function with polynomial timecomplexity. Nevertheless, using the mobility-specialisedclasses and respective methods of iFogSim2, further com-plex and comprehensive mobility management policies canbe developed. In such cases, Algorithm 1 can also be usedas a benchmark.

3.2 Clustering

In highly integrated computing environments, Edge/Fogand Cloud resources are being simultaneously consideredfor service delivery. Such resources are inherently heteroge-neous with complementary characteristics. Distributed fognodes usually have limited computing and storage resourcescompared to Cloud resources, while they can be accessedwith higher bandwidth and less latency. Therefore, re-source augmentation can greatly help resource-limited fogresources to be used for resource-critical applications, espe-cially computing and storage resources [22]. Accordingly,a clustering mechanism to enable resource augmentationamong Fog resources is of paramount importance. Such aclustering mechanism can also benefit multi-Cloud serviceproviders to communicate more efficiently together.

The Clustering component of iFogSim2 enables dynamiccoordination and cooperation among various nodes in a dis-tributed manner. While each node can probe and registertheir cluster members according to their specific cluster-ing policy, the scheduling and other iFogSim2 features aredecoupled so that clustering can be used for both central-ized and distributed scenarios, such as scheduling, mobilitymanagement, and microservices.ClusteringController, which is extended from Controller,

initiates the process of dynamic clustering among differentnodes. In order to adapt different scenarios, Clustering-Controller can trigger the clustering mechanism on variousoccasions, such as at the beginning of the simulation, af-

6

ter a specific simulation time, after a specific simulationevent, or any combinations of these criteria. Nodes re-ceiving clustering messages in FogDevice can start theirclustering process. The FogDevice is updated with sev-eral parameters to keep the list of cluster members (CMs),bandwidth, and latency among CMs, just to mention a few.It also contains a processClustering method which triggersthe clustering process based on the policy implemented inClustering. As each node runs the clustering process in adistributed manner, different policies can be implementedin Clustering.

The current clustering policy, implemented in Clusteringclass, works based on the communication range and/orlatency among different nodes. In Edge/Fog computing en-vironments, heterogeneous nodes, either wired or wirelessexist. So, clustering policies can use various metrics forthe creation of their clusters. The communication scopeof wireless nodes is usually estimated based on their com-munication ranges. Therefore, for each type of Edge/Fognode, a communication range is defined according to theirantenna’s characteristics. Moreover, each node has a geo-graphical position, defined in the FogDevice. If a datasetfor the position of nodes is available, their geographical po-sition can be parsed using DataParser. Accordingly, eachnode based on its geographical position and communicationrange can probe and create its list of CMs. Furthermore,clustering can be performed based on the average latencyamong each pair of nodes, regardless of whether thesenodes are wireless or wired. In such scenarios, a clusteringcommunication latency threshold is defined, through whicheach node can dynamically create its CMs. Algorithm 2represents an overview of Dynamic Distributed Clustering(DDC). First, each Edge/Fog node retrieves the informa-tion about the location of other Edge/Fog nodes (line 2).The information of nodes’ positions can be obtained byeach node in different ways, such as from 1) a centralizednode 2) From a parent node in a hierarchical approach, or3) GPS. Also, each Fog node is aware of the characteristicsof its immediate parent, children nodes, its communicationrange, and acceptable communication latency (lines 3-6).Next, the latitude and longitude information of the currentEdge/Fog node will be compared by all other availableEdge/Fog nodes and those who are in the communicationrange of the current node will be added to the clustering listof the current node, listcmf (lines 9-16). The calculateIn-Range function is responsible to calculate the distance ofthe current node to other Edge/Fog nodes. The latency ofthe current node to each CM will be estimated and storedin mapCMToLatency (lines 17-18). If the communicationlatency of CMs is a clustering factor (which is checked bythe lf flag), the list of current CMs will be pruned to findthe CMs satisfying latency constraint (lines 19-25). Finally,the list of CMs listcmf of the current Fog node alongsidetheir latency mappings mapCMToLatency will be returnedas the outputs (line 26).

3.3 Microservices

To harvest the full potential of the Edge/Fog comput-ing paradigm, application development has migrated frommonolithic architecture towards microservice architecture.

Algorithm 2 Dynamic Distributed Clustering (DDC)Logic1: procedure ManageClustering(f, t, loc, lf)2: lHI ← loc.locationInfo()3: ρ← f.getParent()4: η ← ρ.getChildren()5: δ ← f.getRange()6: σ ← f.getLatencyThresh()7: listcmf ← null8: mapCMToLatency ← {}9: fx ← lHI.get(f).lat

10: fy ← lHI.get(f).long11: for f ′ := η do12: f ′x ← lHI.get(f ′).lat13: f ′y ← lHI.get(f ′).long14: flag ← calculateInRange(fx, fy , f ′x, f

′y , δ)

15: if flag then16: listcmf .add(f ′)

17: latency ← checkLatency(f, f ′)18: mapCMToLatency.get(f ′)← latency

19: if lf then20: temp← listcmf21: for f ′ := listcmf do22: if mapCMToLatency.get(f ′) > σ then23: temp.remove(f ′)24: mapCMToLatency.remove(f ′)

25: listcmf ← temp

26: return listcmf ,mapCMToLatency

Microservices are designed as small and independent com-ponents responsible for carrying out a well-defined businessfunction, enabling them to be moved between Edge/Fogand Cloud tiers easily [23, 24]. Multiple loosely coupledmicroservices coordinate together to build applications.Because of these characteristics, microservices can scaleup and down independently based on the workload and re-source availability of Edge/Fog nodes. Thus, microserviceorchestration is a crucial process that combines distributedmicroservices to create workflows.The Microservices component of iFogSim2 provides or-

chestration support to maintain seamless coordination be-tween application microservices deployed across Edge/Fogand Cloud resources. To provide microservice orchestra-tion, iFogSim2 models two main features: service discoveryand load balancing, which help simulation of the dynamicnature of microservices within Edge/Fog computing envi-ronments.MicroserviceFogDevice, which is created by extending

FogDevice makes it possible for Edge/Fog nodes to performclient-side service discovery and load balancing to enabledecentralized orchestration among microservices. Once arequest is generated in the form of a Tuple by a consumermicroservice deployed on a node, it uses ServiceDiscoveryto retrieve locations of the service provider and applyLoadBalancer logic to determine destination node to routethe created tuple. To support routing of the tuples whenmultiple instances of the same microservice are available onmultiple Edge/Fog nodes, routing of the tuples is modelledbased on the destination node id of the tuple which is setafter executing the load balancer logic.LoadBalancer and ServiceDiscovery are initialized as

members of the MicroserviceFogDevice. The default imple-mentation of the load balancer logic in iFogSim2 is based

7

on Round Robin Load Balancing where requests are dis-tributed equally among microservice instances. The usersof the iFogSim2 can incorporate different load balancinglogic to simulate the microservice behavior by implementingLoadBalancer interface. ServiceDiscovery stores microser-vice to node mapping which can be dynamically updated atany point of time during the simulation using SimEvents.MicroservicesController which is extended from Con-

troller initiates microservice-based application placementand orchestration. To this end, it initializes the LoadBal-ancer and ServiceDiscovery objects within each Fog nodeof the simulation environment and generates routing datato be used by Edge/Fog node to perform node id basedrouting of data tuples generated by modelled applications.Default implementation contains Shortest Path Routingwith flexibility for the user to incorporate different routingprotocols. MicroservicesMobilityClusteringController ex-tends MicroservicesController and integrates it with theMobility component of iFogSim2 to provide mobility sup-port for microservice applications. It is also integrated withthe Clustering component of iFogSim2 to enable dynamicclustering among Edge/Fog nodes that host microservices.Moreover, this controller implements dynamic updating ofservice discovery information with user mobility-inducedmicroservice deployment and routing data updates due touser movements.MicroservicePlacementLogic is the base class to imple-

ment the microservice application placement policy. Usersof the iFogSim2 can extend this class to implement theirplacement policies. As its outputs MicroservicePlacement-Logic provides two mappings:

1. Microservice to node mapping, which indicateswhere each microservice of the application gets de-ployed.

2. Service discovery information per node, whichis calculated based on the microservice to node map-ping. This ensures that all nodes hosting a clientmicroservice is aware of the locations of the serviceinstances, that are accessed by the said microservice.

Algorithm 3 provides an overview of Scalable Microser-vice Placement Logic (SMP), which is the default microser-vice placement policy available in iFogSim2. It is an edge-ward placement algorithm for microservices, which focuseson horizontally scaling microservices among Edge/Fognodes of the same cluster before moving towards upper-tiernodes of the Edge/Fog hierarchy. First, the placementpolicy identifies leaf to root paths (P ) considered for place-ment (line 5) and initializes the mapPtoNextNode with thefirst eligible node in each path for the placement process(line 6). Leaf to root paths are calculated based on the phys-ical topology created by all available Edge/Fog nodes (F ).Each path in P starts with a user/IoT device and traversesupward within the Edge/Fog hierarchy until it reaches theCloud. For the microservice application, the next eligiblemicroservice for placement is determined by traversingits DAG representation (line 7). A microservice becomeseligible for placement if all predecessor microservices aremapped to nodes. Afterward, the policy iteratively triesto place eligible microservices onto the next eligible node

Algorithm 3 Scalable Microservice Placement (SMP)Logic1: procedure ManageMicroservicePlacement(F, a)2: mapNodeToµInst← {}3: mapNodeToSD ← {}4: mplaced ← {}5: P ← getLeafToRootPaths(F )6: mapPtoNextNode← getNextNode(P )7: m← getNextMicroservice(a,mplaced)8: while m is not null do9: for p := P do

10: f ← mapPtoNextNode.get(p)11: if resourcesavailf ≥ resourcesreqm then12: placeModule(f,m)13: mapNodeToµInst.get(f).add(m)14: f.updateResourcesAvail()15: else16: notP lacedPaths.add(p)

17: for p := notP lacedPaths do18: f ← mapPtoNextNode.get(p)19: F ′ ← f.getCMs()20: placed← false21: for f ′ := F ′ do22: if resourcesavail

f ′ ≥ resourcesreqm then23: placeModule(f ′,m)24: mapNodeToµInst.get(f ′).add(m)25: f ′.updateResourcesAvail()26: placed← true27: break28: while placed is false do29: f ← getNextInPath(p, f)30: if resourcesavailf ≥ resourcesreqm then31: placeModule(f,m)32: mapNodeToµInst.get(f).add(m)33: f.updateResourcesAvail()34: placed← true35: mapPtoNextNode.get(p).set(f)

36: mplaced.add(m)37: m← getNextMicroservice(a,mplaced)

38: mapNodeToSD ← generateSD(mapNodeToµInst)39: return mapNodeToµInst,mapNodeToSD

of each path (lines 9-16). If sufficient resources are notavailable within the considered node, the policy consid-ers cluster members (lines 21-27) before moving onto thenext tier (lines 28-35). After all microservice instances aremapped to nodes, the algorithm generates service discoveryinformation for each node hosting client microservices (line38).

These objects together create a platform to model mi-croservices in Edge/Fog computing environments, whilecapturing their dynamic, independent, and scalable nature.

4 Performance Evaluation

This section discusses the simulation of a set of Edge/ en-vironments using iFogSim2 for different application casestudies, including Audio Translation Service (ATS), Car-diovascular Health Monitoring (CHM), and Crowd-sensedData Collection (CDC). Then, we evaluated the efficiencyof various combinations of iFogSim2’s built-in Mobility,Clustering, and Microservice management policies withrespect to latency, network usage, and energy consumptionfor each case study. We also parameterised the lightweightand modular architecture of iFogSim2 in terms of RAMusage and execution time and compared it with the existing

8

simulators, including IoTSim-Edge [12] and PureEdgeSim[14]. The use cases and the experiment results are discussedbelow.

4.1 Case study 1: Audio Translation Ser-vice (ATS)

Translation service is highly recommended for tourists, es-pecially when they are visiting non-native language speak-ing countries. Currently, Google and Microsoft offer dif-ferent translator services to the users, which mainly dealwith text and imagery inputs [25]. Since the frequencyand variations of such inputs can be easily estimated orcontrolled, their processing is usually performed by fol-lowing a specific set of operations without requiring anyadditional services. As a result, most of state-of-the-artsmartphones can execute these translation services withthe available computing resources they have. However,for audio-based translation, various computation-intensivedata pre-processing operations are required as the pitch in-tensity varies between the users, and the background noisesalways couple tightly with the actual data [26]. Conversely,for smartphones, the real-time adjustment or update ofexternal services for performing these operations is not of-ten feasible due to additional overhead. In such scenarios,the exploitation of Fog computation can be a potentialsolution for Audio Translation Service (ATS).However, in any Fog computing-based ATS system, a

majority of users is expected to be mobile and their smart-phones are regarded as the data sources. Therefore, to meetthe desired QoS, efficient mobility-aware service manage-ment techniques are required for such an ATS. Consideringthis issue, we have modelled a mobility-driven simulationcase study on Fog computing-based ATS in iFogSim2. Thedetails of the application model, simulation parameters,comparing mobility management policies and their perfor-mances for this case study are discussed below.

4.1.1 Application Model

To align with the distributed data flow approach adoptedby core iFogSim, we have modelled the application for ATSas a Directed Acyclic Graph (DAG) (shown in Fig. 3). Itconsists of three application modules, which are describedin the following

• Client module: It is deployed on smartphones thatprimarily grasp audio data from the integrated sensors.The Client module also performs necessary authenti-cations to access the ATS and forwards the data tothe Processing module for further analysis.

• Processing module: It is expected to execute by thetier-2 nodes for faster interactions with the smart-phones. However, various computation-intensive Ar-tificial Intelligence (AI)-enabled audio data analysisoperations including data filtration, noise reduction,pitch classification, and speech segmentation are per-formed by the Processing module. The results of theseanalyses are then pushed back to the Client moduleso that they can be displayed to the user via thesmartphone display.

• Storage module: The Processing module forwardsthe input data and analytical outputs to the Storagemodule for periodic updates of the AI models and thusensures the enhanced performance of the ATS.

Considering the amount of audio data generated throughsuch an ATS, it is recommended to host the Storage modulein Cloud for further scalability.

4.1.2 Simulation Environment

The simulation environment for the ATS use case is madehighly aligned with the EUA dataset of iFogSim2 having118 Fog gateways residing at 12 different blocks acrossthe Melbourne CBD. We assume that the smartphones ofmobile users can connect with any of the gateways (tier-2nodes). The gateways of a particular block can also in-teract with a Cloud datacentre (tier-0 nodes) via a proxyserver (tier-1 nodes). The specifications of the comput-ing infrastructure along with that of application modulesare presented in Table 2. The simulation experiments areconducted on an Intel Core 2 Duo CPU @ 2.33-GHz with2GB-RAM configured computer, and the fractional selectiv-ity of input-output relationship within a module is set to be1.0. The numeric values of the simulation parameters havebeen extracted from the existing literature as mentionedin [27, 28].

4.1.3 Comparing Policies

While imitating the ATS case study in iFogSim2, the move-ment of smart-phones are set to vary using both directionaland random mobility pattern. Furthermore, we have usedthree different mobility management techniques in thesimulated Fog computing environment to deal with suchmovement. These techniques are listed below.

• Cloud-centric migration: In this approach, thecurrent upper tier contact (source gateway) of a mobilesmartphone pushes the respective application modulesdirectly to the Cloud VMs. Later, the Cloud VMspushes the modules to the new upper tier contact(destination gateway) of the smartphone.

• Non-hierarchical migration: By connecting allFog gateways through a mesh communication channel,this approach allows the direct migration of applicationmodules between source and destination. As a result,the upper tier Fog nodes remain uninvolved duringmodule migration, leading it to be a non-hierarchicaloperation.

• Intra/Inter-cluster migration: This approachrefers to the built-in mobility management policy ofiFogSim2 as discussed in Algorithm 1. It only involvesthe upper tier Fog nodes in migrating modules if thesource and destination gateway do not belong to thesame cluster. Here, the node clustering is performedby Algorithm 2.

4.1.4 Results

The performance of the comparing techniques is discussedbelow.

9

Storage

Module

Client

ModuleM-Sensor

M-Display

Processing

Module

m-sensor raw_data

processed_data

action_command

actuation_signal

Figure 3: Application model for the Audio Translation Service (ATS)

Table 2: Simulation parameter for the ATSDuration of experiment : 500 secondsNumber of location change events:140Resource type⇒Configuration ⇓

Cloud VM Proxy server Fog gateway Smart-phone

Numbers 10 12 118 1Speed (MIPS) 4480 3600-4000 2800-3000 500RAM (GB) 16 16 8 1Uplink (MBPS) 100 10 50 100Downlink(MBPS)

100 20 100 200

Busy power(MJ)

1468 428 206 60

Idle power (MJ) 1332 333 170 35Attribute ⇒Module ⇓

RAM (GB) Input (MB) Output (MB) CPU length(MI)

Client 0.10 2 2.5 500Processing 4 2.5 1.5 2500Storage 4 1 1 1000

• Migration time: Fig. 4 depicts the delay in migrat-ing application modules for different comparing mobilitymanagement policies. Since Cloud datacentres reside at amultihop distance from the gateways, the transfer of ap-plication modules to the Cloud and later their forwardingto the destination gateway increases the overall migra-tion delay for the Cloud-centric approach. Conversely, theIntra/Inter-cluster migration technique exploits all possibleoptions to select a common accessible node (CAN) for bothsource and destination gateway not only in the node hier-archy but also in horizontal levels. As a result, it is morelikely to find a CAN in the proximity of the gateways thanthat of a Cloud-centric approach, minimising the migra-tion delay. However, the Non-hierarchical policy providesthe most desirable outcome in this case as it exploits themesh connectivity between source and destination gatewayduring module migration reducing the delay significantly.Nevertheless, as the assurance of large-scale mesh connec-tivity across the gateway is costly, such an approach isfeasible only when user mobility is confined.

Furthermore, directional mobility presents better man-agement of migration delay than random mobility in allcomparing policies. It happens because, in directional mo-bility, user speed remains the same. As a result, despitelocation change, their source and destination gateway areless likely to vary within a short distance. Consequently, itreduces the number of total migration events and lessensthe delay. Conversely, during random mobility, the num-ber of migration events can increase unevenly, resulting inincreased migration delay.

• Network usage: Fig. 5 portrays the network resourceusage during module migration for different simulating tech-

185

107121

218

112

156

0

50

100

150

200

250

Cloud-centric Non-Hierarchical Intra/Inter-ClusterTim

e T

o M

igra

te M

od

ule

s (m

s)

Directional Mobility Random Mobility

Figure 4: Time to migrate application modules

niques. As the realisation of the Cloud-centric approachinvolves multiple nodes within the communication pathbetween gateways and Cloud to migrate modules in bothdirections, it consequently increases their collective net-work usage. Nevertheless, the Intra/Inter-cluster techniqueperforms slightly better in this case as it attempts to re-duce the involvement of intermediate nodes during modulemigration and consequently lessens their network usage.Conversely, the Non-hierarchical approach only exploitsthe network resources which is available within the commu-nication link between the source and destination gateway,resulting in the least usage.Moreover, as the number of migrating events increases

with random mobility, network usage increases. On theother hand, by limiting the occurrence of such events,directional mobility can help to lower network resourceusage.• Energy consumption: Fig. 6 illustrates the con-

10

22.1619.25

28.32

Non-Hierarchical Intra/Inter-Cluster

Random Mobility

18.29

8.37

13.56

22.18

10.15

18.57

0

5

10

15

20

25

Cloud-centric Non-Hierarchical Intra/Inter-Cluster

Net

work

Usa

ge

Du

rin

g

Mig

rati

on

(M

B)

Directional Mobility Random Mobility

Figure 5: Network usage during module migration

sumption of energy during module migration for differentadopted policies. For the Cloud-centric approach, the in-crement in energy usage is obvious as it directly involvesCloud datacentres conventionally consuming a significantportion of energy in the distributed computing ecosystem.The involvement of other intermediate nodes further con-tributes to increasing the overall energy consumption. Therandom mobility of users can also elevate energy usageduring mobility management by increasing the migrationfrequency. In comparison, both Intra/Inter-cluster andNon-hierarchical module migration approaches performwell in managing energy usage as they resist the involve-ment of Cloud and intermediate nodes to a greater extentfor such an operation.

34.85

18.9722.16

36.18

19.25

28.32

0

5

10

15

20

25

30

35

40

Cloud-centric Non-Hierarchical Intra/Inter-Cluster

En

erg

y C

on

sum

ed D

uri

ng

Mig

rati

on

(M

J)

Directional Mobility Random Mobility

Figure 6: Energy consumed during module migration

4.2 Case study 2: Cardiovascular HealthMonitoring (CHM)

Electrocardiogram (ECG) monitoring is a widely usedmethod for diagnosing heart diseases using both reactiveand proactive methods [29, 24]. In IoT-based smart health-care, wearable sensors are used to sense and transmit ECGsignals towards analysis platforms that host applicationsdeveloped for detecting concerning heart conditions. Suchapplications are designed to perform multiple tasks, includ-ing: filtering ECG data to remove data anomalies, ECGfeature extraction and generating emergency warnings inreal-time, long-term collection and analysis of data to makepredictions to ensure preventive measures [22].

Design and development of such IoT applications increas-ingly use modular architectures, especially microservices

architecture to enable balanced deployment of real-timetasks within resource-constrained Fog resources and latencytolerant tasks within Cloud datacentres. Hence, Cardio-vascular Health Monitoring (CHM) is designed followingmicroservice architecture and modelled using DAG-basedmodeling of applications integrated into iFogSim2.

4.2.1 Application model

Fig 7 shows the microservice architecture of the CHM appli-cation, where vertices represent each microservice and edgesdepict the data dependencies among microservices. CHMconsists of four microservices, namely Preprocessing Mi-croservice, Emergency Diagnosis Microservice, PredictionMicroservice, and Client Microservice. The specificationsof each microservice are described in what follows:

• Client Microservice: This is the mobile front end ofthe CHM application. Client microservice is deployedon users’ smartphones and receives raw ECG signalstransmitted by the sensors that are wirelessly con-nected to each smartphone. Also, it is responsible forsending sensor data towards Preprocessing Microser-vice placed in either Edge/Fog or Cloud and displayingresults received after processing.

• Preprocessing Microservice: The Preprocessing mi-croservice performs data cleaning using filters to filterout noise added to ECG sensor data during transmis-sion. Moreover, data anomalies in the sensed data arealso removed before sending data for further process-ing.

• Emergency Diagnosis Microservice: This microserviceis responsible for real-time analysis and identificationof concerning health conditions like heart attacks andsending back a warning signal towards the client mi-croservice to trigger an emergency notification.

• Prediction Microservice: Prediction microservicestores and analyses ECG time series data using ma-chine learning models to predict health risks to thepatients. Prediction reports are sent back to the mo-bile front end to be displayed for users.

These microservices communicate together to monitor andpredict the cardiovascular health of users. Preprocessingand Emergency Diagnosis microservices form a latency-critical service to be placed on Fog or Cloud, based on theplacement policy whereas Prediction Microservice repre-sents a service that requires high computation and storageresources and is expected to be placed in the Cloud.

4.2.2 Simulation Environment

For this case study, a physical topology of 7 Fog nodes isused, which consists of 6 Wifi gateways (tier-2 nodes) con-necting to a single proxy server. Besides, the proxy server(tier-1 nodes) is connected to the Cloud datacentre (tier-0nodes). Also, 25 smartphones with randomly generated lo-cations connect with the Wifi gateways to send ECG sensordata towards the Fog environment. The specifications ofthe computing infrastructure along with that of application

11

Sensor

Display

Client

Microservice

Preprocessing

Microservice

Emergency

Diagnosis

Microservice

Prediction

Microservice

sensor raw_data preprocessed_data

preprocessed_data

emergency_event

prediction_data

emergency_notification

prediction_report

Figure 7: Microservice application model for the Cardiovascular Health Monitoring (CHM)

Table 3: Simulation parameter for the CHMDuration of experiment : 20000 secondsNumber of location change events:140Resource type⇒Configuration ⇓

Cloud VM Proxy server Fog gateway Smart-phone

Numbers 16 1 6 25Speed (MIPS) 2500-3000 2500-3000 2500-3000 500RAM (GB) 16 16 8 1Uplink (MBPS) 100 10 50 100Downlink(MBPS)

100 20 100 200

Busy power(MJ)

107.339 107.339 107.339 87.530

Idle power (MJ) 83.433 83.433 83.433 82.440Attribute ⇒Module ⇓

RAM (GB) Input (MB) Output (MB) CPU length(MI)

Client 0.10 0.5 0.5 1000Preprocessing 0.5 0.5 0.5 2000EmergencyDiagnosis

0.5 0.5 0.5 2500

Prediction 2 0.5 0.5 4000

modules are presented in Table 3. Furthermore, to modelthe physical topology, MicroserviceFogDevcie, Sensor andActuator classes of iFogSim2 are used. The simulationexperiments are conducted on an Intel Core i7 CPU @1.80-GHz with a 4GB-RAM computer and the fractionalselectivity of the input-output relationship within a moduleis set to be 1.0. The numeric values of the simulation pa-rameters have been extracted from the existing literatureas mentioned in [24, 30, 31].

6443.18

142.22 38.960

1000

2000

3000

4000

5000

6000

7000

Edgeward SMP-No-Clustering SMP-Clustering

Av

era

ge

Del

ay

(m

s)

Figure 8: The average delay of the CHM application

4.2.3 Comparing Policies

For the experiments, three different placement techniquesare used, as follows:

• Edgeward: This placement technique only consid-ers vertical scalability of application modules without

taking Fog node clustering and horizontal scalability-based load balancing into consideration.

• SMP-No-Clustering: It uses horizontal scalabilityand load balancing features available in microserviceorchestration of iFogSim2 but does not implementclustering of Fog nodes.

• SMP-Clustering: Such an approach makes use ofboth microservice orchestration and clustering featuresavailable in iFogSim2.

4.2.4 Results

The performance of the comparing techniques are discussedbelow.• Average delay of control loop: The control loop of

the CHM application, which is the most latency-sensitiveloop, consists of Client microservice → Preprocessing Mi-croservice → Emergency Diagnosis Microservice → Clientmicroservice. The less average delay for this control loopdemonstrates better placement decision and coordinationamong computational resources. Fig. 8 depicts the aver-age delay for the execution of this control loop for threeplacement techniques. It depicts that the average execu-tion delay of the control loop significantly decreases forSMP-No-Clustering and SMP-Clustering in comparison tothe Edgeward. Edgeward placement moves microservicesupwards to the Fog hierarchy, such that a single instanceof each microservice is deployed for each edge to Cloudpath. Due to the resource-constrained nature of Fog nodes,

12

this approach places microservice instances in higher Fogtiers, thus increasing average delay. Also, the results showthat the SMP-Clustering outperforms SMP-No-Clustering.Even though SMP-No-Clustering uses horizontal scalability,due to lack of clustering, microservices are scaled amongnodes on multiple Fog tiers whereas, in the case of SMP-Clustering, clusters are dynamically formed among Fognodes of the same hierarchical tier, which enables microser-vices to be horizontally scaled within nodes of the sametier before moving to the upper tier. Hence, the SMP-Clustering scenario places latency-critical microservicescloser to the edge network which results in lower averagedelay.• Energy consumption: Figure 9 shows the amount

of energy consumed by different classes of nodes and totalenergy consumption obtained through each technique. AsFig. 9.a demonstrates, the energy consumption of Cloudresources is higher in the Edgeward technique becauseit places most of the microservices on the Cloud VMs.However, the energy consumption of Cloud resources forSMP-No-Clustering and SMP-Clustering are lower whilethey consume more energy in the Fog tier due to runningmore microservices in the resources of that tier. While en-ergy consumption at different tiers depends on the numberof microservices running on that tier, Fig. 9.b depicts thatthe total energy consumption of all nodes is reduced byin SMP-No-Clustering and SMP-Clustering. It highlightsthe positive effect of more efficient distribution, scaling,and load balancing of microservices in the Fog tier com-pared to more centralized approaches. Also, the results ofSMP-Clustering prove the potential of clustering for betterscaling and load balancing of microservices either verticallyor horizontally.• Network usage: The total amount of data trans-

ferred in the network is an important metric for the evalu-ation of different techniques. Techniques resulting in highdata transmission may lead to congestion in the network,service interruption, or increasing average delay of the ap-plications’ control loop, especially in high-density networks.Fig. 10 illustrates the total network usage of different tech-niques in Megabytes (MB). It demonstrates that Edgewardplacement incurs higher network usage due to excessiveusage of higher tier Fog nodes and Cloud datacentre incomparison to the other two techniques. Furthermore, itshows that SMP-Clustering results in lower network usagecompared to SMP-No-Clustering. Although the clusteringmechanism embedded in the SMP-Clustering requires datatransmission for the formation of clusters, it is a lightweightmechanism in terms of network usage. Hence, in a longsimulation time, SMP-Clustering outperforms other tech-niques due to the efficient usage of the lower tier Fog nodesand better load balancing.

4.3 Case study 3: Crowd-sensed DataCollection (CDC)

Crowd-sensing exploits internet-connected sensors to collectvast amounts of data that can be analysed to retrieve com-plex information. Crowd-sensed Data Collection (CDC)application represents a mobile crowd-sensed scenario thataids urban road network planning. Within urban settings,

road system design and traffic signal controlling are ex-tremely challenging. Thus, these tasks can benefit fromcomplex machine learning algorithms. As such algorithmsrequire large amounts of data for accurate decision making,vehicular crowd-sensing is used as a solution for data collec-tion. Sensors onboard mobile vehicles sense and transmitreal-time location and speed data that can be used toderive traffic conditions of the road networks. Using thismethod, any vehicle can voluntarily share data with thedata analytic platforms, which results in a collection oflarge volumes of data. Such applications can benefit fromFog computing environments to process the data closer tothe edge network, thereby reducing the burden on the datatransmission networks connecting sensors to the Cloud. Sowe design a CDC application following the microservicearchitecture and modelled it using DAG-based modelingof applications integrated with iFogSim2.

4.3.1 Application model

Fig 11 shows the microservice architecture of the CDCapplication, where vertices represent each microserviceand edges depict the data dependencies among microser-vices. CDC consists of two microservices, namely NginxMicroservice, Processing Microservice and a database tostore data for further processing. The specifications of eachmicroservice are described in what follows:

• Nginx Microservice: This is the webserver that actsas the gateway to the processing microservice. Ng-inx microservice receives data, which is generated bythe vehicular sensor network, and routes that datatowards the Processing microservice to perform dataanalytics. Also, it is responsible for load balancingrequests among multiple Processing microservices.

• Processing Microservice: The processing microserviceis responsible for sanitizing the sensor data, extractingfeatures that represent trajectories of vehicles, andsending the processed data towards the Cloud to besaved in a time-series database. Data analytic plat-forms can use crowd-sensed data stored in the databasefor urban planning.

4.3.2 Simulation environment

Table 4 presents the specifications of general simulationparameters used in imitating the CDC case study. Thevalue of simulation parameters within a specific range isdetermined by a pseudo-random number generator. More-over, the computing environment is set to be hierarchicalhaving mobile vehicles at the lowest tier. Tier-2 Fog nodesare marked as the gateway followed by proxy servers andCloud at tier-1 and tier-0, respectively. An Intel Core 2Duo CPU @ 2.33-GHz with 2GB-RAM configured com-puter has been used to execute the simulation script andperform the experiments. The numeric values of the simu-lation parameters have been extracted from the existingliterature as mentioned in [24, 27].

13

0

5

10

15

20

25

30

35

Edgeward SMP-No-Clustering SMP-Clustering

En

erg

y C

on

sum

pti

on

(M

J)

Cloud Fog (Proxy+GW) Fog-Proxy Fog-GW

41.0

41.2

41.4

41.6

41.8

42.0

42.2

42.4

42.6

42.8

43.0

Edgeward SMP-No-Clustering SMP-Clustering

To

tal

En

erg

y C

on

sum

pti

on

(M

J)

(a) (b)

Figure 9: Energy consumption for running the CHM application (a) Energy consumed by the resources of differenttiers, (b) Total energy consumption of resources

Table 4: Simulation parameter for the CDCDuration of experiment : 500 secondsTuple generation rate : 5/secondsMobility interval : 10-50 secondsResource type ⇒Configuration ⇓

Mobile vehicles Tier-2 Node Tier-1 Node Tier-0 Node

Percentage of nodes 30% 30% 20% 20%Speed (MIPS) 500-1000 2000-2500 3000-2500 4000-5000RAM (GB) 2 4 8 16Uplink (MBPS) 100 50 10 100Downlink (MBPS) 200 100 50 150Busy power (MJ) 50-100 200-300 400-600 1500-2000Idle power (MJ) 20-30 80-100 150-200 700-900

1625.19

363.13243.28

0

200

400

600

800

1000

1200

1400

1600

1800

Edgeward SMP-No-Clustering SMP-Clustering

To

tal

Net

wo

rk U

sag

e (M

B)

Figure 10: The total network usage of the CHM application

4.3.3 Comparing Simulators

The simulation environment for the CDC use case hasbeen implemented on three different simulators includ-ing IoTSim-Edge [12] and PureEdgeSim [14] along withiFogSim2. These simulators have been selected because oftheir recent inclusions in the literature and open sourcelicense. Furthermore, their Java programming language-based implementation does not arise any compatibilityissues with the proposed iFogSim2 simulator, which alsohelps in reconstructing the experimental results. A briefdiscussion of the simulators is given below.

• IoTSim-Edge: This monolithic simulator supportsthe imitation of microservices in form of microelementsand provides support for customising the user mobility;however, lacks abstractions for node clustering.

• PureEdgeSim: It supports the modularisation ofdifferent simulation components and facilities qualita-tive allocation of tasks using a built-in Fuzzy inference

engine; nevertheless, barely provides functionalitiesfor node clustering and microservice management.

• iFogSim2: The proposed simulator is well-equippedwith APIs and built-in policies for illustrating mobility,microservice, and node clustering-related use cases inEdge/Fog computing environments. The function ofits different components can also be tuned as per thecase studies to create variations in the simulations.

4.3.4 Results

The simulation experiments on the CDC use case study areexclusively exploited to demonstrate the efficacy of IoTSim-Edge, PureEdgeSim, and iFogSim2 simulators in termsof supporting mobility, microservice, and node clusteringissues. The results are discussed below.• RAM usage: Table 5 illustrates the RAM usage of dif-

ferent simulators for varying simulation configurations. Asnoted, IoTSim-Edge supports the imitation of device mobil-ity and microservice orchestration in Edge/Fog computingenvironments. However, as it does not facilitate modulari-sation of the simulation components and mostly operatesin a monolithic manner, its RAM usage does not vary forMobility and Mobility+Microservices simulation configura-tions. Conversely, PureEdgeSim only supports the Mobilityconfiguration. Nevertheless, this simulator consumes moreRAM than other simulators because of its built-in Fuzzyinference engine, supporting task allocation based on qual-itative features. On the other hand, iFogSim2 supports awide range of simulation configurations, including Mobility,Mobility+Microservices, Mobility+Clustering, Microser-vices+Clustering and Mobility+Microservices+Clustering.Despite facilitating such configurations, the RAM usagefor iFogSim2 does not increase significantly because of its

14

SensorNginx

Microservice

Processing

Microservice

sensor raw_data processed_data

Figure 11: Microservice application model for the Crowd-sensed Data Collection (CDC)

Table 5: RAM usage for different simulatorSimulators ⇒

Variations ⇓

IoTSim-Edge

PureEdge-Sim

iFogSim2

Mobility 26% 38% 12%Mobility+Microservices 26% - 21%Mobility+Clustering - - 19%Microservices+Clustering - - 15%Mobility+Microservices+Clustering- - 32%

0

10

20

30

40

50

60

70

10 20 30 40 50

Sim

ula

tion

Pro

cess

ing

Tim

e (m

s)

Number of Location Change Events

iFogSim2 PureEdgeSim IoTSim-Edge

Figure 12: Simulation processing time for different simula-tors

modular architecture and lightweight built-in service andresource management policies.• Simulation time: Fig. 12 depicts the simulation

time of IoTSim-Edge, PureEdgeSim and iFogSim2 for Mo-bility configuration. As PureEdgeSim deliberately exploitsthe Fuzzy inference for managing mobility, it requires moretime to imitate the effect of changing locations, which alsoelevates with the increasing number of such events. How-ever, iFogSim2 performs well in this case because of its lowcomplexity built-in mobility management techniques. Itsmodular architecture further helps to outperform IoTSim-Edge, which cannot ensure mobility management in asegmental manner.

5 Conclusions and Future Work

Efficient resource management in Edge/Fog computingenvironment is an important challenge due to the dynamicand heterogeneous nature of Edge/Fog nodes and IoT de-vices. In this paper, we put forward iFogSim2 simulator,which is an extension of the iFogsim simulator, to addressservice migration for different mobility models of IoT de-vices, distributed cluster formation among Edge/Fog nodesof different hierarchical tiers, and microservice orchestra-tion. To support different simulation scenarios, the newcomponents of the iFogSim2 simulators are loosely coupled,so that components (Mobility, Clustering, and Microser-vices) can be solely used for the simulation, or they can beintegrated for more complex scenarios. Besides, to enhance

the usability of iFogSim2, several case studies and testscripts are implemented and integrated with this simulator,which simplifies the process of defining new policies andcase studies for its users. The results demonstrate theeffectiveness of using iFogSim2 for different case studiesand also prove its low footprint compared to other relatedsimulators.As a future work, iFogSim2 simulator can be further

improved by integration of monetary-based policies, simu-lating distributed ledgers across Fog nodes, setting differentcommunication profiles for sensors such as LoRa and Blue-tooth, and simulating distributed and federated machinelearning approaches.

Software Availability

The source code of the iFogSim2 simulator is accessiblefrom: https://github.com/Cloudslab/iFogSim

References

[1] The World Economic Forum, “How muchdata is generated each day?” https://www.weforum.org/agenda/2019/04/how-much-data-is-generated-each-day-cf4bddf29f/,2019, online; accessed 29 August 2021.

[2] M. Afrin, J. Jin, A. Rahman, A. Gasparri, Y.-C. Tian,and A. Kulkarni, “Robotic edge resource allocationfor agricultural cyber-physical system,” IEEE Trans-actions on Network Science and Engineering, pp. 1–1,2021.

[3] R. Mahmud and A. N. Toosi, “Con-Pi: A distributedcontainer-based edge and fog computing framework,”IEEE Internet of Things Journal, pp. 1–1, 2021.

[4] Q. Deng, M. Goudarzi, and R. Buyya, “FogBus2: alightweight and distributed container-based frameworkfor integration of iot-enabled systems with edge andcloud computing,” in Proceedings of the InternationalWorkshop on Big Data in Emergent Distributed Envi-ronments, 2021, pp. 1–8.

[5] J. Mass, S. N. Srirama, and C. Chang, “STEP-ONE:simulated testbed for edge-fog processes based on theopportunistic network environment simulator,” Jour-nal of Systems and Software, vol. 166, p. 110587, 2020.

[6] M. Goudarzi, M. Palaniswami, and R. Buyya, “A fog-driven dynamic resource allocation technique in ultradense femtocell networks,” Journal of Network andComputer Applications, vol. 145, p. 102407, 2019.

15

[7] M. Merenda, C. Porcaro, and D. Iero, “Edge machinelearning for AI-enabled IoT devices: a review,” Sen-sors, vol. 20, no. 9, p. 2533, 2020.

[8] R. Mahmud, K. Ramamohanarao, and R. Buyya,“Edge affinity-based management of applications infog computing environments,” in Proceedings of the12th IEEE/ACM International Conference on Utilityand Cloud Computing, ser. UCC’19. New York, NY,USA: ACM, 2019, p. 61–70.

[9] H. Gupta, A. Vahid Dastjerdi, S. K. Ghosh, andR. Buyya, “iFogSim: A toolkit for modeling and sim-ulation of resource management techniques in theinternet of things, edge and fog computing environ-ments,” Software: Practice and Experience, vol. 47,no. 9, pp. 1275–1296, 2017.

[10] C. Sonmez, A. Ozgovde, and C. Ersoy, “Edge-cloudsim: An environment for performance evaluationof edge computing systems,” Transactions on Emerg-ing Telecommunications Technologies, vol. 29, no. 11,p. e3493, 2018.

[11] T. Qayyum, A. W. Malik, M. A. K. Khattak,O. Khalid, and S. U. Khan, “FogNetSim++: A toolkitfor modeling and simulation of distributed fog en-vironment,” IEEE Access, vol. 6, pp. 63 570–63 583,2018.

[12] D. N. Jha, K. Alwasel, A. Alshoshan, X. Huang, R. K.Naha, S. K. Battula, S. Garg, D. Puthal, P. James,A. Zomaya et al., “IoTSim-Edge: A simulation frame-work for modeling the behavior of internet of thingsand edge computing environments,” Software: Prac-tice and Experience, vol. 50, no. 6, pp. 844–867, 2020.

[13] C. Puliafito, D. M. Goncalves, M. M. Lopes, L. L.Martins, E. Madeira, E. Mingozzi, O. Rana, and L. F.Bittencourt, “MobFogSim: Simulation of mobility andmigration for fog computing,” Simulation ModellingPractice and Theory, vol. 101, p. 102062, 2020.

[14] C. Mechalikh, H. Taktak, and F. Moussa, “PureEd-geSim: A simulation toolkit for performance evalua-tion of cloud, fog, and pure edge computing environ-ments,” in Proceedings of the International Conferenceon High Performance Computing Simulation (HPCS),2019, pp. 700–707.

[15] J. Mass, S. N. Srirama, and C. Chang, “STEP-ONE:Simulated testbed for edge-fog processes based on theopportunistic network environment simulator,” Jour-nal of Systems and Software, vol. 166, p. 110587, 2020.

[16] I. Lera, C. Guerrero, and C. Juiz, “YAFS: A simulatorfor IoT scenarios in fog computing,” IEEE Access,vol. 7, pp. 91 745–91 758, 2019.

[17] M. Salama, Y. Elkhatib, and G. Blair, “IoTNetSim: Amodelling and simulation platform for end-to-end iotservices and networking,” in Proceedings of the 12thIEEE/ACM International Conference on Utility andCloud Computing, ser. UCC’19. New York, NY, USA:ACM, 2019, p. 251–261.

[18] J. Wei, S. Cao, S. Pan, J. Han, L. Yan, and L. Zhang,“SatEdgeSim: A toolkit for modeling and simulation ofperformance evaluation in satellite edge computing en-vironments,” in Proceedings of the 12th InternationalConference on Communication Software and Networks(ICCSN), 2020, pp. 307–313.

[19] G. Amarasinghe, M. D. de Assuncao, A. Harwood,and S. Karunasekera, “ECSNeT++: A simulator fordistributed stream processing on edge and cloud envi-ronments,” Future Generation Computer Systems, vol.111, pp. 401–418, 2020.

[20] K. Alwasel, D. N. Jha, F. Habeeb, U. Demirbaga,O. Rana, T. Baker, S. Dustdar, M. Villari, P. James,E. Solaiman, and R. Ranjan, “IoTSim-Osmosis: Aframework for modeling and simulating iot applica-tions over an edge-cloud continuum,” Journal of Sys-tems Architecture, vol. 116, p. 101956, 2021.

[21] A. N. Toosi, R. Mahmud, Q. Chi, and R. Buyya,“Management and orchestration of network slices in5g, fog, edge, and clouds,” Fog and Edge Computing:Principles and Paradigms, vol. 8, pp. 79–96, 2019.

[22] M. Goudarzi, M. Palaniswami, and R. Buyya, “Adistributed application placement and migration man-agement techniques for edge and fog computing envi-ronments,” in Proceedings of the 16th Conference onComputer Science and Information Systems (FedCSIS2021). IEEE Press, USA, 2021.

[23] C. T. Joseph and K. Chandrasekaran, “Straddling thecrevasse: A review of microservice software architec-ture foundations and recent advancements,” Software:Practice and Experience, vol. 49, no. 10, pp. 1448–1484,2019.

[24] S. Pallewatta, V. Kostakos, and R. Buyya,“Microservices-based iot application placement withinheterogeneous and resource constrained fog computingenvironments,” in Proceedings of the 12th IEEE/ACMInternational Conference on Utility and Cloud Com-puting, 2019, pp. 71–81.

[25] V. Kepuska and G. Bohouta, “Next-generation of vir-tual personal assistants (microsoft cortana, apple siri,amazon alexa and google home),” in Proceedings ofthe IEEE 8th annual computing and communicationworkshop and conference (CCWC). IEEE, 2018, pp.99–103.

[26] S. Yoon, S. Byun, and K. Jung, “Multimodal speechemotion recognition using audio and text,” in Pro-ceedings of the IEEE Spoken Language TechnologyWorkshop (SLT). IEEE, 2018, pp. 112–118.

[27] F. H. Rahman, T. W. Au, S. S. Newaz, and W. S. H.Suhaili, “A performance study of high-end fog andfog cluster in iFogSim,” in Proceedings of the Inter-national Conference on Computational Intelligence inInformation System. Springer, 2018, pp. 87–96.

16

[28] X. Liu, L. Fan, J. Xu, X. Li, L. Gong, J. Grundy, andY. Yang, “FogWorkflowSim: an automated simulationtoolkit for workflow performance evaluation in fogcomputing,” in Proceedings of the 34th IEEE/ACMInternational Conference on Automated Software En-gineering (ASE). IEEE, 2019, pp. 1114–1117.

[29] O. Cheikhrouhou, R. Mahmud, R. Zouari, M. Ibrahim,A. Zaguia, and T. N. Gia, “One-dimensional CNN ap-proach for ECG arrhythmia analysis in fog-cloud envi-ronments,” IEEE Access, vol. 9, pp. 103 513–103 523,2021.

[30] M. Goudarzi, H. Wu, M. Palaniswami, and R. Buyya,“An application placement technique for concurrent iotapplications in edge and fog computing environments,”IEEE Transactions on Mobile Computing, vol. 20,no. 4, pp. 1298–1311, 2020.

[31] X. Xu, Q. Liu, Y. Luo, K. Peng, X. Zhang, S. Meng,and L. Qi, “A computation offloading method over bigdata for iot-enabled cloud-edge computing,” FutureGeneration Computer Systems, vol. 95, pp. 522–533,2019.

17