1Cloud of Things for Sensing-as-a-Service:Architecture, Algorithms, and Use Case

Sherif Abdelwahab∗, Bechir Hamdaoui∗, Mohsen Guizani†, and Taieb Znati‡

? Oregon State University, abdelwas,hamdaoui@eecs.orst.edu† University of Idaho, mguizani@ieee.org

‡ University of Pittsburgh, znati@cs.pitt.edu

Abstract—We propose Cloud of Things for Sensing-as-a-Service: a global architecture that scales up cloudcomputing by exploiting the global sensing resources ofthe Internet of Things (IoT) to enable remote sensing.Cloud of Things enables in-network distributed processingof sensors data offered by the globally available IoTdevices and provides a global platform for meaningfuland responsive data analysis and decision making. Wepropose a distributed sensing resource discovery andvirtualization algorithms that efficiently deploy virtualsensor networks on top of a subset of the selected IoTdevices. We show, through analysis and simulations, thepotential of the proposed solutions to realize virtual sensornetworks with minimal physical resources, reduced com-munication overhead, and low complexity. We also designan uncoordinated, distributed algorithm that relies on theselected sensors to estimate a set of parameters withoutrequiring synchronization among the sensors. Our simula-tions show that the proposed estimation algorithm, whencompared to conventional ADMM (Alternating DirectionMethod of Multipliers), reduces communication overheadsignificantly without compromising the estimation error. Inaddition, the convergence time, though increases slightly,is still linear as in the case of conventional ADMM.

I. INTRODUCTION

Remote sensing applications will evolve through on-demand sensing services provided by the global networkof sensor equipped devices in our homes, factories,cities, and bodies known as the Internet of Things (IoT).Today in smatphones ’only’, there are seven sensors onaverage per device including: magnetometer, barometer,light, heart, humidity, and temperature sensors that onecan use as participatory sensors to carry out applicationslike short-term weather forecasting [1], [2]. The den-sity of smartphones’ sensors in London today exceeds14,000 sensor per square kilometer1. By 2020, theglobal number of sensor-equipped and location-awaredevices (e.g. wearable, smart home, and fleet manage-ment devices) will reach tens of Billions, potentiallycreating dense, dynamic, location-aware, and onerous to

Copyright (c) 2012 IEEE. Personal use of this material is permitted.However, permission to use this material for any other purposesmust be obtained from the IEEE by sending a request to pubs-permissions@ieee.org.

1The population density in London exceeds 4,000 inhabitants persquare kilometer and the UK smartphone penetration reaches 55%.

manage networks of devices that can realize the visionof providing a versatile remote sensing services, knownas ’Sensing as a Service’ [3], [4], [5].

We conjecture that employing IoT devices’ sensingresources in a cloud computing like platform to supportremote sensing applications may be an effective ap-proach to realize the Sensing as a Service vision [3]. Theidea is to dynamically augment and scale up existingcloud resources (compute, storage, and network) byexploiting sensing capabilities of devices through cloudagents near the network edge to form a global systemnamed the Cloud of Things (see Fig. 1). The Cloudof Things is a geographically distributed infrastructurewith cloud agent elements that continuously discoverand pool sensing resources of IoT devices to be usedby cloud users on-demand. This infrastructure provideselastic sensing resources that scale up and down accord-ing to remote sensing applications’ demands, providingan optimized and controllable sensing resource utiliza-tion and pricing based on measurable usage.

Cloud of Things shifts the current, conventional re-mote sensing use of cloud platforms from a ’collectsensor data now and analyze it later’ scenario to a usagescenario that directly provides meaningful informationfrom in-network processing of sensing data by IoT de-vices. Without such a conjectured infrastructure, remotesensing users can still gain access to sensing resourcesthrough conventional cloud back-end systems (see [6],[4]), with less opportunities to scale out sensing ap-plications over the globally available sensing resourcesand with intolerable performance to applications thatrequire responsive exploitation and fusion of sensingdata and agile in-network decisions (e.g. localization[7] and estimation [8]).

A. Cloud of Things InfrastructureCloud platforms near the network edge already ex-

ist in different forms such as smartphones, personalcomputers, gateways, and servers to offer computationoffloading to nearby devices in real-time (e.g. cloudlets[9], [10] and edge computing platforms [11], [12]).We envision a new role of edge platforms as cloudagents that incorporate IoT devices as sensing resources(Fig. 1) to scale up the conventional cloud with globaland location specific sensing resources. We propose

Fig. 1: Sensor network virtualization in Sensing as a Service by different cloud agents near the edge. First tierclouds are conventional cloud computing platforms, and cloud agents are edge computing platform with evolvedrule for Sensing as a Service. Arrows and numbers illustrate messages flow and sequence of the proposed usagescenario.

algorithmic solutions that provide: (1) fast discoveryof devices’ dynamic sensing resources in specific ge-ographical areas, (2) optimized device virtualization toserve as virtual sensor networks by exploiting the dis-covered sensing resources, and (3) efficient in-networkprocessing of sensing data from unreliable but densesensors in IoT devices.

Cloud agents implement remote sensing applicationsas virtual sensor networks to be deployed on virtualiz-able IoT devices in a geographical area. A virtual sensornetwork performs distributed in-network processing ofsensing data such as: aggregation, feature extraction,belief propagation, and consensus estimation to serveapplications such as: distributed computer vision, dataanalytics, or on-demand context awareness. These vir-tual sensor networks may employ devices’ sensingresources that are discovered by the various multiplecloud agents. Conventional cloud platforms provide aunified interface to cloud users to seamlessly use suchglobal sensing resources from anywhere and at anytimewhile hiding complexities and supporting interactionbetween cloud agents. In Cloud of Things, IoT devicesbecome surrogates of federated sensor networks (i.e ad-ministrated by a single organization) that can potentiallyreduce the total cost of ownership for remote sensingapplications.

However, the IoT devices usually incorporate cheapand unreliable sensors to serve specific task that is notintended for remote sensing applications. For example,augmented reality applications in smartphones makeuse of the measurements from magnetic field sensors.

(a) Magnetic field sensor readingclose to a window (low energyenvironment).

(b) Same sensor reading closeto a power source (high energyenvironment).

Fig. 2: Example of energy profiling from cheap mag-netic field sensor in smartphones.

The same magnetic field sensors can be used to profileenergy levels in different environment (see Fig. 2 foran example). The main problem with remote sensingapplications based on IoT devices sensors is that in-dividual measurements from the sensors of a singleIoT device (e.g. magnetic field) are insufficient for areliable sensing task. In Fig. 2, it is hard to distinguishthe real context of a magnetic sensor reading changes;

2

whether it is a result of user proximity to the device,changes in the device’s orientation, or presence in ahigh energy environment. Similar unreliability problemsappear in traffic congestion estimation in navigationapplications (e.g. Google maps), weather predictionfrom smartphones barometers, or indoor localization.

A virtual sensor network of a group of independentIoT devices can solve this problem through distributedconsensus estimation. Instead of analyzing measure-ments from an individual sensor, the virtual network useindependent sensor measurements from several devicesand executes an efficient in-network distributed consen-sus estimation algorithm to be able to efficiently achieveits goal (e.g. estimating energy level in an environmentsurrounding a user). In this paper, we focus our analysisand evaluation on distributed consensus estimation asthe sensing task under study.

B. Contribution and OrganizationIn this paper, we propose a system to perform in-

network analytics, such as distributed parameter esti-mation, based on commodity IoT devices that act assurrogates of wireless sensor networks (i.e. virtual sen-sor networks). We design a virtualization algorithm thatsuits the use case of describing analytics as on-demandvirtual sensor networks and the challenges of the con-jectured architecture in Fig. 1. We also propose a dis-tributed consensus parameter estimation algorithm to beexecuted by the optimized virtual sensor network. Thedistributed consensus algorithm provides a reliable, highquality parameter estimates from the low-quality andunreliable sensors in commodity IoT devices.

We discuss the technical challenges and our envi-sioned use case of the proposed architecture in Sec-tion II. In Section III, we first propose a sensing resourcediscovery algorithm that uses a gossip policy for prop-agating a sensing task requirements to devices (or theirvirtual instance at the edge cloud) and selects feasibledevices to execute the task while responding to thedynamic changes of devices as fast as possible. Then,we propose RADV, an efficient virtualization algorithm,that deploys a virtual sensor network corresponding tothe sensing task on top of a subset of the selecteddevices with minimal physical resources. In Section IV,we propose RADE; an efficient estimation algorithmthat relies on the virtual sensor network, formed by ourproposed virtualization algorithm, to estimate a set ofunknown parameters in a distributed way and withoutrequiring synchronization among the IoT devices. Wediscuss several related work in Section VI. Finally,we numerically evaluate our proposed algorithms inSection V and conclude this paper in Section VII.

II. ARCHITECTURE USABILITY AND CHALLENGES

The proposed Cloud of Things architecture allowscloud users to run remote sensing tasks, with certainspecifications, virtually on any sensor-equipped IoTdevices (see Fig. 1). For example, a cloud user can

profile pollution changes in cities from real-time temper-ature and CO2 concentration measurements collected bysensors in vehicles with defined precision and accuracy.The architecture consists of three main elements: IoTdevices, first tier clouds, and cloud agents. IoT devicesare sensor-equipped devices that can serve both specificand general purpose remote sensing applications. Firsttier clouds are conventional cloud platforms that pro-vide unified interfaces to users to access the system andhide complexities underlying the realization of sensingservices. Throughout, we refer to a first tier cloudas simply ’cloud’. Finally, cloud agents are trustedand resource-rich elements near the network edge thatare well-connected to the Internet and to conventionalcloud platforms. Cloud agents can be as powerful assupercomputers, or as flexible as smartphones accordingto the types of devices they serve and the computingresources these devices demand. Throughout, we referto a cloud agent simply as ’agent’.

This architecture offers new sensing features andservice level guarantees with several benefits. Deploy-ing agents (cloud agents) close to devices improvesresponsiveness to sensing task requests and enablesaccess to a globally available sensing resources. Fromthe devices’ viewpoint, cloud resources can be splitinto local resources (agents’ resources) and global re-sources (clouds’ resources) that can improve resiliencyby migrating sensing tasks as the states of the de-vices - which carry out the sensing task - change. Acloud (first tier cloud) also acts as liaison to supportcoordination between distributed agents, while theseagents can rapidly capture dynamics of the devices(e.g. utilization, connectivity, and availability). Thisapproach simplifies analytics and big data with possibledirect device access for agile in-network data processingand decision making. The proposed architecture finallyallows the design of network-aware and performance-optimized cloud procedures.

A. Use case and System Model

Fig. 1 summarizes message sequence and flow be-tween the different architectural elements. These aredetailed as next.

1) First tier clouds: A cloud (first tier cloud) handlessensing tasks initiated by a user with a unified interface(step 1 in Fig. 1). A sensing task defines physicalparameters (e.g pollution changes) that the user wishesto estimate in a defined geographical area during a pre-defined time with certain sensing capabilities of the IoTdevices carrying out the task. The sensing task objectivecan be: information retrieval of raw sensed data, orexecution of distributed algorithms on a virtual sensornetwork deployed on multiple interconnected devices.We represent a sensing task by the triple, 〈g, c, δ〉, whereg denote the number of virtual sensors requested toperform the sensing task and the two parameters c andδ define the center and the radius of a geographical areaof interest to the user’s remote sensing application.

3

2) Sensing task requests: The cloud translates a sens-ing task to a corresponding sensing task request that itsends to its agents. A sensing task request defines thevirtual sensors set, V , to be deployed on g connecteddevices, which are all located within distance δ fromthe area center c. For each virtual sensor j ∈ V , thecloud defines a minimum sensing capability, R(j). Theminimum sensing capability represents the minimumstorage capacity, the minimum CPU computing power,and/or the minimum amount of time that devices (tocarry out the sensing task) must fulfill. The cloud mayalso choose a suitable virtual topology that interconnectsthe virtual sensors so that they can execute distributedalgorithms for in-network processing of devices’ sensedmeasurements.

For example, for aggregation and belief propagationalgorithms, the cloud organizes the virtual sensor net-work as a spanning tree. A star topology can also beadopted for distributed algorithms that are implementedusing the map-reduce or graph-processing paradigms.Although consensus algorithms, which are our mainfocus, can run with any arbitrary topology, we showthat a complete topology results in faster convergence.For our evaluation, we focus on three common virtualtopologies: complete, cyclic, and star. For a given topol-ogy, let E denote the set of virtual links connectingthe virtual sensors and Υ = (V,E) be the graph datastructure that represents the virtual sensor network ofthe virtual sensors (connected according to the givenvirtual topology). After translating a sensing task to itscorresponding sensing task request, the cloud sends thisrequest (i.e. the graph data structure Υ) to its agents(step 2 in Fig. 1).

3) The IoT devices capabilities: Agents manage alarge number of interconnected IoT devices. A devicei, at time t, maintains its geographical location denotedby loc(i) and its current sensing capability denoted byC(i). C(i) defines the currently allowed sensing time,available processing capacity, and/or available memorycapacity that the i-th device can allocate (at time t)to fulfill the minimum sensing capability demandedby a virtual sensor j (i.e. R(j)) to be deployed oni. The sensing capability of a device can correspondto local device’s resources (i.e. CPU, memory, storage,and sensors) or to resources at the edge cloud (agent)that the device may opportunistically use through com-putation offloading mechanisms. We also assume thattwo devices can directly communicate with each otherif they are within a transmission radius r. We model thenetwork of all n devices, connected to a single agent,as the Euclidean geometric random graph, G = (S,L),where S denote the set of n devices, and L denote theset of all links connecting the devices. We assume thateach sensor i ∈ S is capable of estimating a vector ofunknown parameters, θ ∈ RN , through noisy sensorsmeasurements, xi ∈ RM . That is,

xi = Hiθ + ui, i = 1, . . . , n

where Hi ∈ RM×N is sensor i’s sensing model (typ-ically known to i only) relating xi to θ, and ui is anadditive Gaussian noise with zero mean and varianceσ2i . We assume that ui and uj are independent from

one another for all i, j ∈ S. Because different sensorsmay have different sensing models and/or differentmeasurement methods, it is very likely that differentsensors have different estimates of θ. Also, we do notassume/require that the sensors are synchronized; thatis, the consensus algorithms we develop in this paperto estimate θ are asynchronous.

4) Service Level Agreement (SLA) implications onagents configurations: Agents handle sensing task re-quests under agreed SLAs with users through the cloud.An SLA generally consists of: i) a maximum timewithin which the sensing task must be completed, ii) afeasible selection of IoT devices to carry out the sensingtask under certain tolerances of the results accuracy,and iii) a maximum task rejection rate defined as theratio of the number of failures to handle sensing taskrequests to the total number of requests. The cloudtranslates an SLA to parameters that agents can use intheir algorithmic solutions to discover sensing resourcesand virtualize devices efficiently. Defining all possibleparameters that reflect any SLA is beyond the scope ofthis work and we consider only four parameters.

The first parameter is the absolute error of theestimated parameter θ, denoted by εabs. The secondparameter is the relative error, εrel, of the parameterθ estimated by the different virtual sensors such thatall sensors estimate θ within εrel. The third parameter,defined earlier, is the minimum sensing capability R(j)of the j-th virtual sensor. The fourth parameter is themaximum allowed path length, h, between any pair ofvirtual sensors. h limits the number of devices/hopsa message, exchanged between virtual sensors, can gothrough.

A virtual link between two virtual sensors may mapto devices that do not necessarily deploy a virtualsensor and the virtual sensor network just use thesedevices for message forwarding. We use h to imposean upper limit on these intermediate devices for tworeasons. First, restricting the number of intermediatedevices shall bound the sensing task performance by anSLA. Second, the sensor discovery and virtualizationalgorithms shall use the least number of hops and theleast possible physical resources when mapping thevirtual network, so as to maximize the Sensing-as-a-Service benefits (step 3 in Fig. 1). The implication ofh on the virtualization design will be discussed later inthis section.

5) Sensing task execution (consensus): Consensusestimation resembles the most commonly used sensingtask relying on a collection of measurements fromunreliable sensors. In consensus estimation, the sensingtask is to estimate a set of parameters, θ, based onthe measurements sensed by the IoT devices so thatthe estimated parameters are at most εabs away fromtheir actual value, and so that all the sensors consent

4

to the same estimate value of θ with a tolerance of εrelaccording to the SLA.

Without loss of generality, consider indexing theselected g sensors in the virtual sensor network as 1 . . . g

and let x =[xT1 , . . . , x

Tg

]T, H =

[HT

1 , . . . ,HTg

]T, and

u =[uT1 , . . . , u

Tg

]T. The combined measurements can

then be written as x = Hθ + u. One simple approachof estimating θ is to have the cloud agent first collectfrom each virtual sensor i its measurement vector, xi,and its sensing model, Hi, and then solve the followingLeast Squares (LS) problem

minimize 12‖x−Hθ‖

2 (1)

where θ is here the optimization variable. The unbi-ased maximum-likelihood (ML) estimate of θ is simplyθLS =

(HTH

)−1HTx.

B. Technical Challenges and Solutions ObjectivesThe proposed Cloud of Things architecture and use

case envision designing algorithmic solutions with spe-cific objectives, given the following set of challenges:

1) Sensing resource discovery: In the sensor networkvirtualization (step 3 in Fig. 1), an agent searches for de-vices with sensing capabilities that meet the sensing taskrequirements specified by the virtual sensor networkdata structure Υ. For a given Υ, the agent discoversdevices’ sensing capabilities and searches for a subsetof devices, S′ ⊂ S, such that a device i ∈ S′, if it isgeographically located within δ distance from the centerc, and the discovered sensing capability C(i) satisfiesthe minimum sensing capability R(j) demanded by atleast one virtual sensor j ∈ V . We define the virtualdomain, D(i), of a device i as

D(i) = {j ∈ V : C(i) ≥ R(j), ‖loc(i)− c‖ ≤ δ} (2)

henceS′ = {i ∈ S : |D(i)| > 0}. (3)

The design objective of a sensing resource discoveryalgorithm is to construct the virtual domains, D(i)for all i ∈ S, as fast as possible and with minimalcommunication overhead between the agent and thedevices and between the devices themselves.

The challenges related to sensing resource discoveryarise from the large number of devices and their onerousto maintain dynamics. The large number of devicesconnected to an agent requires a scalable solution todiscover devices’ sensing capabilities and to decide if adevice’s current state (e.g. connectivity to other devices)allows it to deploy a particular virtual sensor. Moreover,the dynamics and rapid changes in the whole network,G, including device availability, mobility, connectivity,and resource utilization, make it too difficult to maintaindevices’ states in a centralized manner. To addressthese challenges, we propose a distributed algorithm thatpropagates the graph data structure Υ to devices in Gusing a gossip policy as detailed in Section III-A.

2) Virtualization: After performing the sensing re-source discovery, an agent deploys the virtual sensornetwork, Υ, by means of devices virtualization. Thevirtualization task consists of finding: i) a set A ⊂ S′

of exactly g connected devices according to the virtualtopology chosen by the cloud, and ii) a set MA ⊂{(i, j) ∈ A × V : j ∈ D(i)} of (device,virtual sensor)mapping pairs such that one virtual sensor maps toexactly one device and a device maps to one and onlyone virtual sensor in g. Also, the length h(i, i′) of anysimple path connecting two distinct devices i, i′ ∈ Athat maps a virtual link (j, j′) ∈ E must be less thanor equal to h. We refer to a {A,MA} pair that satisfiesthe previous conditions as a feasible virtualization of therequested virtual sensor network Υ. Note that for anypossible set A, there can exist multiple mappings,MA,and each can form a feasible virtulization. The designobjective of a virtualization algorithm is then to find the’optimal’ feasible virtualization, {A,MA}∗, that usesthe least possible physical network resources.

We now define and introduce what an ’optimal’feasible virtualization means. We consider that thenumber of virtual sensors and the number of virtuallinks of a given Υ = (V,E) determine the cloudcost of providing the sensing service, which is givenby Cost(Υ) = α|V | + β|E|. The scalar α denote anincentive paid to each device that maps a virtual sensor,and the scalar β denote an incentive divided and paidto each device on a physical path that maps to a virtuallink. An incentive could be monetary or could be in anyother form (e.g., credit, service, etc.). On the other hand,the total devices’ benefit from mapping the requestedvirtual network, Υ, can be expressed as

Benefit =∑

(i,j)∈MA

αC(i)−R(j)

C(i)+∑

(i,i′)∈P

βh− h(i, i′)

h,

(4)where h(i, i′) is again the length (in number of hops)of the path connecting the device pair, (i, i′), mappingthe virtual link between j and j′, and P = {(i, i′) ∈A×A : (i, j), (i′, j′) ∈MA , (j, j

′) ∈ E}.The total devices’ benefit in (4) implies that the lesser

the used physical resources, the greater the benefit to thedevices. The first term of (4) captures the benefit lossof the i-th device from allocating resources to map avirtual sensor j. As the minimum demanded sensingcapability R(j) becomes negligible (w.r.t. the sensingcapability C(i)), i gets higher benefit as it investslesser fraction of its resources (e.g. energy, CPU, ormemory) to map j for the same incentive α. Similarly,the second term captures the benefit loss of devices i andi′, which map the virtual sensors j and j′ respectively.Such benefit loss results from mapping the virtual linkbetween j and j′ with more intermediate devices, as thesame incentive β for the virtual link (j, j′) is dividedon a greater number of devices (i.e. number of hopsh(i, i′)) compared to h. Theoretically, h can take avalue up to the diameter of G. However, this shall notwork in practice as the diameter of G is assumed to

5

be much greater than a user desired diameter Υ. Thevirtualization algorithm that we propose in Section III-Bconsists of finding an ’optimal’ feasible virtualizationthat maximizes the total benefit given in (4). We referto the optimal solution as {A,MA}∗. Clearly, finding{A,MA}∗ is hard due to the factorial size of thesolution space in n and due to the challenges, discussedearlier, associated with the sensing resources discoverytask.

3) Distributed consensus estimation: The virtual sen-sor network determined during the virtualization phaseexecutes the distributed sensing algorithms. The simplesolution to the LS problem, proposed in (1), requiresthat each virtual sensor exchanges its measurementvector and its sensing model with the cloud agent,which can create significant communication overhead.Therefore, we instead propose a decentralized approachthat relies on the virtual sensor network to providean estimation of the parameter vector θ. We rely onthe recent results presented in [13] to develop ourdistributed estimation algorithm, which reduces commu-nication overhead significantly when compared to theconventional Alternating Direction Method of Multipli-ers (ADMM) approach [14] in addition to not requiringsynchronization among sensors. The proposed algorithmis presented in Section IV.

III. PROPOSED SOLUTIONS FOR SENSINGRESOURCE DISCOVERY AND VIRTUALIZATION

A. Sensing Resource DiscoveryAlthough devices are directly accessible by cloud

agents, contacting the devices at fine-grained time slotsto discover their current sensing capabilities createssignificant communication and computation inefficiencyfor large n. Such a centralized approach requires ex-changing O(n) messages, in each time slot, whileconstructing the virtual domains, given by (2), requiresO(n) time. Moreover, activating devices periodically toupdate their current sensing capabilities to their cloudagents is power inefficient, especially if the devices arebattery operated.

We propose to perform sensing resource discoverythrough a gossip based algorithm that requires a timecomplexity of O(r−1 log n) and an average Θ(1) mes-sages per device. In this algorithm, an agent propagatesinformation about a received sensing task request, Υ,using the following ’gossip policy’.

The agent sends Υ to a randomly chosen devicestarting at t = 0. Then, any device that receives Υcontinues sending Υ to a random device of its directneighbors until one neighbor acknowledges that it hasalready double received the same version of Υ in aprevious step; by then the device stops sending Υ. Theagent does not need to send Υ to each device as theutilized gossip policy allows devices to disseminate Υautonomously, and the network of devices is guaranteedto be connected with high probability if each device isconnected to k neighbors and k ≥ 0.5139 log n [15].

while True dowait ∆ts←− random neighborif Υ is ∅ then

solicit Υ from selse

send Υ to send ifreceive Υ′ from sif Υ′ = Υ then

stop sending Υelse

Υ′ is newer thanΥΥ←− Υ′

evaluate D(i)end if

end whilei) active thread at de-vice i

while True doreceive Υ′ orsolicit request fromsif Υ is not ∅ then

send Υ to send ifif Υ′ is new then

Υ←− Υ′

evaluate D(i)end if

end whileii) passive thread at i

Fig. 3: proposed sensing resources discovery gossipbased threads at device i.

Since G is a connected network, this simple gossippolicy guarantees that Υ reaches all the devices inO(r−1 log n) time (see [16] for time complexity analy-sis of general gossip protocols in Euclidean geometricrandom graphs). Hence a device i can construct D(i)according to (2) once it receives Υ and the agent candiscover sensing resources of devices that are capableof satisfying the requirements of Υ as fast as possiblewith minimal communication overhead.

The agent and all its connected devices implement theactive and passive threads shown in Fig. 3. At the k-thtime slot, let the device i be active and contact a randomneighbor device i′ (i.e., (i, i′) ∈ L) with probabilityTi,i′ > 0. Ti,i denote the probability that i does notcontact any other device. Let the n×n matrix T = [Ti,i′ ]be a doubly stochastic transition matrix of non-negativeentries [17]. A natural choice of Ti,i′ is

Ti,i′ =

1

di + 1, if i = i′ or (i, i′) ∈ L,

0, otherwise,(5)

where di = |{i′ ∈ S : (i, i′) ∈ L}| is the degree of i.When i contacts i′, they exchange information as

follows (see Fig. 3 ). i pushes Υ to i′ only if i′ does nothave Υ, or pulls Υ from i′ only if i does not have Υ. Ifi contacts i′ and both devices have received Υ before,i stops contacting any other device. If G is connected,the proposed protocol guarantees that Υ is delivered toall IoT devices.

The actual running time of the proposed algorithmdepends on the choice of the transition matrix T andthe communication range of the used device-to-device

6

communication technology. The running time is relatedto the mixing time of any random walk on G [17],which suggests that there is an optimal value of Ti,i′to minimize the mixing time and it is related to thesecond eigenvalue of the transition matrix. Moreover,in case of small r, the proposed algorithm is generallyslow. Practically, this algorithm is suitable for device-to-device communication technologies that support com-munication ranges of few hundreds of meters, as in WiFidirect and LTE D2D and when G is sufficiently dense.

B. VirtualizationWe present our proposed Randomized and Asyn-

chronous Distributed Virtualization (RADV) algorithm.RADV consists of three phases: (I) pruning of virtualdomains D(i) for all i ∈ S, (II) construction of benefitmatrices in a distributed manner, and (III) solvingassignment problems at virtual sensors. RADV results inmultiple solutions each evaluated by a different sensor(device), and the cloud agent selects the solution withthe maximum benefit.

Phase I—Virtual Domain Pruning: During thisphase, we ensure that all virtualized sensors maintainthe topology E by allowing a senor to receive thevirtual domains of other sensors and delete a virtualsensor j from its domain if there exists a virtual link(j, j′) such that j′ is not included in any other receiveddomains. Let Ds ⊂ {D(i) : i ∈ S} denote thevirtual domains set that sensor s has at time k. InitiallyDs = {D(s)} and h(i, s) = 0 for all i ∈ S2. Usingthe same transition matrix, T , defined in Eq. (5), scontacts only one of its neighbors s′ at time k. Then,for allD(i) ∈ Ds : i 6= s′, s pushes D(i) to s′ only ifs′ did not receive D(i) before and h(i, s) < h. Also,for allD(i) ∈ Ds′ : i 6= s, s pulls D(i) from s′ onlyif s did not receive D(i) before and h(i, s′) < h. If noinformation is exchanged between s and s′ at time k, sstops contacting any of its neighbors. However, s mayrestart contacting its neighbors again if it updated Ds

after time k + 1.When s constructs its Ds, it starts by pruning D(s).

The pruning is performed by deleting a virtual sensorj ∈ D(s) (i.e., D(s) ← D(s) \ {j}) if none of thevirtual sensors that are connected to j, {j′ ∈ V :(j, j′) ∈ E}, is not included in any received D(i), i.e.j /∈ D(i) : D(i) ∈ Ds. This pruning rule ensures thatthe virtualized sensors maintain the required topologyE and the constructed benefit matrices shall result in afeasible virtualization.

Phase II—Construction of Benefit Matrices: Asmentioned earlier, finding a feasible virtualization,{A,MA}∗, that maximizes the total benefit given inEq. (4) is a hard problem due to the large size ofthe solution space. Therefore, this phase proposes anefficient way of solving this virtualization problem.Specifically, we propose a method that solves this

2Knowledge about other sensors existence is not needed, and h istypically evaluated dynamically.

problem in a distributed manner and without requiringany synchronization among sensors, as described next.

During this phase, each sensor s locally constructs itsown set, A(s), of g sensors that s chooses as virtualizedsensors to assign to virtual sensors in V . Each sensors also maintains g row vectors, B(s)

i ∈ R1×g and i ∈A(s), that we define as the benefit vector of sensor iseen by s, where the j-th element, B(s)

i,j , denotes thebenefit of assigning participatory sensor i ∈ A(s) to thevirtual senor j ∈ V as seen by s, and is given by

B(s)i,j =

αC(i)−R(j)

C(i)+ β

h− h(j, s)

hif j ∈ D(i),

0 otherwise.

Our objective is then to construct, for each s ∈ S, thebenefit matrix B(s) = [B

(s)

i∈A(s) ] as fast as possible, andfind a feasible virtualization, {A,MA}, that maximizesthe total benefit, ∑

(i,j)∈MA

B(s)i,j ,

among all s ∈ S without knowing the G structure.Moreover, the path length between a sensor s and anyother sensor i that s includes in its benefit matrix mustnot exceed h. Finally, a sensor s shall include only thebenefit vectors of the g sensors with the largest possiblebenefit.

Each sensor s initially sets A(s) = A(s) ∪ {s} ifD(s) /∈ ∅, sets h(i, s) = 0 for all i ∈ S, and sets

B(s)s,j =

αC(s)−R(j)

C(s)+ β, j ∈ D(s),

0, otherwise.

Also, s maintains a scalar, bmins , defined as the minimum

total benefit it has received from any other sensor andwritten as

bmins = min

i

∑j∈V

B(s)i,j .

s also maintains the corresponding sensor,

imins = argmin

i

∑j∈V

B(s)i,j .

Initially, bmins = 0 and remains so until |A(s)| = g.

Using the same transition matrix, T , defined inEq. (5), s contacts its neighbor s′ only once at eachtime k. Then, for all i ∈ A(s) : i 6= s′, s pushes thebenefit vector B(s)

i to s′ only if h(i, s) < h and∑j∈V

(B

(s)i,j −

β

h

)> bmin

s′ .

Also, for all i ∈ A(s′) : i 6= s, s pulls the benefit vectorB

(s′)i from s′ only if h(i, s′) < h and∑

j∈V

(B

(s′)i,j −

β

h

)> bmin

s .

7

If no information is exchanged between s and s′ attime k, s stops contacting its neighbors at time k + 1.However, s may restart contacting its neighbors againif B(s) is updated after time k + 1.

When s receives B(s′)i , s updates B(s)

i,j as

B(s)i,j =

B(s)i,j −

β

hif j ∈ D(i),

0 otherwise.

If i /∈ A(s), then we have two scenarios. In thefirst scenario, s still has not received g benefit vectors,so bmin

s = 0 and |A(s)| < g, then s updates its setof candidate sensors as A(s) = A(s) ∪ {i}. In theother scenario in which |A(s)| = g, s replaces thesensor corresponding to the minimum total benefit,imins , with i so that A(s) = A(s) \ {imin

s } ∪ {i}. Onthe other hand, if i ∈ A(s), then s updates B

(s)i,j if∑

j∈VB

(s′)i,j >

∑j∈V

B(s)i,j . Finally, s updates bmin

s , imins , and

h(i, s) as h(i, s) = h(i, s′) + 1.Finding a feasible virtualization that maximizes the

benefit B(s) = [ B(s)

i∈A(s) ] instead of the benefit givenin Eq. (4) makes the problem easier because everysensor has a different value for the benefit Bi,j thatdepends only on the length of the physical path betweeni and s instead of the path lengths of all possiblecombinations of sensor pairs (i, i′) that can virtualizea virtual link. Intuitively, this relaxation still leads to anoptimal or near optimal virtualization, because if G isvery large and connected, the number of sensors thatare directly connected by a single physical link (clique)grows logarithmically in n and hence this number islarger than g almost surely as g � n. In such a case, it issufficient to ensure that the length of the paths betweeni and s and between i′ and s are the shortest possibleones to ensure that the length of the path between iand i′ is also the shortest, as in this case, s, i, andi′ reside in the same clique with high probability. Weevaluate the effectiveness of this relaxation in SectionV and show that our virtualization algorithm performswell even when the condition g � n does not hold.

Phase III—Solving Local Assignment Problem: Af-ter reception of the g benefit vectors, s proceeds to thisphase of the algorithm only if it stops communicatingand |A(s)| = g. Each sensor s ∈ S with |A(s)| = gsolves locally the following assignment problem:

maximize∑

i∈A(s)

∑j∈D(i)

B(s)i,jmij

subject to∑

j∈D(i)

mij = 1, i ∈ A(s),∑{i:j∈D(i)}

mij = 1, j ∈ V,

mij ∈ {0, 1},

(6)

where mij are binary optimization variables indicatingwhether the participatory sensor i is assigned to thevirtual sensor j. The problem formulated in (6) isequivalent to the perfect maximum weight matching

problem in a bipartite graph, and hence, we proposeto use the classical Hungarian method to solve it (theworst case time complexity is O(g3) [18], [19]).

We can also tolerate an error ε > 0 of the resultingtotal benefit and relax the restriction of finding a perfectmatching for large g. This relaxation is reasonable whenthere are enough sensors involved in solving these localoptimization problems, as in this case we can pickthe best solution and discard those without a perfectmatching. In such a scenario, we can also use a lineartime (1− ε)-approximation algorithm to solve (6) [20].In this paper, we use the Hungarian method to solveour formulated optimization problems. Details of thealgorithm are omitted due to space limitation; readersare referred to [18], [19], [20] for detailed information.

Each sensor solves locally the optimization problemgiven in (6) and sends its obtained solution to the cloudagent. This is done asynchronously. The cloud agentthen selects the solution that leads to the maximum totalbenefit, and keeps all other solutions for later use in theevent that the network dynamics invalidate the selectedsolution before the virtual sensing task completes.

Complexity and message overhead. We assume thatthe topology of G, devices mobility, and sensing capa-bility are not changed during the execution of the vir-tualization phase. The time required to spread Υ acrossthe network is O(r−1 log n) [16]. It takes O(g) worstcase time to evaluate D(i) locally at sensor i. Also,the time required to spread information in the pruningand benefit construction phases is O(r−1n log n). Thepruning of the virtual domain D(i) requires node i toexamine g received virtual domains, each having atmost g entries. The worst case local running time ofpruning is then O(g2). Finally, the local running timeof the Hungarian method is O(g3). Hence, the overallcomplexity is O(max{r−1n log n, g3}).

The average number of messages communicated persensor during the sensor search phase is Θ(1) and eachmessage is O(g) in size. During pruning of virtualdomains, since every sensor exchanges a maximum ofn domains each of size that is also O(g), the averagenumber of messages communicated per sensor is O(n).However, because we restrict that messages to be com-municated up to h hops for only a group of sensorsthat support the requirements of Υ, the average numberof messages per sensor is typically small. Fig. 4 showsthe total time and the average number of messages persensor required during both the domain pruning and thebenefit construction phases. The total time growth islinearithmic in n when Υ is sent to exactly one sensorand when G is connected. This time can, in practice, bedecreased significantly if Υ is initially sent to multiplesensors. Additionally, the average number of messagesper sensor is shown to scale linearly with n, and istypically a very small fraction of n.

8

16

17

18

19

20

21

22

23

24

400 800 1200 1600 2000 0.6

0.8

1

1.2

1.4

1.6

1.8

2

2.2

2.4Ti

me

Ave

rage

No.

of M

essa

ges

n

MessagesTime

Fig. 4: Time (in number of iterations) and message over-head (in average number of communicated messages)resulting from constructing the benefit matrices, g = 10.

IV. PROPOSED SOLUTIONS FOR DISTRIBUTEDESTIMATION

After completing the sensor virtualization task, usingRADV, the virtual sensors run an in-network param-eter estimation algorithm to compute θ distributedly.In this section, we present our proposed Randomizedand Asynchronous Distributed Estimation (RADE) al-gorithm. We first follow the standard ADMM approachto derive primal, dual and Lagrangian variable updateequations, then we describe the proposed RADE algo-rithm. For clarity of notation, in what follows, we referto the set of g selected devices, determined by RADV,simply as A.

The centralized estimation approach given in (1)is first decomposed into g local estimates of θ(one θi for each i ∈ A) while constraining thelocal estimates with the coupling constraints θi =θj for all (i, j) ∈ P . This results in the following op-timization problem:

minimize 12

∑i∈A‖xi −Hiθi‖2

subject to θi − θj = 0 for all (i, j) ∈ P,(7)

where {θi, i ∈ A} are the optimization variables.By introducing an auxiliary variable, z, we decouple

the constraints in (7), so that θi − z = 0 for all i ∈ A[21]. However, this requires that z be shared among allthe g virtual sensors. Instead, we introduce g auxiliaryvariables, zi, and equivalently write the optimizationproblem as

minimize 12

∑i∈A‖xi −Hiθi‖2

subject to θj − zi = 0 for all (i, j) ∈ P.(8)

Let λ = {λi,j ∈ RN×1 : (i, j) ∈ P} and ρ ={ρi,j ∈ R : (i, j) ∈ P} denote respectively theset of Lagrangian multipliers and the set of penalty

parameters. The augmented Lagrangian is given by

Lρ(θ, z, λ) =∑i∈A

[12‖xi −Hiθi‖2

−∑

j∈A:(i,j)∈PλTi,j(θi − zj)

+∑

j∈A:(i,j)∈P

ρi,j2 ‖θi − zj‖

2

].

(9)By setting the gradient w.r.t θi of Eq. (9) to zero andsolving for θi, we get

θi =

(HTi Hi +

∑j∈A:(i,j)∈P

ρi,jI

)−1

.

(HTi xi +

∑j∈A:(i,j)∈P

(λi,j + ρi,jzj)

).

Similarly, we solve for zi by setting the gradient w.r.t tozi to zero and rearranging the indices of the Lagrangianmultipliers and the penalty parameters. It follows that

zi =1

g

∑j∈A:(i,j)∈P

(θj −

1

ρj,iλj,i

).

The former analysis leads to the conventionalADMM-based distributed consensus estimation algo-rithm given by

θ(k+1)i =

(HTi Hi +

∑j∈A:(i,j)∈P

ρi,jI

)−1

.

(HTi xi +

∑j∈A:(i,j)∈P

(λ

(k)i,j + ρi,jz

(k)j

)),

z(k+1)i = 1

g

∑j∈A:(i,j)∈P

(θ

(k)j − 1

ρj,iλ

(k)j,i

),

λ(k+1)j,i = λ

(k)j,i − ρj,i

(θ

(k+1)j − z(k+1)

i

),

(10)where the superscript k denotes the value of the vari-able at the k-th iteration. This conventional ADMMalgorithm, given in (10), requires synchronization andvariable update among the sensors [22], [23]. Moreover,at each iteration k, each sensor i must send z(k)

i and θ(k)i

to all other sensors it is connected to, so as to evaluatetheir k + 1 primal, dual, and Lagrangian multipliers.When M is small, this algorithm incurs communicationoverhead that can be shown to be worse than the com-munication overhead incurred by centralized estimationmethods. However, when M is large, the conventionalADMM algorithm incurs lesser communication over-head than what centralized estimation methods incur,but it still remains practically unattractive due to otherweaknesses, detailed later in Section V.

Given the absolute and relative tolerances, εabs andεrel, specified by the SLAs, we define the primal anddual tolerances, controlling the convergence of the al-gorithm at iteration k, as

εprii (k) =√gεabs + εrel max(‖θ(k)

i ‖, ‖ − z(k)i ‖),

9

and

εduali (k) =√gεabs + εrel

∑j∈A‖ρj,iλj,i‖.

The tolerances, εprii and εduali , define the stoppingcriteria of sensor i; i.e., sensor i stops updating θi andzi when

‖θ(k+1)i − z(k+1)

i ‖ < εprii (k), (11)

and‖z(k+1)i − z(k)

i ‖ < εduali (k). (12)

The stopping criteria of RADE are different from thoseof the conventional ADMM. Unlike the conventionalADMM where all sensors shall stop computations allat the same time using a common stopping criteriaand common primal and dual tolerances, the stoppingcriteria (Eq. (11) and Eq. (12)) of RADE allow a sensori to stop its computations asynchronously and indepen-dently from other sensors. However, these criteria arenot enough to ensure asynchronous implementation, assynchronization is still required for dual and primal vari-able updates at iteration k+1 due to their dependencieson k.

To ensure full asynchronous implementation, we usethe doubly stochastic transition matrix, T ∈ Rg×g ,where Ti,j is the probability that a sensor i contactsanother sensor j at any iteration, for deciding thecommunications among sensors. We can have

Ti,j =

1

d′i + 1if i = j or (i, j) ∈ P,

0 otherwise,

where d′i = |{j ∈ A : (i, j) ∈ P}| is the degree of thevirtual sensor, in Υ, that i virtualizes. At iteration k+1,sensor i may need to contact only one sensor j, unlessboth of i’s stopping criteria, Eq. (11) and Eq. (12), arealready satisfied. Whereas sensor j can be contactedby more than one sensor if j is not contacting anyother sensor, even when both of j’s stopping criteriaare satisfied.

Upon contacting j, sensor i pushes θ(k)i to j only if

i’s primal stopping condition is not satisfied and pushesz

(k)i to j only if i’s dual stopping condition is not

satisfied. Also, i pulls θ(k)j from j only if j’s primal

stopping condition is not satisfied and pulls z(k)j only

if j’s dual stopping condition is not satisfied. Finally,both i and j update their k+1 variables using the mostrecent values they received from other sensors.Mean square error and convergence. The asyn-chronousness and randomization design of RADE donot impact the Mean Square Error (MSE) achieved byRADE when compared to ADMM. This is explainedas follows. In both ADMM and RADE, the numberof necessary dual and primal variables updates that areneeded until convergence remains unchanged, so thatconvergence to the same estimate is guaranteed in bothalgorithms. Fig. 5 shows the MSE achievable under both

0.01

0.1

1

-10 -8 -6 -4 -2 0 2 4

MS

E

Noise Power (dB)

LSADMM-CompleteRADE-Complete

RADE-CyclicRADE-Star

Fig. 5: MSE of RADE compared to those achievedunder ADMM and LS at different noise power and fordifferent virtual sensor network topologies, g = 10.

RADE and ADMM when compared to LS under each ofthe three studied sensor network topologies: complete,star, and cycle. These results show the optimality ofRADE that we intuitively discussed. All approacheshave the same accuracy. But of course each of them doesso at a different performance cost, as will be discussedlater.

On the other hand, RADE exhibits a linear conver-gence rate (O(1/k)), similar to what the conventionalADMM does. Fig. 6 shows the number of time stepsrequired for both RADE and ADMM to converge un-der different relative tolerance parameters, εrel. RADEconvergence tends to be more restricted by the random-ization nature of the algorithm for smaller values of εrel,which can be seen by the increasing number of steps asg increases if εrel = 10−2. ADMM generally requiresa lesser number of steps to converge by relaxing theconsensus constraint (through reducing εrel). However,as will be seen in the numerical results section later, thisincrease in the number of convergence steps is accept-able when considering the amount of communicationoverhead that the algorithm saves.

V. NUMERICAL RESULTS

In this section, we evaluate the performance of theproposed RADV and RADE algorithms through simu-lations. In our simulations, G and Υ, are generated usingthe parameters summarized in Table I. We evaluate theperformance for a complete, cyclic, or star virtual sensornetwork topology, with a randomly chosen central lo-cation, c. We consider receiving and servicing only onevirtual sensing task request at a time. The G topologyand connectivity can change rapidly. For a single Υ, weassume that the network change rate is slow enough forthe completion of the sensor search and virtualizationphases. The absolute and relative tolerances, εabs andεrel, are set to 10−4 unless specified otherwise.

Fig. 7 shows the rejection rate encountered with dif-ferent Υ topologies and n values. As we only consider

10

0

20

40

60

80

100

120

140

160

180

3 6 9 12 15 18 21 24

k

g

ADMM,10-4

RADE, 10-4

ADMM, 10-2

RADE, 10-2

Fig. 6: Number of time steps (k) needed until con-vergence of RADE when compared to ADMM forcomplete topology under different relative tolerancevalues εrel.

TABLE I: Simulation Parameters

Parameter g r C(i) R(j) δ h

Value 10 0.1 ∼ U(50, 100) ∼ U(25, 50) 0.2 20

one single request at a time, the results shown in thisfigure reflect mainly the impact of the virtual sensornetwork topology, the number of sensors n, and thesimulations parameters given in Table I on the rejectionrate. The denser the network of IoT devices is, the lowerthe rejection rate, implying that the cloud is capable ofgranting higher number of requests.

One way of assessing the effectiveness of the vir-tualization algorithm is by measuring the differencebetween the total virtualization benefit given in (4)and the cost associated with the sensor virtualizationintroduced in Section III-B. For a given number ofvirtual sensors, the cost is mainly determined by thechoice of the topology (star topology has the lowestcost and complete topology has the highest one). For

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 400 800 1200 1600 2000

Rej

ectio

n R

ate

n

RADV-CompleteRADV-Cyclic

RADV-Star

Fig. 7: Rejection rate encountered at different n.

5

6

7

8

9

10

11

12

0 400 800 1200 1600 2000

Tota

l Ben

efit

- Cos

t

n

CompleteBound - Complete

CyclicBound - Cyclic

StarBound - Star

Fig. 8: Virtualization cost of RADV when compared tothe upper bound under different topologies.

1

10

100

1000

3 6 9 12 15 18 21 24

k

g

ADMM-CompleteRADE-Complete

ADMM-CyclicRADE-CyclicADMM-StarRADE-Star

Fig. 9: Number of time steps until convergence ofRADE when compared to ADMM under differenttopologies.

a given topology, the total benefit is maximized wheneach virtual sensor is assigned to the IoT devices withthe maximum capacity and each virtual link is mappedto exactly one physical link. We refer to this maximizedbenefit as the upper bound.

In Fig. 8, we evaluate the virtualization effective-ness achieved by RADV under different virtual topolo-gies. As the network gets denser, RADV achieves aTotal Benefit − Cost that is very close to the upperbound. Since the lowest possible virtualization cost iswith star or cyclic topologies, it is desired by the cloudto arrange each virtual sensing task in a star or acyclic topology. This observation holds true for a moregeneral topologies. On the other hand, convergence andcommunication overhead of the distributed estimation isalso impacted by the cloud agent’s choice of the virtualtopology. This creates a design trade-off, as we will seein the next two paragraphs.

Fig. 9 shows the impact of the virtual topologychoice on the convergence performance of RADE whencompared to ADMM. If g is small (three to eight), the

11

100

1000

10000

100000

1e+06

3 6 9 12 15 18 21 24

Ave

rage

No.

of M

essa

ges

g

LSADMM-CompleteRADE-Complete

ADMM-CyclicRADE-CyclicADMM-StarRADE-Star

Fig. 10: Communication overhead when comparingRADE to ADMM and LS under different g values.

impact of the virtual topology on convergence of RADEand ADMM is minimal. This is because the degreeof parallelism (number of virtual sensors active at thesame time) is restricted by the small number of virtualsensors g. In such a scenario, it is convenient for thecloud agent to always arrange the virtual sensors in astar topology. However, as g increases, the impact ofchoice of the virtual topology becomes significant as thedegree of parallelism is higher in a complete topology,enabling RADE to converge much faster as g gets larger.This convergence becomes slower with star and cyclictopologies. This is because in star and cyclic topologies,only few sensors are active at a time, making RADE andADMM converge in a number of steps comparable tothat of the ADMM’s sequential implementation. In thislater scenario, the cloud agent shall arrange the virtualsensors as a complete topology unless the SLA permitsslower convergence.

Moreover, RADE converges in a higher number ofsteps when compared to the conventional ADMM. Thisis because in ADMM, all sensors are active at each time,and a sensor exchanges its updated variables with all ofits neighbors, whereas in RADE, only disjoint sensorpairs are active at a time and variables are updatedonly between pairs of sensors. Nevertheless, we arguethat this loss in speed of convergence for RADE ismarginal when compared to the significant savings incommunication overhead.

Fig. 10 shows the total number of O(N) sizedmessages exchanged during estimation when comparingRADE, ADMM, and LS for M = 100. The number ofmessages exchanged by RADE is at least an order ofmagnitude less than the number of messages generatedunder ADMM. Also the communication overhead ofRADE is less than the centralized LS especially as Mbecomes large. This savings in communication overheadis attributed to the asynchronous design of RADE inwhich messages among sensors are only exchanged ifnew values of a primal or dual variables are changedaway from their specified tolerances.

VI. RELATED WORK

A. Network Virtualization

Network virtualization techniques proposed in thepast decade consist mainly of virtual network embed-ding algorithms, which instantiate virtual networks onsubstrate infrastructures [24], [25]. Most of these virtualnetwork embedding algorithms are centralized (e.g. [5],[26]) due to the ease of deployment of centralizedapproaches in cloud platforms where the cloud providerdesires to have full control on the physical networkresources. Distributed network virtualization techniquesare suited for Cloud of Things, given the size of thenetwork, and are also proposed for applications inresource allocation in distributed clouds, wireless sensornetwork virtualization, and cloud network as a service[27], [28].

Beck et al. propose a hierarchical partitioning of anysubstrate network [29] and solve the network virtual-ization (virtual network embedding) problem on thescale of smaller partitions by delegating the problemto delegate nodes. In our context, their algorithm canprogress in four steps: (1) partitioning the networkof IoT devices, (2) assigning delegation nodes amongthe IoT devices that actually perform the network vir-tualization, (3) setting distributed lock trees to avoidinconsistent solutions among different delegation nodes,and (4) embedding the virtual sensor network withinthe scope of the delegation nodes. Several assump-tions in this work prevent this method applicability inCloud of Things. The authors require that a centralizednode manage the IoT network topology to perform thepartitioning. The centralized node partitions the IoTdevices in groups that are highly interconnected. Thisrequirement is very hard to achieve, if not impossible,in an Internet-scale network of IoT devices, not onlybecause of the size of the network but also due to itshighly dynamic nature that prevents tracking the statesof the devices and their connectivity. Moreover, to applythe same method in cloud of things, the delegation nodesof Beck’s method needs to learn a significant amountof information about the IoT devices in their neighbor-hood, which creates significant practical problems suchas: timely information retrieval, privacy concern, andcomputation power.

Esposito and Ibrahim propose to model the networkvirtualization as a network utility maximization prob-lem, where the utility is a general function that ismeasured on each hosting node (i.e. IoT device) [30].They solve the problem distributivity using primal dualdecomposition. To employ their algorithm in our Cloudof Things context, the non-convex constraints of thenetwork virtualization problem need be relaxed. Sucha relaxation is known to have a negative impact on theaccuracy of the solution and may lead to false decisions[5]. Moreover, this method requires the definition of asingle utility for the IoT device that shall not reflect theactual embedding cost due to network conditions, andonly captures network conditions seen locally by the

12

IoT device. Finally, solving the problem using primal-dual decomposition requires cooperation between allIoT devices in the network, which prevents this methodfrom scaling on large-sized networks without adequatenetwork partitioning as approached by Beck et al. in[29].

In [31], [32], the authors propose a distributed virtualnode mapping algorithm using consensus-auction. Toapply it in our context, a utility function needs tobe defined for each IoT device that is based on thedevices local capacity and link bandwidth (known onlyto the device). The IoT devices then start bidding forvirtual sensors to maximize its utility. Although the nodemapping is very close to optimal, the overall result ofthe algorithm is not necessarily optimal since the linkmapping uses only first hop link information of thesubstrate nodes. The authors also assume that paths arecomputed using 3−shortest paths, which overlook otherpath diversity that can be found in a large substratenetwork.

The earlier methods are designed for federated datacenters, and cannot be applied to the Cloud of Thingscontext. In our approach, each device with a non-emptyvirtual mapping domain first solves the problem within aradius of h of the subgraph centered at the IoT device.Then, the agent selects the best solution by reducingdifferent solutions found by the devices to a single bestsolution. Unlike the work of Beck et al. in [29], par-titioning and delegation are an implicit process of thealgorithm performed through the gossip policy used dur-ing the sensing resource discovery phase where only theIoT devices with a non-empty virtual mapping domainstart to execute the next steps of the algorithm. Thisapproach does not require updated knowledge about allthe IoT devices and the topology of the network. TheIoT devices perform virtualization by exchanging mini-mal information to construct their local benefit matricesand solving an assignment problem. Unlike [30], [31],[32], a benefit matrix constructed by a device capturesall network information within a radius of h around thatdevice, thereby improving the obtained solution close-ness to optimal without overlooking important networkcharacteristics. And unlike [30], [29], we do not restrictthe link mapping to use k−shortest paths, thereby allow-ing the use of diverse paths in the network constructedduring the randomized gossip policy used during thebenefit matrices construction.

B. Distributed Estimation

Distributed parameter estimation approaches havebeen proposed in [14], [33], [34]. Estimation can fore.g. be carried out by first computing a local estimateat each virtual sensor and then perform a distributedweighted average of the local estimates [33]. Thisapproach results in an ML estimate, but does notlimit/bound the variation between mean square errors oflocal estimates. More recently, Paul et al. [14] propose a

distributed estimation algorithm based on ADMM. Al-though this approach results in an optimal mean squareerror when compared to LS, it exhibits a significantin-network communication overhead that requires evenmore messages to be exchanged among sensors thanthat exchanged in the centralized LS. One approachalso proposed in [14] to overcome this problem is toapproximate the computation of primal and dual vari-ables at each step of the algorithm by using predictionsand earlier versions of these variables instead of sharingthem at each iteration which marginally reduces thecommunication overhead. In addition to the increasedcommunication overhead, conventional ADMM requiressynchronous operation of the sensors. This is verychallenging from a practical viewpoint, and does notscale well especially when applied in the Internet ofThings (IoT) context. It has been shown recently that anasynchronous implementation of ADMM has O(1/k)convergence [13]. Our proposed estimation algorithm isboth asynchronous and distributed, and reduces commu-nication overhead significantly when compared to theconventional ADMM approach [14].

VII. CONCLUSION AND DISCUSSION

We have shown the potential of Cloud of Thingsto scale cloud computing vertically by exploiting sens-ing resources of IoT devices to provide Sensing as aService. We have proposed a global architecture thatscales Cloud of Things horizontally by employing edgecomputing platforms in a new role as cloud agentsthat discover and virtualize sensing resources of IoTdevices. We have described cloud agents technicalchallenges and design objectives for sensing resourcesdiscovery and virtualization that can dispatch offeringvirtual sensor networks deployed on IoT devices tocloud users with in-network processing capabilities. Wegave a taxonomy of the potential sensing tasks, theirapplications, and there challenges. We have proposedour sensing resource discovery solution based on agossip policy to discover sensing resrouces as fast aspossible and RADV: our virtualization solution. Wehave shown through analysis and simulations the po-tential of RADV to achieve reduced communicationoverhead, low complexity, and closeness to optimalsuch that RADV employs minimal physical resourcesin devices virtualization with maximal benefit. We alsoproposed RADE for distributed consensus estimationas we believe it is one major sensing task in Sensingas a Service. Using simulation, we show that RADEreduces the communication overhead significantly with-out compromising the estimation error when comparedto the traditional ADMM algorithm. We also showthat the convergence time of our proposed algorithmsmaintain linear convergence behavior, as in the case ofconventional ADMM.

VIII. ACKNOWLEDGMENT

This work was made possible by NPRP grant #NPRP 5- 319-2-121 from the Qatar National Research

13

Fund (a member of Qatar Foundation). The statementsmade herein are solely the responsibility of the authors.The authors would like to thank Thinh Nguyen andMohamed Grissa for their early comments on this work.

REFERENCES

[1] N. D. Lane, E. Miluzzo, H. Lu, D. Peebles, T. Choudhury,and A. T. Campbell, “A survey of mobile phone sensing,”Communications Magazine, IEEE, vol. 48, no. 9, pp. 140–150,2010.

[2] C. F. Mass and L. E. Madaus, “Surface pressure observationsfrom smartphones: A potential revolution for high-resolutionweather prediction?” Bulletin of the American MeteorologicalSociety, vol. 95, no. 9, pp. 1343–1349, 2014.

[3] S. Abdelwahab, B. Hamdaoui, M. Guizani, and A. Rayes,“Enabling smart cloud services through remote sensing: Aninternet of everything enabler,” Internet of Things Journal,IEEE, vol. 1, no. 3, June 2014.

[4] C. Perera, A. Zaslavsky, P. Christen, and D. Georgakopoulos,“Sensing as a service model for smart cities supported by inter-net of things,” Transactions on Emerging TelecommunicationsTechnologies, vol. 25, no. 1, pp. 81–93, 2014.

[5] S. Abdelwahab, B. Hamdaoui, and M. Guizani, “Bird-vne:Backtrack-avoidance virtual network embedding in polynomialtime,” in Global Communications Conference (GLOBECOM),2014 IEEE. IEEE, 2014, pp. 4983–4989.

[6] X. Sheng, J. Tang, X. Xiao, and G. Xue, “Sensing as a service:Challenges, solutions and future directions,” Sensors Journal,IEEE, vol. 13, no. 10, pp. 3733–3741, 2013.

[7] A. Rai, K. K. Chintalapudi, V. N. Padmanabhan, and R. Sen,“Zee: zero-effort crowdsourcing for indoor localization,” inProceedings of the 18th annual international conference onMobile computing and networking. ACM, 2012, pp. 293–304.

[8] S. Abdelwahab, B. Hamdaoui, M. Guizani, and T. Znati, “Cloudof things for sensing as a service: Sensing resource discoveryand virtualization,” in 2015 IEEE Global CommunicationsConference (GLOBECOM). IEEE, 2015, pp. 1–7.

[9] M. Satyanarayanan, P. Bahl, R. Caceres, and N. Davies, “Thecase for vm-based cloudlets in mobile computing,” PervasiveComputing, IEEE, vol. 8, no. 4, pp. 14–23, 2009.

[10] E. Koukoumidis, D. Lymberopoulos, K. Strauss, J. Liu, andD. Burger, “Pocket cloudlets,” ACM SIGPLAN Notices, vol. 47,no. 4, 2012.

[11] F. Bonomi, R. Milito, J. Zhu, and S. Addepalli, “Fog computingand its role in the internet of things,” in Proceedings of thefirst edition of the MCC workshop on Mobile cloud computing.ACM, 2012, pp. 13–16.

[12] C. Shi, K. Habak, P. Pandurangan, M. Ammar, M. Naik, andE. Zegura, “Cosmos: computation offloading as a service formobile devices,” in Proceedings of the 15th ACM internationalsymposium on Mobile ad hoc networking and computing.ACM, 2014, pp. 287–296.

[13] E. Wei and A. Ozdaglar, “On the o (1/k) convergence ofasynchronous distributed alternating direction method of mul-tipliers,” in Global Conference on Signal and InformationProcessing (GlobalSIP), 2013 IEEE. IEEE, 2013, pp. 551–554.

[14] H. Paul, J. Fliege, and A. Dekorsy, “In-network-processing:Distributed consensus-based linear estimation,” Communica-tions Letters, IEEE, vol. 17, no. 1, pp. 59–62, 2013.

[15] P. Balister, B. Bollobas, A. Sarkar, and M. Walters, “Highlyconnected random geometric graphs,” Discrete Applied Math-ematics, vol. 157, no. 2, pp. 309–320, 2009.

[16] D. Shah, Gossip algorithms. Now Publishers Inc, 2009.

[17] S. Boyd, A. Ghosh, B. Prabhakar, and D. Shah, “Random-ized gossip algorithms,” Information Theory, IEEE Trans. on,vol. 52, no. 6, 2006.

[18] H. W. Kuhn, “The hungarian method for the assignment prob-lem,” Naval research logistics quarterly, vol. 2, no. 1-2, pp.83–97, 1955.

[19] J. Munkres, “Algorithms for the assignment and transportationproblems,” Journal of the Society for Industrial & AppliedMathematics, vol. 5, no. 1, pp. 32–38, 1957.

[20] R. Duan and S. Pettie, “Linear-time approximation for maxi-mum weight matching,” Journal of the ACM (JACM), vol. 61,no. 1, p. 1, 2014.

[21] D. P. Palomar and M. Chiang, “A tutorial on decompositionmethods for network utility maximization,” Selected Areas inCommunications, IEEE Journal on, vol. 24, no. 8, pp. 1439–1451, 2006.

[22] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Dis-tributed optimization and statistical learning via the alternatingdirection method of multipliers,” Foundations and Trends R© inMachine Learning, vol. 3, no. 1, pp. 1–122, 2011.

[23] G. Xu, H. Paul, D. Wuebben, and A. Dekorsy, “Fast dis-tributed consensus-based estimation (fast-dice) for cooperativenetworks,” in Smart Antennas (WSA), 2014 18th InternationalITG Workshop on. VDE, 2014, pp. 1–8.

[24] A. Fischer, J. F. Botero, M. Till Beck, H. De Meer, andX. Hesselbach, “Virtual network embedding: A survey,” Com-munications Surveys & Tutorials, IEEE, vol. 15, no. 4, pp.1888–1906, 2013.

[25] A. Belbekkouche, M. Hasan, A. Karmouch et al., “Resourcediscovery and allocation in network virtualization,” Communi-cations Surveys & Tutorials, IEEE, vol. 14, no. 4, pp. 1114–1128, 2012.

[26] X. Cheng, S. Su, Z. Zhang, H. Wang, F. Yang, Y. Luo, andJ. Wang, “Virtual network embedding through topology-awarenode ranking,” ACM SIGCOMM Computer CommunicationReview, vol. 41, no. 2, pp. 38–47, 2011.

[27] P. T. Endo, A. V. de Almeida Palhares, N. N. Pereira, G. E.Goncalves, D. Sadok, J. Kelner, B. Melander, and J.-E. Mangs,“Resource allocation for distributed cloud: concepts and re-search challenges,” Network, IEEE, vol. 25, no. 4, pp. 42–46,2011.

[28] H. Alshaer, “An overview of network virtualization and cloudnetwork as a service,” International Journal of Network Man-agement, vol. 25, no. 1, pp. 1–30, 2015.

[29] M. Till Beck, A. Fischer, H. de Meer, J. F. Botero, andX. Hesselbach, “A distributed, parallel, and generic virtualnetwork embedding framework,” in Communications (ICC),2013 IEEE International Conference on. IEEE, 2013, pp.3471–3475.

[30] F. Esposito and I. Matta, “A decomposition-based architecturefor distributed virtual network embedding,” in Proceedings ofthe 2014 ACM SIGCOMM workshop on Distributed cloudcomputing. ACM, 2014, pp. 53–58.

[31] F. Esposito, D. Di Paola, and I. Matta, “On distributed virtualnetwork embedding with guarantees,” Transactions on Net-working, 2014.

[32] F. Esposito and F. Chiti, “Distributed consensus-based auctionsfor wireless virtual network embedding,” in Communications(ICC), 2015 IEEE International Conference on. IEEE, 2015,pp. 472–477.

[33] P. Tarrıo, A. M. Bernardos, and J. R. Casar, “Weighted leastsquares techniques for improved received signal strength basedlocalization,” Sensors, vol. 11, no. 9, pp. 8569–8592, 2011.

[34] F. Garin and L. Schenato, “A survey on distributed estimationand control applications using linear consensus algorithms,” inNetworked Control Systems. Springer, 2010, pp. 75–107.

14