Wireless Distributed Sensing and Computing inSDR based Radio Network - Implementation and

ApplicationsChangchun Zhang, Robert C. Qiu

Cognitive Radio Insitute, Department of Electrical and Computer Engineering,Center for Manufacturing Research, Tennessee Technological University,

Cookeville, Tennessee 38501, USAEmail: czhang42@students.tntech.edu, rqiu@tntech.edu

Abstract—As the important step towards building the largescale cognitive radio network in TTU(Tennessee TechnologicalUniversity), a wireless distributed sensor network has beenembedded into a novel radio cluster, in which the cluster nodesare running over the SDR(Software Defined Radio) platform,e.g., USRP(Universal Software Radio Peripheral). This networkdemonstrates the combination of the wireless distributed sensingapplications with corresponding communications capability. TheSDR based sensing nodes are capable of sensing the environmentin real time manner, while keeping the wireless networking simul-taneously. The centralized network topology with round-robincontrol protocol has been deployed. The network implemented inthis paper enables the real time wireless sensing and big networkdata fusion. The centralized and distributed computing schemesare demonstrated considering wireless distributed sensing appli-cations. For low data rate networking, the in-network distributedprocessing of the SDR nodes plays key role to the mobile sensingapplications, in accelerating the computing as well as reducingthe traffic burden and processing delay.

Index Terms—Wireless Sensor Network, Distributed Comput-ing, Software Defined Radio (SDR)

I. INTRODUCTION

Mobility, as the most important attribute of the wireless sen-sor network, brings challenges to the researcher and industrialuser. Two limitations, communications and computing, in thewireless network compose the main bottleneck to pursue realtime capabilities for many applications.

The goal of the sensing nodes control for the wireless sensornetwork is to provide efficient data collection procedureswith strict time and energy budget. The opportunistic accessprotocol and the predictable data access control are bothinvestigated [1] [2]. The latter is chosen for our testbed due tobetter expected efficiency . A round robin control protocol iswell designed and implemented on the software defined radionetwork.

Distributed or parallel computing has been introduced intowireless sensor network for many data intensive applica-tions [3]. These applications require real time informationupdate while big sensing data are needed. To minimizethe communications traffic, some computing accelerators canbe considered. Three levels of computing accelerators have

been discussed: parallelizing algorithm, scoped computing,and hardware/software parallel computing. In the algorithmlevel, the task or data decomposition is the main method.The computing resource of the whole network could be fullyutilized to mitigate the limitation of each node. In addition,instead of moving all the collected raw data to a centralizedcomputing server, the local scope computing can be involvedwith more weight, depending on the applications. Only thenecessary intermediate filtered data are exchanged within thebandwidth limited network.

The combination of communication and computing over thewireless sensor network raises the challenges on the nodesplatform development. The traditional commercial communi-cations devices could be embedded into sensors. However, thecapability of local processing is a bottleneck, especially whenflexible collaborative computing is needed. Software definedradio platform is a good candidate to implement the duality ofsensing and communications for the wireless sensor network.In the experimental wireless sensor network testbed presentedin this paper, the USRP [4] [5] has been used as the sensornode platform.

The rest of the paper is organized as following: in the secondsection, we propose a centralized architecture of the wirelesssensor network testbed, in which a round-robin control proto-col is implemented for efficient data collection and processing.Then hardware and software architecture of each sensor nodeand the control head is presented with some proposed features.In the next section, the distributed/parallel computing thatcan be embedded into this wireless sensor network testbed isdiscussed. The message passing abstraction is also discussedfor SDR based wireless sensor network.

II. NETWORK ARCHITECTURE AND CONTROL FORWIRELESS SENSING DATA COLLECTION AND PROCESSING

A large scale wireless sensor network can be deployed tocollect big data for data intensive applications. This networkcould contain hundreds of sensing nodes, in a wide geomet-rical scope. Two categories of network architecture can beconsidered. The first choice is the ad-hoc network model.

978-1-4799-2035-8/14/$31.00@2014 IEEE 1402

In this model, the network with a self-organized topologyis robust in security and maintainability, due to its self-healing attribute. The routing of the sensing data in the ad-hocwireless sensor network is very flexible but opportunistic, asthe traffic is mainly relayed by adjacent nodes with resourceoptimization. The other choice is the centralized wirelessnetwork, in which some of the network nodes act as thehead and the others work as sensing nodes. The sensingdata collected by the sensors are all transfered to the head.And the head is also responsible for the data access controlof the network nodes. The centralized wireless network isvulnerable to be attacked. The whole network could collapsewhen the head is down. However, the centralized networkis more feasible in our testbed. The communications in ad-hoc network has overhead in nodes relaying which decreasedthe bandwidth utilization. Meanwhile, within the centralizednetwork, the data collection procedure is easier to be controlledwith simple synchronization. The operation/control delay canbe less opportunistic by careful design of the data accessprotocol.

A centralized wireless cluster implemented over the soft-ware defined radio platform is presented as Figure 1. In thiscluster, one cluster head will lead up to ten cluster nodes.The cluster nodes are capable of performing both sensing andcommunications with the cluster head. The wireless commu-nications between the cluster head and the cluster node canbe implemented based on the arbitrary physical layer, like theOFDM modulation, or the traditional GMSK [6], PSK, etc.,depending on the hardware/software resource of the adoptedSDR platform. On the cluster head, a cluster control statemachine is implemented to control the data collection fromthe cluster nodes.

The overall procedure of the data collection is: at the firstround of the data collection, the cluster head broadcasts thecommand of start sensing to all the sensing nodes, indicatingthe sensing start time T1 and how many samples are to becollected from each node in this round of collecting. Allthe sensing nodes who receives the start sensing commandwill initiate the sensing at same time T1 and collect thesamples with specified numbers. Also, the sensing duration isdetermined by the sampling rate and sample numbers. Afterthe sensing is completed, the sensing nodes will switch to thecommunications mode to report the sensing data to the clusterhead individually. Once the cluster head detects the completionof the data reporting from all the sensing nodes, it will initiateanother round of sensing and data collection with the sameprocedure, controlled by the system user programming.

The protocol used for the data collection control can bebased on CSMA or round robin manner. If CSMA is adopted,the messages exchanging for the collection procedure is sim-pler than round robin manner. The sequence diagram of theCSMA based data collection procedure is shown as Figure 2.In the stage of reporting data, the cluster head does notneed to synchronize the transmission of the sensing nodes.By adding the node number into the data packet format, thecluster head can easily monitor if all the nodes have completed

Fig. 1. Cluster.

data reporting, even though the reported data can be outof order due to the random transmitting conflict. However,with the expanding of the cluster size (number of nodes), theprobability of the transmission could be increased. Sometimesthe time overhead on resolving the conflict can bring toomuch delay to the data collection. This unnecessary delaywill impact the real-time performance of some applications,like the passive target position tracking which requires instantposition update. On the other side, to ensure the network workproperly, the CSMA threshold at each sensor node and clusterhead should be adjusted carefully to increase the detectionprobability while decreasing the false detection probability.This threshold could be set manually or adaptively. In either ofthe ways, the unnecessary complexity and the indeterministicattribute are not good for the data collection procedure.

If the data collection procedure in round robin manner isused, as shown in Figure 3, the cluster head has full controlon the data transmission of the sensing nodes during the datareporting stage. The state machine on the cluster head willinitiate the data reporting of the nodes sequentially, from thenode 1 to node n. Only after the previous node completesthe data reporting, the head notifies the next one to report thedata. When other nodes are reporting the data, the node itselfjust needs to keep the sensing data in its local database. Thecost of such full control of the data collection procedure isthat more commands are needed to notify the next node aboutthe completion of the data transmission of the previous node.But this command could be very short which introduces onlyvery little delay. In this model, all the delay, being composedof the data transmission and commands exchange, is almostpredictable. With careful sequence optimization, the redundantdelay can be very limited. To some extent, the round robincontrol procedure digs out most of the efficiency of the datacollection, with deterministic behavior.

This cluster can be extended to a large scale wireless sensornetwork very easily in a scalable way, as Figure 5 whichis proposed in [7]. The heterogeneous wireless clusters are

978-1-4799-2035-8/14/$31.00@2014 IEEE 1403

Fig. 2. Sequence Diagram of CSMA Based Data Collection.

Fig. 3. Sequence Diagram of Round Robin Control Based Data Collection.

integrated into a bigger network. The connection betweenclusters can be wired or wireless.

Our current wireless sensor network testbed is using thesoftware define radio - USRP as the node platform. GPS isattached onto each node to achieve the network time synchro-nization, as shown in Figure 4. Thus, for each round of thedata collection, the start sensing time can be specified by the

Fig. 4. USRP platform with GPS installed.

Fig. 5. Overall architecture of the testbed for large scale cognitive radiosensing network [7].

Fig. 6. Software Architecture of USRP Based Sensor with Communicator.

head by getting the GPS time from the USRP. The UHD driveron the host PC has the software API to read the GPS time.The software architecture at the cluster node and cluster headis presented in Figure 6. For the cluster node, the frequenciesused for communications and sensing are different. The sensedsamples and the wireless communications information are bothdelivered to the host PC by the Gigabytes Ethernet interface.

III. DISTRIBUTED AND PARALLEL COMPUTING OVER THEWIRELESS SENSOR CLUSTER AND THE RELATED

APPLICATIONS

Communications capability is always a challenge for thewireless sensor network. In this section, our goal is mini-mizing the communication traffic in need by introducing thedistributed or parallel computing within the network scope.

The first aspect in discussion is the algorithm decompositionbased on our network testbed with round robin control pro-tocol. As we know, the parallel computing interface like theMPI(Message Passing Interface) is supported in the one-to-all and all-to-one communications mode between the leadingprocessing node and sub-nodes. The MPI programming can

978-1-4799-2035-8/14/$31.00@2014 IEEE 1404

Fig. 7. Data Decomposition of Matrix-Vector Multiplication.

support the data decomposition parallel computing for thetypical atomics algorithm like the Matrix-Matrix/Vector mul-tiplication. Similarly, our centralized wireless sensor networkcluster supports the one-to-all and all-to-one communicationswith predicable performance by using the round robin controlprotocol. For example, some applications need to fully utilizethe network computing resource to perform tasks like theMatrix-Vector computation. As indicated in Figure 7, thedimension of matrix is 4 by N . The dimension of vector isN by 1. The cluster head can broadcast the vector to all thenodes firstly. Then a block decomposition of the row data isperformed in a even manner, i.e, the row 1 is sent to node 1,the row 2 is sent to node 2, and so on. Thus the computing isparallelized by dividing the Matrix-Vector multiplication to theVector-Vector multiplication at each node. Finally, the clusterhead collects all the partial results from the nodes and does thedata fusion. This procedure has no difference with MPI parallelcomputing over the wired connection. But for the resourcelimited wireless network, it can be seen that our testbed can beused as an ideal experimental network system for investigatingthe real challenges of implementing the parallel computingover the wireless connections.

The more practical view should be placed on those appli-cations which need the local processing, i.e., the in-networkprocessing. Instead of delivering the raw sensing data to thecluster head, all the nodes should filter the raw data firstly andthen transfer the necessary information to the cluster head tomake the decision. The simplest but useful example is fromthe application that requires the frequency domain informationat each sensor node. In the paper [8], the network data usedfor machine learning is composed of the OFDM sub-carriersinformation derived from all the network nodes. It will bringbig burden and delay for the real-time target tracking if all the

time domain data are transfered to the cluster head and then theFast Fourier Transform Algorithm (FFT) is performed at thehead. For example, if the sensing waveform is using 40 sub-carriers of OFDM signal. To get the sub-carriers informationat each node, the 1024 samples are collected by every nodein each round. Each sample is of complex type with size of8 bytes. Thus the total data delivered to the cluster head, in anetwork cluster with 10 nodes, would be 81920 bytes. If theFFT operation and the following feature extraction are doneat each node and only the vector after feature extraction isdelivered, only 240 Bytes of data are needed by the clusterhead. In the latter way, not only the communications trafficis significantly decreased but also the processing delay isdecreased as the cluster head does not need to perform the FFToperation for all the nodes. The in-network processing methodcan be regarded as a kind of task-decomposition parallelism.

Finally, beside the network scope parallelism, each SDRbased node can be improved in computing capability withcombining some shared memory parallel computing mech-anism. For the USRP platform plus host PC, the availableparallel tools include the integrated GPU [9], multi-threadsprogramming, SIMD accelerator - Vector-Optimized Libraryof Kernels (VOLK) [10], etc.

Concrete examples in wireless spectrum sensing will showthe capabilities of this SDR based radio network.

a) Covariance Based Spectrum Sensing: The covari-ance based signal detection method has been widely usedfor wireless spectrum sensing in cognitive radio network[11] [12] [13] [14]. The eigenvalues or trace of the samplecovariance matrix are used for signal detection in these al-gorithms. The computing overhead mainly reside in samplecovariance matrix generation based on the sensing samples.We have demonstrated how the SIMD based programming isused to accelerate the computing speed for calculating samplecovariance matrix, as shown in Figure 8.

3210

3210

Vector Multiplier (SIMD)

C[0,0]

C[1,1]

C[2,2]

C[3,3]

3210

3210

Vector Multiplier (SIMD)

3210

3210

Vector Multiplier (SIMD)

C[0,0]

C[1,1]

C[2,2]

C[3,3]

3210

3210

Vector Multiplier (SIMD)

C[0,0]

C[1,1]

C[2,2]

C[3,3]

C[0,1]

C[1,2]

C[2,3]

C[0,1]

C[1,2]

C[2,3]

C[0,2]

C[1,3]

C[0,0]

C[1,1]

C[2,2]

C[3,3]

C[1,2]

C[1,2]

C[0,1]

C[1,3]

C[2,3]

C[0,3]

1

2

3

4

Thread 1

Thread 2

Fig. 9. Parallel computing over SIMD and multi-thread for outer product.

978-1-4799-2035-8/14/$31.00@2014 IEEE 1405

Stream to vector

USRP Source

Start Indicator of Matrix

n: Vector LengthN: Number of Sample Vectors

In example, n = 32, N = 800

Trace/Eigen Value Calculator

of MatrixTo users

Sample Covariance

Matrix Generator

Matrix of n*n

With unoptimized C++ programming15ms to process 32*800 complex samplesSupport Rate <= (32*800/15)*10e3 = 1.7Ms

With SIMD parallel computing2.5ms to process 32*800 complex samplesSupport Rate <= (32*800/2.5)*10e3 = 10.2Ms

0.8ms to process a 32*32 complex matrix

Sample Rate constrained bySignal ProcessSpeed

6 Speeds accelerated by simple parallel

computing!!

Fig. 8. Real-time computing for calculating sample covariance matrix and its eigenvalues.

Consider a random vector y ∈ Cn. The correspondingcovariance matrix is Y ∈ Cn×n. In the experiment, eachobservation is denoted by yi ∈ Cn. The sample covariancematrix calculated, i.e.,

Y =1

N

N∑i=1

yi

⊗yi − y

⊗y (1)

where⊗

represents the outer product of two vectors, and yrepresents the sample mean of the random vectors. By ordinaryprogramming, the computing delay for a sample covariancematrix calculation with n = 32 and N = 800 is around15ms. Thus, some preliminary optimization for generating thissample covariance matrix has been made.

In this example, calculating the outer product of two iden-tical vectors is the main time consuming procedure. As ex-plained in Figure 9, the outer product operation is decomposedinto vector multiplication via a sliding way. In each step, onlythe green elements of the vectors are used to generate thecorresponding elements for the output outer product matrix.The SIMD programming is adopted to perform the vectormultiplication and the different multiplications can be arrangedinto multiple threads. After such an optimization over parallelcomputing, the computing delay is greatly reduced to less than2.5ms, which is a 6-speed improvement.

The result of the real-time computing of the sample co-variance matrix is encouraging. It unveils the possibility tointegrate the desired algorithms into the network testbed.Parallel computing as well as the high-performance computingwill be explored for algorithm speedup.

b) Distributed and Collaborative Sensing: Other poten-tial important applications of our SDR based radio networkinclude the distributed sensing or collaborative sensing. Byleveraging the spatial diversity of the network deployment,the distributed sensing can be applied to deal with the hiddenterminal problem that the single sensor faces. The distributedsensing system can also decrease the sensing time whilecollecting more sensing data for decision making. Meanwhile,the challenges come from the constraints in communications,computing, energy and efficiency. The in-network processing

capability shown in previous sections will enable the soft deci-sion combination where only the intermediate sensing resultsare needed to be reported to the leading node instead of theraw data. Thus the communications overhead is reduced signif-icantly. Sometimes, to further improve resource efficiency, theoptimal sensor management mechanism is needed to handlethe communications constraints [15]. Dynamic programmingalgorithm can be used for this application [16] [17]. Theparallelization and implementations of the corresponding op-timization algorithm are interesting topic on this SDR networkplatform.

IV. CONCLUSION

This paper presents a SDR based raido network which isapplicable to perform wirless distributed sensing and comput-ing. The network is designed as composed of centralized wire-less clusters which is beneficial to perform efficient networkcontrol and data collection. A round robin control protocol isimplemented over the popular USRP platform. The experimentand analysis have shown the feasibility to introduce the data-decomposition parallelism and task-decomposition parallelisminto this network, for those applications requiring the real-timeperformance. In the future, more applications and researchexperiments will be carried out on this wireless distributedsensing and computing network, especially when the networkis extended to highly mobile platform like UAV(UnmannedAerial Vehicle).

REFERENCES

[1] L. Mottola and G. Picco, “Programming wireless sensor networks:Fundamental concepts and state of the art,” ACM Computing Surveys(CSUR), vol. 43, no. 3, p. 19, 2011.

[2] S. Singhal, A. Gankotiya, S. Agarwal, and T. Verma, “An investigation ofwireless sensor network: A distributed approach in smart environment,”in Advanced Computing & Communication Technologies (ACCT), 2012Second International Conference on. IEEE, 2012, pp. 522–529.

[3] D. Datla, X. Chen, T. Tsou, S. Raghunandan, S. Hasan, J. Reed,C. Dietrich, T. Bose, B. Fette, and J. Kim, “Wireless distributedcomputing: a survey of research challenges,” Communications Magazine,IEEE, vol. 50, no. 1, pp. 144–152, 2012.

[4] [Online]. Available: http://gnuradio.org/[5] [Online]. Available: http://ettus-

apps.sourcerepo.com/redmine/ettus/projects/uhd/wiki

978-1-4799-2035-8/14/$31.00@2014 IEEE 1406

[6] F. Ge, C. Chiang, Y. Gottlieb, and R. Chadha, “Gnu radio-based digitalcommunications: Computational analysis of a gmsk transceiver,” inIEEE Global Communications Conference, 2011.

[7] R. Qiu, C. Zhang, Z. Hu, and M. Wicks, “Towards a large-scalecognitive radio network testbed: Spectrum sensing, system architecture,and distributed sensing.”

[8] C. Zhang, Z. Hu, T. Guo, R. Qiu, and K. Currie, “Cognitive radio net-work as wireless sensor network (iii): Passive target intrusion detectionand experimental demonstration.”

[9] W. Plishker, G. Zaki, S. Bhattacharyya, C. Clancy, and J. Kuykendall,“Applying graphics processor acceleration in a software defined radioprototyping environment,” in Rapid System Prototyping (RSP), 201122nd IEEE International Symposium on. IEEE, 2011, pp. 67–73.

[10] [Online]. Available: http://gnuradio.org/redmine/projects/gnuradio/wiki/Volk[11] Y. Zeng and Y.-C. Liang, “Covariance based signal detections for cog-

nitive radio,” in New Frontiers in Dynamic Spectrum Access Networks,2007. DySPAN 2007. 2nd IEEE International Symposium on. IEEE,2007, pp. 202–207.

[12] ——, “Spectrum-sensing algorithms for cognitive radio based on statis-tical covariances,” Vehicular Technology, IEEE Transactions on, vol. 58,no. 4, pp. 1804–1815, 2009.

[13] ——, “Eigenvalue-based spectrum sensing algorithms for cognitiveradio,” Communications, IEEE Transactions on, vol. 57, no. 6, pp. 1784–1793, 2009.

[14] F. Lin, R. C. Qiu, Z. Hu, S. Hou, J. P. Browning, and M. C. Wicks,“Generalized fmd detection for spectrum sensing under low signal-to-noise ratio,” Communications Letters, IEEE, vol. 16, no. 5, pp. 604–607,2012.

[15] A. O. Hero and D. Cochran, “Sensor management: Past, present, andfuture,” Sensors Journal, IEEE, vol. 11, no. 12, pp. 3064–3075, 2011.

[16] J. L. Williams, J. W. Fisher, and A. S. Willsky, “Approximate dynamicprogramming for communication-constrained sensor network manage-ment,” Signal Processing, IEEE Transactions on, vol. 55, no. 8, pp.4300–4311, 2007.

[17] D. A. Castanon, “Approximate dynamic programming for sensor man-agement,” in Decision and Control, 1997., Proceedings of the 36th IEEEConference on, vol. 2. IEEE, 1997, pp. 1202–1207.

978-1-4799-2035-8/14/$31.00@2014 IEEE 1407