On-demand Linux for Power-aware EmbeddedSensors

Carl Worth, Mike Bajura, Jaroslav Flidr, and Brian SchottUSC/Information Sciences Institute

{cworth,mbajura,jflidr,bschott}@east.isi.edu

Abstract

We introduce a distributed sensor architecturewhich enables high-performance 32-bit Linuxcapabilities to be embedded in a sensor whichoperates at the average power overhead of asmall microcontroller. Adapting Linux to thisarchitecture places increased emphasis on theperformance of the Linux power-up/shutdownand suspend/resume cycles.

Our reference hardware implementation is de-scribed in detail. An acoustic beamformingapplication demonstrates a 4X power improve-ment over a centralized architecture.

1 Introduction

Traditional sensor platform architectures arebased on a hub-and-spoke model with periph-erals clustered around a central processor asshown in Figure 1(a). In this model, the lower-bound of total system power is set by the low-est active mode of the central processor whichmust be continually active to broker peripheraloperations.

System power is typically reduced by usingless-capable processors or microcontrollers inplace of the central processor. Although sensoractivity is mostly infrequent and bursty withlow average computational requirements, peakprocessing requirements can still be quite high.

Processor

RadioSensor

Input/Output

(a) Hub-and-spoke

Sensor

Radio

Processor

Input/Output

(b) Distributed

Figure 1: Alternate sensor node architectures

Within a system design, this creates tension be-tween the desire for the high-performance pro-cessing capability of a larger processor and thelow-power operation of a smaller one.

This tension is further complicated by the ob-servation that while many large processors re-quire significantly more power than small oneswhen inactive, they also often provide signifi-cantly more power-efficient computation whenactive. Another tradeoff in this design spaceweighs the strength of development and de-bugging tools, such as Linux, available forlarger processors versus the constrained pro-gramming environments available for smallones.

Replacing the hub-and-spoke architecture witha distributed model as shown in Figure 1(b)can decouple processing from peripheral op-eration and create a system that combines thestrengths of both large and small processors.

574 • Linux Symposium 2004 • Volume Two

In this model, processor and peripherals be-come autonomous modules that are each pow-ered independently. High-performance pro-cessing can be made available when needed,but without increasing the lower-bound of to-tal system power. Low average system powercan be achieved by operating for a majority ofthe time in extremely low-power modes withonly essential modules active.

This distributed architecture places Linux in anunconventional role as a peer module ratherthan as a central processor. This emphasizesthe performance of the power-up/shutdown andsuspend/resume cycles as keys for achievinglow average system power.

The remainder of this paper is organized as fol-lows. Section 2 describes several popular re-search and commercial sensor platforms. Sec-tion 3 recounts the design challenges we facedas we built a reference sensor node with au-tonomous modules. As usual, real-world issuesforced difficult engineering decisions. Sec-tion 4 details the modules we have built so far.

Our hardware is in a more complete state thanour software. We have identified some aspectsof the behavior of Linux that we need to inves-tigate more fully. These issues are discussedin Section 5. Section 6 contains power resultswe have obtained with a vehicle tracking algo-rithm. Finally, Section 7 draws conclusion andSection 8 describes areas for future work.

2 Related Work

Applied research in wireless sensor networkshas made use of a variety of platforms withvarying processing capabilities and power re-quirements, but almost always with a hub-and-spoke model. Several platforms are describedbelow in order from more-capable, higher-power platforms to less-capable, lower-powerplatforms.

2.1 PC/104

PC/104 [3] is a well-supported specification forPC-compatible, embedded systems consistingof stacking modules. Cerpa et al [2] chosePC/104 systems for their “high end” sensorsin a tiered deployment for habitat monitoring.Their cited reasons for choosing PC/104 in-clude the ability to run PC-compatible software(i.e. Linux) and the wide spectrum of availablePC/104 modules.

PC/104 provides great flexibility and the powerrequirements are lower than that of desktopPCs. However, at 1-2 Watts per module[3], andwith most sensors requiring at least two mod-ules, this platform requires too much power formany sensor applications.

2.2 Embedded StrongARM devices/PDAs

Off-the-shelf devices based on low-power, em-bedded processors such as the Intel SA-1110or PXA25x offer another convenient platformfor sensor network research. Compared to x86-class processors, these processors offer a sig-nificant power savings along with a reductionin maximum clock rate and the absence of ahardware floating-point unit.

Representative devices of this class include theHP iPAQ, the CerfCube[4], and the CrossbowStargate[14]. These devices are extensible viaCompact Flash, Bluetooth, etc. but have lessflexibility than PC/104. And while the powerrequirements of these systems can be less thana comparable PC/104 stack, Mainwaring etal[9] found that at 2.5W active power, thepower usage of the CerfCube was excessive forlong-term use in a sensor network.

Several research sensors have been developedwith architectures similar to these off-the-shelfplatforms. These include theµAMPS[10] andWINS[1] nodes. For example, the WINS node

Linux Symposium 2004 • Volume Two • 575

Compact Flash

SDRAMCPU

b) Acoustic Tripwire* Radio Module* Power/Battery* Sensor + DSP

a) Radio Relay* Radio* Power/Solar

FPGA

c) Tracker/Imager* Radio Module* Power/Battery* Sensor + DSP* FPGA/Imager* Embedded Processor* Compact Flash

Figure 2: Power-aware sensor concept

has a central 133MHz StrongARM SA-1100processor along with radio and sensor periph-erals. As measured by Raghunathan et al[12]the WINS node operates in the range of 360-1080 mW and can also be placed into a 64 mWsleep mode.

2.3 Motes

An example of a very low-power sensor ar-chitecture is that of the Berkeley Motes[6, 7,8]. Current Motes are based around a centralmicrocontroller (MCU) such as the ATmega90LS8535, an 8-bit MCU with 128KB Flashand 8KB SRAM. The Mote includes a radioand has serial connections and 10-bit analogADC ports to various sensors on expansionmodules. Typical power consumption for thissensor when active is in the 10-100 mW range.Sleep power is about 60µW.

Mote-class sensors demonstrate that a widevariety of low-bandwidth sensing applicationscan be accomplished with very small proces-sors and with very little memory. The limita-tion of these systems is encountered when an

application doesn’t fit within the memory andprocessing footprint of the MCU. High band-width sensor processing is beyond the capabil-ities of these small-scale sensors and there islittle room for expansion.

3 Implementation

Our primary system design goal was to con-struct a family of interchangeable processor,sensor, and communication modules that canbe mixed and matched according to the appli-cation requirements. Ideally, our architecturewould be able scale from simple sensors asshown in Figure 2(a) and Figure 2(b), to com-plex sensors as shown in Figure 2(c) withouthaving to learn and port to a new platform atevery scale.

Other goals were driven by practical experi-ences of using other platforms in the field.Rapid prototyping is an important concern.The ability to create testbeds using COTS pe-ripherals is a strength of platforms such asPC/104. Availability of Linux device drivers

576 • Linux Symposium 2004 • Volume Two

and a friendly programming environment werestrong motivators in our implementation deci-sions. Data collection is an important step insensor network algorithm development. Wewanted lots of data storage and data network-ing options in our new platform.

Primary constraints on the system design in-clude size and power. We targeted the sizeof the Berkeley Mote, while still supporting aLinux-capable processor in the stack. In theend, the size was dictated by the minimumfootprint of a Compact Flash socket and ourchosen stack connector. Power in our systemneeds to be able to scale from 1 mW to a fewWatts. The low power target limited many ofour implementation choices.

An early design decision was how the au-tonomous modules would communicate. Inter-faces such as ethernet were quickly dismisseddue to power requirements. In small embeddeddevices, the most power-efficient communica-tion is available with hardware-supported inter-faces such as Serial Peripheral Interface (SPI),Controller Area Network (CAN), UniversalAsynchronous Receiver Transmitter (UART),and Inter-Integrated Circuit (IIC or I2C). Eachof these interfaces has strengths and weak-nesses. I2C supports multi-master operation,but has a limit of 100kbps on most devices. SPIhas faster transfers, (up to a few megabits persecond), but has limited multi-master supportin most devices and little flow control. UARTis widely supported, but requires clock agree-ment on both interfaces. CAN is multi-master,but the bus drivers in the supporting devices arerelatively high power, (since CAN is designedfor long cables).

We decided to support the three major interfacestandards (I2C, SPI, and UART). We allocatedsix 8-bit channels on our connector and spec-ified two preferred channels for each standardinterface. In order to prevent bus contention

and power leakage when modules are off, allmodules have bus isolation switches betweenthemselves and the connector. We also intro-duced a separate I2C control network for mod-ule discovery and coordination of the switches.

A small microcontroller (MCU) is standardon most modules to control the bus isolationswitches, a power switch, and any module-specific functions. The MCUs are intended tobe always on when the node is operating, (witha power overhead as low as .05 mW). Theynetwork with each other over I2C to facilitatemodule discovery and coordinate access to thechannels using a common messaging protocol.

Figure 3 contains a diagram of the featurescommon to each module, as well as an optionalprocessor expansion bus so that high-speed,high-power peripherals, (USB, Compact Flash,LCD, and AC97 audio), can be used in thestack or removed for low-power operation.

Figure 3: Power-aware module diagram

4 Available Modules

Our hardware modules are small boards ap-proximately 6.5×4.5cm (2.5×1.75"), with a

Linux Symposium 2004 • Volume Two • 577

180-pin connector on either side. So far, wehave designed, built, and tested 4 modules, 3of which are shown in Figures 4 and 5.

PXA A module including an Intel PXA255XScale processor, 64MB of SDRAM and32MB of Flash. This board supports dy-namic voltage scaling, an active clock raterange of 100-400MHz, and a 33MHz idlemode. An SA-1111 coprocessor providessupport for USB master and two CompactFlash cards. All interface lines are routedto the stack connector.

ADC A four-channel, 12-bit analog-to-digitalconverter module. The MCU has suffi-cient memory for a dedicated 7kB sam-ple buffer. Basic signal processing canbe performed by the local MCU or sam-ples can be efficiently transmitted over anSPI interface to the PXA module for ad-vanced processing. An important point isthat the PXA255 processor can be off orsuspended during data sampling.

IOB The power & I/O board is the only re-quired module in the stack. It provides theprimary power supply and contains mostof the digital I/O connectors, (USB mas-ter and slave, SPI, I2C, and UART).

FPGA This module was developed as an em-ulation board for a low-power DSP be-ing developed at MIT[15], but is interest-ing for other high-performance applica-tions. It contains a Xilinx Virtex-II 3000FPGA, 2MB of synchronous SRAM, and32MB of SDRAM. This module operatesas a coprocessor on the memory bus of thePXA255 and supports the two SPI chan-nels on the stack.

We also have a Compact Flash (CF) board,two of which can be placed into the stack.This board connects to the processor expansion

Figure 4: PXA, ADC, and IOB modules at ac-tual size

Figure 5: Stacked modules

578 • Linux Symposium 2004 • Volume Two

bus so that it acts as a daughter-board of thePXA module rather than an independent mod-ule. Figure 6 shows a stack consisting of IOBand PXA modules along with a CF board pop-ulated with a CF ethernet adapter.

Figure 6: Stack including CF board

Table 1 shows design-time estimates of thepower consumed by each module for variousoperational modes. In the “off” mode every-thing on the module is powered off except forthe power-control MCU. The PXA module hasthe widest operational power range due to thethe range of processor clock rates and variouspossibilities in processor and memory utiliza-tion. Figure 7 provides more details of thehow the PXA module power can scale from0.05 mW to 1.5 W. The innermost portion ofthis diagram includes figures for the overheadof power conversion and the 32kHz MCU onthe IOB module.

Module Mode PowerPXA off 0.05 mWPXA suspended 2.5 - 7.5 mWPXA active 150 - 1530 mWIOB active 0.1 mWADC off 0.05 mWADC active 40 mW

Table 1: Power modes for various modules

5 Impact on Linux

In a conventional hub-and-spoke model, Linuxruns on a central processor and manages somenumber of peripheral devices. In contrast, ourdistributed architecture places Linux on an au-tonomous module which is a peer to othermodules. Any other module might request apower transition of the Linux module, from offto powered, from suspended to active, etc.

The efficiency of these power transitions isa critical component of the average systempower. The distributed platform is designedto achieve low average system power throughaggressive duty-cycling of high-powered com-ponents. The time and energy spent duringpower-mode transitions is overhead that mustbe amortized, imposing limits on practical dutycycles that can be used.

We are currently using Linux version 2.4.21with the standard ARM and PXA patches aswell as customizations for our PXA module.The user-level software distribution is derivedprimarily from the handhelds.org[11] Famil-iar distribution. Our reference sensor appli-cation (see Section 6) does not turn the PXAmodule off, but does suspend/resume the pro-cessor aggressively to achieve active operationfor a few milliseconds once per second. Thetime spent during suspend/resume is dividedbetween time spent in driver callbacks and timespent in the kernel proper. Table 2 shows thetimes we have measured for these transitionson our PXA module.

Transition TimeSuspend drivers 507 msSuspend kernel 13 msResume kernel 78µsResume drivers 25 ms

Table 2: Linux transitions with driver callbacks

Linux Symposium 2004 • Volume Two • 579

Figure 7: PXA module power states

We expect the suspend/resume transitions ofthe Linux kernel itself to behave in a symmetricfashion. However, we measured a very respon-sive resume time of 78µs and a much slowersuspend time of 13 ms. We do not yet have acomplete explanation for why the suspend pro-cess is so much slower, although we have ac-counted for a 1 ms delay that is caused by theMCU on our PXA board, and therefore not afeature of the standard Linux kernel.

A much more significant problem is the timespent in the power management callbacks ofvarious subsystems and drivers. A suspendtime of 520ms spells disaster for an applicationsuch as the one described in Section 6.

We quickly tracked down the source of thislong suspend time to the USB OHCI driver(hcd). An ill-fated decision in our design wasthe choice of the SA-1111 coprocessor. Onedifficulty we encountered was that we had toprovide our own suspend/resume callbacks asthey do not exist for the SA-1111-based hcd in

the standard kernel. To simplify the task, wehave ported the PCI-based OHCI code whichcontains a call tomdelay (500) along withthe comment, “Suspend chip and let things set-tle down a bit.” This single 500ms delay ac-counts for over 98% of the time required tosuspend drivers. We suspect that this constantcan be safely reduced so that much of the timelost during suspend can be recovered. Even so,USB-related timeouts, etc. are on the order ofmilliseconds—orders of magnitude more thanthe time required by the kernel.

Clearly, this poses a serious problem for ap-plications with a high duty cycling require-ment. We are currently working around thelong driver suspend times by simply remov-ing drivers for non-essential devices and sub-systems, (such as SA-1111), prior to runningan application with a restricted power budget.This allows the convenience of things such asusing a USB 802.11 adapter during develop-ment and debugging without the long suspendtimes of the USB drivers during execution.

580 • Linux Symposium 2004 • Volume Two

6 Results

We implemented a vehicle tracking algorithmusing 4-channel acoustic beamforming. In thisapplication, data is continually sampled at arate of 1kHz, but signal processing only needsto be performed at a maximum rate of 1Hz.

In previous work[13], this algorithm was im-plemented on a successor to the WINS node,(a hub-and-spoke platform with an Intel SA-1110 processor). Although efficient signal pro-cessing software was developed, system powersavings were modest since the processor hadto remain active (yet mostly idle) at all timessimply to drive data collection.

We have ported the algorithm to the distributedplatform described in this paper. On this plat-form, the signal processing for one second’sworth of data can be completed in 3 ms bythe PXA255 processor running at 100MHz.Since this is a newer processor than the SA-1110 of the WINS platform, direct compari-son of power numbers between the two plat-forms would be unfair. Instead, we estimatethe power needed for two implementations ofthe algorithm on the distributed node.

The first version is intended to behave as if in ahub-and-spoke system. The processor remainsactive at all times to store samples into mainmemory. The second version takes advantageof the distributed nature of the platform. Linuxon the PXA module is suspended as much aspossible while the ADC module continues tosample and buffer data. This approach addsthe overhead needed to suspend/resume Linuxand to transfer data from the ADC module tothe PXA module over the SPI channel. Theamount of data to be transferred is 8192 bytes,(4 channels∗ 1024 samples/s∗ 2 bytes/sample∗ 1 s). The SPI transfer rate is 1.8 Mbps yield-ing a total transfer time of 36.4 ms.

We measured the power consumed by the PXA

module at two different stages in the algo-rithm. During active computation the PXAmodule consumes 528 mW. When mostly idle,(e.g. when transferring 1kHz data from theADC module), it consumes 370 mW. We havenot yet measured the average power consumedduring the suspend or resume transitions, butwe use an estimate of 370 mW. This estimateshould be conservative as the actual power us-age should ramp down to less than 10 mW dur-ing the transition.

Combining these measurements with estimatesfrom Table 1 and the time measurements fromTable 2, we compute the total energy spent tocompute one result per second. From this wecan determine the average power necessary forthe complete algorithm. These results are givenfor both versions of the algorithm in Tables 3and 4.

Module/Mode Power Time EnergyIOB active 0.1 mW 1 s 0.1 mJ

ADC active 40 mW 1 s 40.0 mJPXA active 528 mW 3 ms 1.6 mJPXA idle 370 mW 997 ms 368.9 mJ

Estimated energy per second410.6 mJEstimated system power: 411 mW

Table 3: Hub-and-spoke power requirementsfor beamforming

Module/Mode Power Time EnergyIOB active 0.1 mW 1.0 s 0.1 mJ

ADC active 40 mW 1.0 s 40.0 mJPXA suspended 7.5 mW 948 ms 7.1 mJPXA resuming 370 mW 78µs 28.9µJPXA transferring 370 mW 36.4 ms 13.5 mJPXA processing 528 mW 3 ms 1.6 mJPXA suspending 370 mW 13 ms 4.8 mJ

Estimated energy per second95.9 mJEstimated system power: 96 mW

Table 4: Distributed power requirements forbeamforming

Linux Symposium 2004 • Volume Two • 581

7 Conclusion

The 96 mW beamforming result marks a suc-cess for the distributed sensor platform—a 4Xpower reduction over the 411 mW required forthe hub-and-spoke platform. This shows that itis possible to take advantage of 32-bit, Linuxprocessing without average power exceedingthe 10-100 mW range of a less-capable sensorbased on a 8-bit microcontroller, (i.e. a Mote).

8 Future Work

The field of power-aware sensing is rich, andwe have only just begun to explore the possibil-ities, even within our own platform. Many ap-plications require a much smaller power budgetthan the 96 mW result we have demonstrated.Our long-term goal is to design sensors capa-ble of operating entirely from scavenged en-ergy, (e.g. solar), which requires operation inthe range of 1 mW[5].

We are currently building a low-power “trip-wire” module which will implement single-channel acoustic vehicle detection. This willallow the PXA module to be completely offwhen a vehicle is not present. We antici-pate that this will allow beamforming withina power budget as low as 10 mW.

As mentioned in Section 5, there remains a fairamount of engineering and research with re-gards to the role of Linux within a distributedsensor. This includes reducing the time re-quired in the power management callbacks ofall relevant drivers as much as possible. Anadditional task is to move from Linux version2.4 to 2.6. The dynamic nature of the new uni-fied device model in 2.6 should make a naturalfit with a platform consisting of autonomousmodules that can be powered on and off at anypoint.

Additionally, new power-scheduling researchcould better take advantage of the wide rangeof power modes in this platform. For exam-ple, Linux could monitor the frequency andduty cycles of the power-up/shutdown and sus-pend/resume cycles. It would then be possi-ble to provide intelligent feedback into thesecycles based on the relative overhead of eachtransition. This work would complement cur-rent efforts that dynamically adjust voltage andclock rate based on system load.

9 Availability

This work has been developed as part of thePower-Aware Sensing Tracking and Analysis(PASTA)project, but we hope that it will beuseful to researchers and hobbyists with a widerange of applications. To that end we areworking toward making the hardware mod-ules available at cost. Further details will bemade available at the PASTA website,http://pasta.east.isi.edu . All of the soft-ware developed under PASTA is also availablethere under the terms of the GNU General Pub-lic License (GPL).

10 Acknowledgements

This research is sponsored by the Defense Advanced Re-

search Projects Agency and Air Force Research Labora-

tory, Air Force Materiel Command, USAF, under agree-

ment number F33615-02-2-4005. The U.S. Government

is authorized to reproduce and distribute reprints for

Governmental purposes notwithstanding any copyright

annotation thereon. The views and conclusions con-

tained herein are those of the authors and should not be

interpreted as necessarily representing the official poli-

cies or endorsements, either expressed or implied, of the

Defense Advanced Research Projects Agency, the Air

Force Research Laboratory, or the U.S. Government.

582 • Linux Symposium 2004 • Volume Two

References

[1] Jonathan R. Agre, Loren P. Clare,Gregory J. Pottie, and Nikolai P.Romanov. Development platform forself-organizing wireless sensor networks.In Proceedings of Aerosense, pages257–268. International Society ofOptical Engineering, 1999.

[2] Alberto Cerpa, Jeremy Elson, DeborahEstrin, Lewis Girod, Michael Hamilton,and Jerry Zhao. Habitat monitoring:Application driver for wirelesscommunications technology. InACMSIGCOMM Workshop on DataCommunications in Latin America andthe Caribbean, April 2001.

[3] PC/104 Consortium. PC/104Embedded-PC Modules, 2004.http://www.pc104.org/ .

[4] Intrinsic Corporation. CerfCubeembedded strongarm system.http://www.intrinsyc.com/products/cerfcube/ .

[5] L. Doherty, B.A. Warneke, B. Boser, andK.S.J. Pister. Energy and performanceconsiderations for smart dust. InInternational Journal of Parallel andDistributed Sensor Networks, 2001.

[6] Jessica Feng, Farinaz Koushanfar, andMiodrag Potkonjak.System-architectures for sensor networksissues, alternatives, and directions. InInternational Conference on ComputerDesign: VLSI in Computers andProcessors, page 226. IEEE, 2002.

[7] Jason Hill, Robert Szewczyk, Alec Woo,Seth Hollar, David E. Culler, andKristofer S.J. Pister. System architecturedirections for networked sensors. InArchitectural Support for Programming

Languages and Operating Systems,pages 93–104, 2000.

[8] J.M. Kahn, R.H. Katz, and K.S.J. Pister.Next century challenges: mobilenetworking for smart dust. InProceedings of the fifth annualACM/IEEE international conference onMobile computing and networking,pages 271–278. ACM Press, 1999.

[9] Alan Mainwaring, David Culler, JosephPolastre, Robert Szewczyk, and JohnAnderson. Wireless sensor networks forhabitat monitoring. InProceedings of the1st ACM international workshop onWireless sensor networks andapplications, pages 88–97. ACM Press,2002.

[10] R. Min, M. Bhardwaj, S. Cho, A. Sinha,E. Shih, A. Wang, and A. Chandrakasan.An architecture for a power-awaredistributed microsensor node, 2000.

[11] The Handhelds.org Project.Handhelds.org (web document).http://www.handhelds.org .

[12] V. Raghunathan, C. Schurgers, S. Park,and M. Srivastava. Energy awarewireless microsensor networks, 2002.

[13] R. Riley, S. Thakkar, J. Czarnaski, andB. Schott. Power-aware acousticbeamforming. InProceedings of theFifth International Military SensingSymposium, 2002.

[14] Crossbow Technology. Stargate xscaleprocessor platform.http://www.xbow.com .

[15] A. Wang and A. Chandrakasan.Energy-efficient dsps for wireless sensornetworks, 2002.

Proceedings of theLinux Symposium

Volume Two

July 21st–24th, 2004Ottawa, Ontario

Canada

Conference Organizers

Andrew J. Hutton,Steamballoon, Inc.Stephanie Donovan,Linux SymposiumC. Craig Ross,Linux Symposium

Review Committee

Jes Sorensen,Wild Open Source, Inc.Matt Domsch,DellGerrit Huizenga,IBMMatthew Wilcox,Hewlett-PackardDirk Hohndel,IntelVal Henson,Sun MicrosystemsJamal Hadi Salimi,ZnyxAndrew Hutton,Steamballoon, Inc.

Proceedings Formatting Team

John W. Lockhart,Red Hat, Inc.

Authors retain copyright to all submitted papers, but have granted unlimited redistribution rightsto all as a condition of submission.