Networks of Tiny Devices Embedded in the Physical World

Networks of Tiny Devices Embedded in the Physical World

David Culler Computer Science Division

U.C. Berkeleywww.cs.berkeley.edu/~culler

Intel Research Berkeley

http://www.cs.berkeley.edu/~culler

5/13/2002 TinyOS IPAM 2

Technology Push

• Complete network embedded systems going microscopic

ProcessingStorage

Sensing

Actuation

Communication

LNAmixerPLL basebandfilters

I Q

Power


Application Pull

• Complete NW embedded systems going microscopic

• Huge space of new applications

Circulatory Net

Habitat Monitoring

Condition-based maintenance

Disaster Management

Ubiquitous

computing

Monitoring & Managing Spaces


Bridging the Technology-Application Gap

• Power-aware, communication-centric node architecture

• Tiny Operating System for Range of Highly-Constrained Application-specific environments

• Network Architecture for vast, self-organized collections

• Programming Environments for aggregate applications in a noisy world

• Distributed Middleware Services (time, trigger, routing, allocation)

• Techniques for Fine-grain distributed control• Demonstration Applications


Critical issues

• Highly constrained devices– power, storage, bandwidth, energy, visibility

– primitive I/O hierarchy

• Observation and action inherently distributed– many small nodes coordinate and cooperate on overall task

• The structure of the SYSTEM changes

• Devices ARE the infrastructure– ad hoc, self-organized network of sensors

• Highly dynamic– passive vigilance most of the time

– concurrency-intensive bursts

– highly correlated behavior

– variation in connectivity over time

– failure is common


A de facto platform for EmNets

• Developed a series of wireless sensor devices

• TinyOS concurrency framework• Messaging Model• Networking stacks (RF and Serial)• Multihop routing• Several Key components

– sensing, logging, data filters, broadcast

• Simulation tools• DARPA NEST OEP

USC robomote


Many Research Groups using it

• UCB– NEST

– SensorWeb

– Blackout

– Glaser structures

– CBE

– BFD

– BRWC

• UCLA

• USC, ISI

• Rutgers winlab

• Intel

• Bosch

• Crossbow

• U Wash

• Rutgers

• UIUC

• NCSA

• U Virginia

• Ohio State

• UCSD

• Dartmouth

• MIT

• UT Austin, ASU, Iowa

• Accenture

• and many more


The MICA architecture

• Atmel ATMEGA103 – 4 Mhz 8-bit CPU– 128KB Instruction Memory– 4KB RAM

• 4 Mbit flash (AT45DB041B)– SPI interface– 1-4 uj/bit r/w

• RFM TR1000 radio– 50 kb/s – ASK– Focused hardware acceleration

• Network programming• Rich Expansion connector

– i2c, SPI, GIO, 1-wire

– Analog compare + interrupts

• TinyOS tool chain • sub microsecond RF

synchronization primitive2xAA form factor

Atmega103 Microcontroller

TR 1000 Radio Transceiver4Mbit External Flash

51-Pin I/O Expansion Connector

Power Regulation MAX1678 (3V)

DS2401 Unique ID

8 Analog I/O8 Programming

Lines

SP

I Bus

CoprocessorTransmission Power Control

Hardware Accelerators

Digital I/O


Rich Sensor board

PHOTO

TEMP

MAGNETOMETER ACCELEROMETER

MICROPHONE

SOUNDER

Mica PINS

ADC Signals (ADC1-ADC6)I2C BusOn/Off ControlInterrupt

X AxisY Axis Gain Adjustment

Mic Signal

ToneIntr

2.25 in

1.25 in

Microphone

AccelerometerLightSensor

TemperatureSensor

SounderMagnetometer


More Sensors and Actuators

• Motor-Servo board interfaces any combination of two motors, servos, and solenoids to a toy car platform

• whisker board for obstacle detection

• digital accelerometer (ADXL202) board for crude odometry

• GPS Board

• Weatherboard

• light, temp, humidity, barometric pressure, occupancy (thermopile)


Acadia National ParkMt. Desert Island, ME

Great Duck IslandNature Conservancy

Getting Ready for Outdoors


A Operating System for Tiny Devices?

• Traditional approaches– command processing loop (wait request, act, respond)

– monolithic event processing

– bring full thread/socket posix regime to platform

• Alternative– provide framework for concurrency and modularity

– never poll, never block

– interleaving flows, events, energy management

=> allow appropriate abstractions to emerge


Tiny OS Concepts

• Scheduler + Graph of Components– constrained two-level scheduling model:

threads + events

• Component:– Commands, – Event Handlers– Frame (storage)– Tasks (concurrency)

• Constrained Storage Model– frame per component, shared stack, no

heap

• Very lean multithreading• Efficient Layering

Messaging Component

init

Po

we

r(m

od

e)

TX

_p

ack

et(

bu

f)

TX

_p

ack

et_

do

ne

(s

ucc

ess

)RX

_p

ack

et_

do

ne

(b

uff

er)

Internal

State

init

po

we

r(m

od

e)

sen

d_

msg

(ad

dr,

ty

pe

, d

ata

)

msg

_re

c(ty

pe

, d

ata

)

msg

_se

nd

_d

on

e)

internal thread

Commands Events


Application = Graph of Components

RFM

Radio byte

Radio Packet

UART

Serial Packet

ADC

Temp photo

Active Messages

clocks

bit

by

tep

ac

ke

t

Route map router sensor appln

ap

pli

ca

tio

n

HW

SWExample: ad hoc, multi-hop routing of photo sensor readings

3450 B code 226 B data

Graph of cooperatingstate machines on shared stack


TOS Execution Model

• commands request action– ack/nack at every boundary

– call cmd or post task

• events notify occurrence– HW intrpt at lowest level

– may signal events

– call cmds

– post tasks

• Tasks provide logical concurrency

– preempted by events

• Migration of HW/SW boundary

RFM

Radio byte

Radio Packet

bit

by

tep

ac

ke

t

event-driven bit-pump

event-driven byte-pump

event-driven packet-pump

message-event driven

active message

application comp

encode/decode

crc

data processing


Dynamics of Events and Threads

bit event filtered at byte layer

bit event => end of byte =>

end of packet => end of msg send

thread posted to start

send next message

radio takes clock events to detect recv


Maintaining Scheduling Agility

• Need logical concurrency at many levels of the graph

• While meeting hard timing constraints– sample the radio in every bit window

Retain event-driven structure throughout application

Tasks extend processing outside event windowAll operations are non-blocking

lock-free scheduling queue


Demonstration applications

• 29 Palms• Cory Hall network

– ½ million packets over 3 weeks

• Surge network and environment display• 800 node ad hoc network• CBE • Glaser Shakes• Granlibakken retreat watcher• Robomote• Group response

=> continued application focus• more real and long lived• more dynamics• extract architecture and create framework


Example TinyOS study

• UAV drops 10 nodes along road,– hot-water pipe insulation for package

• Nodes self-configure into linear network

• Synchronize (to 1/32 s)

• Calibrate magnetometers

• Each detects passing vehicle

• Share filtered sensor data with 5 neighbors

• Each calculates estimated direction & velocity

• Share results

• As plane passes by, – joins network

– upload as much of missing dataset as possible from each node when in range

• 7.5 KB of code!

• While servicing the radio in SW every 50 us!


Structural performance due to multi-directional ground motions (Glaser & CalTech)

.

Wiring for traditional structural instrumentation+ truckload of equipment

Mote infrastructure15

13

14

6

5`

15

118

Mote Layou

t 129

Comparison of Results


Energy Monitoring/Mgmt System

• 50 nodes on 4th floor• 5 level ad hoc net• 30 sec sampling• 250K samples to database

over 6 weeks


Energy Monitoring Network Arch

sensor net

GW GW

802-11

control net

GW

20-ton chiller

PC

scada term

modbus

UCB power monitor net

PC telegraphMYSQL

Browser


Meeting Social Network


Wealth of Research Challenges

• Large numbers of highly constrained (energy & capability), connected devices

– able to be casually deployed in infrastructure (existing or in design)

– imperfect operation and reliability

– operating in aggregate

• New family of issues across all the layers

application

service

network

system

architecture

technology

mg

mt

/ dia

g /

deb

ug

alg

ori

thm

/ th

eory

pro

g /

dat

a m

od

el


Node Communication Architecture

Application Controller

RF Transceiver

Protocol Processor

Narrow, refined Chip-to-Chip Interface

Raw RF Interface


RF Transceiver

Serialization Accelerator

Timing Accelerator

Mem

ory

I/O

BU

S

Hardware Accelerators


RF Transceiver

Classic Protocol

Processor

Direct Device

Control

Hybrid Accelerator


Novel Protocol Examples

• Low-power Listening

• Really Tight Application-level Time Synchronization

• Localization

• Wake-up

• MACs

• Self-organization


Low-Power Listening

• Costs about as much to listen as to xmit, even when nothing is received

• Must turn radio off when there is nothing to hear.

• Time-slotting and rendezvous cost BW and Latency

• Can turn radio on/of in <1 bit Small sub-msg recv sampling Trade small, infrequent tx

cost for freq. Rx savings Low Power Listening

Receiving individual bits

Start Symbol Detection

Synchronization

Radio Samples

MAC Delay Transmitting individual bits

Start Symbol Transmission

Bit Modulations

Transmission

Reception

Start Symbol Search

Slow, Periodic Sampling

Preamble


Exposing Time Synchronization Up

• Many applications require correlated data sampling

• Distributed time sync accuracy bounded by ½ the variance in RTT.

• Successful radio transmission requires sub-bit synchronization

Provide accurate timestamping with msg delivery

Jitter < 0.1us (propagation) + 0.25 us (edge capture accuracy) + 0.625 us (clock synch)


Localization

• Many applications need to derive physical placement from observations

– Spatial sampling, proximity, context-awareness

• Radio is another sensor• Sample baseband to

estimate distance– Need a lot of statistical data– Calibration and multiple-

observations are key

• Acoustic time-of-flight alternative

– Requires good time synchronization

NoiseErrorNoiseError


Statistical Approach

Distance

Pro

babi

lity

X

Y

Node A Node B


Integrated Architecture

• Chip Area ~5 mm2

– AVR core with protocol Accelerators .5 mm2

– 16 Kbytes on-chip ram 4 mm2

– ADC

– 800-1GHz FSK transceiver, -90dBm receive sens’y .5 mm2

• Expected sleep current = 1 uW – lifetime on a single AA = 400+ years

• Expected active (processing current) – Processor @ 4 Mhz < 1 mW

– Radio: 1mW power consumption, 100Kbps

– ADC: 20 pJ/sample 10 Ksamps/second = .2 uW.


Networking

• Hands-on Experience with Large Networks of Tiny Network sensors

intense constraints, freedom of abstraction

• Re-explore entire range of networking issues

– encoding, framing, error handling

– media access control, transmission rate control

– discovery, multihop routing

– broadcast, multicast, aggregation

– active network capsule (reprogramming)

– localization, time synchronization

– security, network-wide protection

– density independent wake-up and proximity est.

• Fundamentally new aspects in each


The Nodes are the Infrastucture

• Simple Epidemic Algorithm Schemaif (new mcast) then

take local actionretransmit modified request

• Examples: Network wakeup, command propagation

– Build spanning tree» record parent

• Naturally adapts to available connectivity• Minimal state and protocol overhead

=> surprising complexity in this simple mechanism


Network Discovery: Radio Cells


Network Discovery


Controlled Empirical Study

• Experimental Setup– 13x13 grid of nodes– separation 2ft– flat open surface– Identical length antennas, pointing vertically upwards.– Fresh batteries on all nodes– Identical orientation of all nodes– The region was clean of external noise sources.

• Range of signal strength settings• Log many runs


Example “epidemic” tree formation


Final Tree


Power Laws ?

• Most nodes have very small degree (ave = .92)• Some have degree = 15% of the population• Few large clusters account for most of the edges

1

10

100

1000

1 10 100

Cluster Size (1 + # children)

Co

un

t

1

10

100

1000

1 10 100

Cluster SizeL

ink

s


Open Territory => Many Children

• Example: Level 1



• Example: Level 2 – variation in distance



• Example: Level 3 – long links


Understanding Connectivity

• 16 transmit power settings• For each transmit power setting,

each node transmits 20 packets.• Receivers log successfully

received packets.• Nodes transmit one after the other

in a token-ring fashion • No collisions.

• Define “range”: radius where 75% of enclosed nodes receive 75% of packets

• Often good nodes at a distanceprobability of reception from center node vs xmit strength


Importance of Asymmetric Links

• Asymmetric Link: – >65% successful reception in one direction – <25% successful reception in the other direction

• 10%-25% of links are asymmetric• Many long links are asymmetric

– in large field it is likely that someone far away can hear you– what does this mean for protocol design?


Collisions are primary factor

• Nodes out of range may have overlapping cells– hidden terminal effect

• Collisions => these nodes hear neither ‘parent’– become stragglers

• As the tree propagates – folds back on itself

– rebounds from the edge

– picking up these stragglers.

• Seen in many experiments

• Mathematically complex because behavior is not independent beyond singe cell


Stragglers

• significant fraction of links point ‘backwards’


Minimal lessons learned

• Don’t think about wireless networks as bunch of circles of radius r

– connectivity is a probability distribution

– falls off with distance, but not as simple fading law

– shape varies with time and context

• With large, dense arrays the low-probability events are common

• Must strike a balance in exploiting structure and adapting to observed behavior

• Want simple local rules that have predictable, robust global behavior


More typical routing for sensor nets

• Current applications dominated by data acquisition

– route from many nodes to nearest gateway

– aggregate from many nodes

– routing determined by simple local rules

• Nodes listen to data transfers from neighbors– carries hop-count info

– monitors link goodness of potential ‘parents’

– dynamically selects best node is lesser hop count

– includes hysterisis and continuous rediscovery

– gateways emit null data with 0-hop

• Much to understand about how such algorithms manage major change


Self-propagating Programs?

• TinyOS components support class of applns.

• Tiny virtual machine adds layer of interpretation for specific coordination

• Primitives for sensing and communication

• Small capsules (24 bytes)

• Propagate themselves through network

Network Programming Rate

0%

20%

40%

60%

80%

100%

0 20 40 60 80 100 120 140 160 180 200 220 240

Time (seconds)

Per

cen

t P

rog

ram

med


Multihop Bandwidth Management

• Should self-organize into fair, dynamic multihop net• Hidden nodes between each pair of “levels”

– CSMA is not enough

• P[msg-to-base] drops with each hop– Investment in packet increases with distance– need to optimize for low-power fairness!

• RTS/CTS costly (power & BW) Local rate control to approx. fairness

Priority to forwarding, adjust own data rate Additive increase, multiplicative decrease

Listen for retransmission as ack


Example: Multihop Adaptive Transmission Control

B

14

15

18

17

16

Max rate: 4 samples/sec

- rate = 4p

Channel BW ~20 p/s

- cannot expect more than 1/3 thru parent

Monitor number of children (n)

(n) ~ 1/n = ½

p’ = p + (n) on success (echo)

p’ = p *

without rate control, success drops ~½ per hop

node mean p/s (COV)14 0.36 (64%)15 0.56 (14%)16 0.55 (11%)17 0.55 (12%)18 0.39 (11%)


Key Experience

• Really good at building tinyOS subsystems– non-blocking, split-phase event structures

• Internalized the “state of constant change” paradigm

– ex: maintain routing tree by constantly rebuilding it– soft state that is always suspect– simple one-way protocols

• Operating in the aggregate• Simple mechanisms to accomplish large goals

– MAC, ATC

• Out of the box on networking abstractions– Low-power listen, wake-up, statistical sampling, weighted

aggregation

• Understanding of large scale dynamics


Feeding experience back into simulation


Rich set of additional challenges

• Efficient and robust security primitives

• Density independent wake-up, aggregation– sensor => can use radio in ‘analog’ mode

• Resilient aggregators

• Programming support for systems of generalized state machines

• Programming the unstructured aggregate– SPMD, Data Parallel, Query Processing, Tuples

• Understanding how an extreme system is behaving and what is its envelope

– adversarial simulation

• Self-configuring, self-correcting systems


The “Law of Miniaturization”

• Each major generation is increasingly smaller, more deeply interactive, arrives when previous is at its strength

• Vast majority of computing will be small, embedded, devices connected to the physical world

– actually the case today, but...

– not connected to us, the web, or each other – this will change

Time

Integration

Log R

Mainframe

99

Innovation

Minicomputer

Personal ComputerWorkstationServer


Where to go for more?

• http://www.tinyos.net/tos/• Jason Hill, Robert Szewczyk, Alec Woo, Seth Hollar, David

Culler, Kristofer Pister. System architecture directions for network sensors. ASPLOS 2000.

• David E. Culler, Jason Hill, Philip Buonadonna, Robert Szewczyk, and Alec Woo. A Network-Centric Approach to Embedded Software for Tiny Devices. EMSOFT 2001.

http://www.tinyos.net/tos/

Documents

Networks of Tiny Devices Embedded in the Physical World