Outdoor Distributed Computing*

Outdoor Distributed Outdoor Distributed Computing*Computing*

Cristian BorceaCristian BorceaDepartment of Computer ScienceDepartment of Computer Science

New Jersey Institute of TechnologyNew Jersey Institute of Technology

* Work done at Rutgers prior to September 2004

22

Computers Embedded Computers Embedded EverywhereEverywhere

33

Distributed object Distributed object tracking over a largetracking over a large

geographical areageographical area

Cars collaborating Cars collaborating for a safer and for a safer and

more fluid trafficmore fluid traffic

Outdoor Distributed ApplicationsOutdoor Distributed Applications

HowHow to write distributed programs over outdoor to write distributed programs over outdoor networks of embedded systems?networks of embedded systems?

44

Traditional (Indoor) Distributed Traditional (Indoor) Distributed ComputingComputing

Commonly, computation is distributed for Commonly, computation is distributed for performance or fault tolerance performance or fault tolerance

Relatively easy to programRelatively easy to program– Nodes are computationally equivalentNodes are computationally equivalent– Networking is robust and has acceptable delaysNetworking is robust and has acceptable delays– Configuration is stableConfiguration is stable

55

Outdoor Distributed ComputingOutdoor Distributed Computing

Computation is physically distributedComputation is physically distributed– Nodes distributed across the physical spaceNodes distributed across the physical space– Location-aware computingLocation-aware computing

Hard to programHard to program– Nodes are functionally heterogeneous and Nodes are functionally heterogeneous and

potentially mobilepotentially mobile– Ad hoc wireless networks of embedded systemsAd hoc wireless networks of embedded systems– Volatile configurationsVolatile configurations

66

Outdoor Programming ExampleOutdoor Programming Example

Left HillLeft Hill Right HillRight Hill

Mobile robotwith camera

Motion Sensor

Intrusion detection across the hills using motion Intrusion detection across the hills using motion sensors and autonomous mobile robots with sensors and autonomous mobile robots with cameras cameras

Number and location of systems are unknownNumber and location of systems are unknown Configuration is not stable over timeConfiguration is not stable over time

– Intrusions can appear randomlyIntrusions can appear randomly– Robots can moveRobots can move

77

Traditional Distributed Traditional Distributed Computing Computing

Does Not Work Well OutdoorsDoes Not Work Well Outdoors End-to-end data transfers may hardly completeEnd-to-end data transfers may hardly complete Fixed address naming and routing (e.g., IP) are Fixed address naming and routing (e.g., IP) are

too rigidtoo rigid Difficult to deploy new applications in existing Difficult to deploy new applications in existing

networksnetworks Outdoor distributed computing requires novel Outdoor distributed computing requires novel

programming models and system architecturesprogramming models and system architectures

88

OutlineOutline MotivationMotivation Spatial Programming ModelSpatial Programming Model Smart Messages System ArchitectureSmart Messages System Architecture Implementation DetailsImplementation Details Experimental ResultsExperimental Results ConclusionsConclusions

99

Traditional (Indoor) ProgrammingTraditional (Indoor) Programming

Programs access data through variablesPrograms access data through variables Variables mapped to physical memory locationsVariables mapped to physical memory locations Page Table and OS guarantee reference consistency Page Table and OS guarantee reference consistency Access time has an (acceptable) upper boundAccess time has an (acceptable) upper bound

ProgramProgram

Virtual Address SpaceVirtual Address Space

Page Table + OSPage Table + OS

Physical Memory

Variable accessVariable access

1010

From Indoor to Outdoor From Indoor to Outdoor ComputingComputing

Virtual Address SpaceVirtual Address Space Space RegionSpace Region

VariablesVariables Spatial ReferencesSpatial References

Variables mapped to Variables mapped to physical memoryphysical memory

Spatial references mappedSpatial references mappedto systems embedded in to systems embedded in the physical spacethe physical space

Reference consistencyReference consistency

Bounded access timeBounded access time

??

??

1111

Spatial Programming (SP) at a Spatial Programming (SP) at a GlanceGlance

Provides a Provides a virtual name spacevirtual name space over outdoor over outdoor networks of embedded systemsnetworks of embedded systems

Embedded systems named by their locations Embedded systems named by their locations and propertiesand properties

Runtime system takes care of name resolution, Runtime system takes care of name resolution, reference consistency, and networking aspectsreference consistency, and networking aspects

Implementation on top of Smart Messages: SP Implementation on top of Smart Messages: SP applications execute, sequentially, on each applications execute, sequentially, on each system referenced in their codesystem referenced in their code

1212

radiusradius

Hill = new Space({lat, long}, radius);Hill = new Space({lat, long}, radius);

{lat,long}{lat,long}

Space RegionsSpace Regions

Virtual representation of a physical spaceVirtual representation of a physical space Similar to a virtual address space in a Similar to a virtual address space in a conventional conventional computer systemcomputer system Defined statically or dynamicallyDefined statically or dynamically

1313

Spatial ReferencesSpatial References

Defined as Defined as {space:property}{space:property} pairs pairs Virtual names for embedded systemsVirtual names for embedded systems Similar to variables in conventional programmingSimilar to variables in conventional programming Indexes used to distinguish among similar Indexes used to distinguish among similar systems systems in the same space regionin the same space region

{Hill:robot[0]}{Hill:robot[0]}

{Hill:robot[1]}{Hill:robot[1]}{Hill:motion[0]}{Hill:motion[0]}

HillHill

r2r2 r7r7

m5m5

1414








??

??

1515

Reference ConsistencyReference Consistency

At first access, a spatial reference is mapped to At first access, a spatial reference is mapped to an embedded system located in the specified an embedded system located in the specified space space

Mappings maintained in per-application Mapping Mappings maintained in per-application Mapping Table (MT) – similar to a page tableTable (MT) – similar to a page table

Subsequent accesses to the same spatial Subsequent accesses to the same spatial reference will reach the same system (using MT) reference will reach the same system (using MT) as long as it is located in the same space region as long as it is located in the same space region

{space, property, index}{space, property, index}{unique_address, location}{unique_address, location}

1616

Reference Consistency ExampleReference Consistency Example


{Left_Hill:robot[0]}.move = ON;{Left_Hill:robot[0]}.move = ON;

r2r2 r7r7r5r5

{Left_Hill:robot[0]}.move = OFF;{Left_Hill:robot[0]}.move = OFF;

1717

Space CastingSpace Casting

{Left_Hill:robot[0]}{Left_Hill:robot[0]}


{Right_Hill:(Left_Hill:robot[0])}{Right_Hill:(Left_Hill:robot[0])}

r7r7

1818








Mapping TableMapping Table

??

1919

Bounding the Access TimeBounding the Access Time How to bound the time to access a spatial How to bound the time to access a spatial

reference?reference?– Systems may move, go out of space, or disappearSystems may move, go out of space, or disappear

Solution: associate an explicit timeout with Solution: associate an explicit timeout with each spatial reference accesseach spatial reference access

try{try{ {Hill:robot[0], timeout}.{Hill:robot[0], timeout}.move = ON;move = ON;}catch(TimeoutException e){}catch(TimeoutException e){ // the programmer decides the next action// the programmer decides the next action}}

2020

for(i=0; i<1000; i++)for(i=0; i<1000; i++) try{try{ if (if ({Left_Hill:motion[i], timeout}.{Left_Hill:motion[i], timeout}.detect > Max_motion)detect > Max_motion) Max_motion = Max_motion = {Left_Hill:motion[i], timeout}{Left_Hill:motion[i], timeout}.detect;.detect; Max_id = i;Max_id = i; }catch(TimeoutException e) }catch(TimeoutException e) break;break; intrusionSpace = rangeOf(intrusionSpace = rangeOf({Left_Hill:motion[Max_id].location}, {Left_Hill:motion[Max_id].location}, Range); Range); {intrusionSpace:robot[0]}.camera = ON;{intrusionSpace:robot[0]}.camera = ON; {intrusionSpace:robot[0]}.focus = {intrusionSpace:robot[0]}.focus = {Left_Hill:motion[Max_id].location}{Left_Hill:motion[Max_id].location};;

Spatial Programming Spatial Programming ExampleExample

- Find the sensor - Find the sensor that detectedthat detectedthe “strongest” the “strongest” motion on Left Hillmotion on Left Hill- Turn on a camera - Turn on a camera in the proximity in the proximity of this sensorof this sensor


2121


2222

““Dumb” Messages vs. “Smart” Dumb” Messages vs. “Smart” MessagesMessages

Data migrationData migration

Lunch:Lunch:

AppetizerAppetizerEntreeEntreeDessertDessert

Execution migrationExecution migration

2323

Smart Messages at a GlanceSmart Messages at a Glance

User-defined distributed applicationsUser-defined distributed applications Composed of code bricks, data bricks, and Composed of code bricks, data bricks, and

execution control stateexecution control state Execute on nodes of interest named by Execute on nodes of interest named by

propertiesproperties Migrate between nodes of interestMigrate between nodes of interest Self-route at every node in the path during Self-route at every node in the path during

migrationsmigrations

2424

Cooperative Node ArchitectureCooperative Node Architecture

NetworkNetwork

SMSMPlatformPlatform

AdmissionAdmissionManagerManager NetworkNetworkSMSM SMSM

Operating System & I/OOperating System & I/O

CodeCodeCacheCache

VirtualVirtualMachineMachine

Tag Tag SpaceSpace

SM ReadySM ReadyQueueQueue

AuthorizationAuthorization

2525

SM Execution at a NodeSM Execution at a Node

Takes place over a virtual machineTakes place over a virtual machine Non-preemptive, but time boundedNon-preemptive, but time bounded Ends with a migration, or terminatesEnds with a migration, or terminates During execution, SMs canDuring execution, SMs can

– Spawn new SMsSpawn new SMs– Create smaller SMs out of their code and data bricksCreate smaller SMs out of their code and data bricks– Access the tag spaceAccess the tag space– Block on a tag to be updated (update-based Block on a tag to be updated (update-based

synchronization)synchronization)

2626

Tag SpaceTag Space Collection of application tags and I/O tagsCollection of application tags and I/O tags

– Essentially, tags are Essentially, tags are (name, value)(name, value) pairs pairs Application tags: persistent memory across SM Application tags: persistent memory across SM

executionsexecutions I/O tags: access to operating system and I/O I/O tags: access to operating system and I/O

subsystemsubsystem Tags used forTags used for

– Content-based naming Content-based naming migrate(tag)migrate(tag)– Inter-SM communication Inter-SM communication write(tag, data), read(tag)write(tag, data), read(tag)– Synchronization Synchronization block(tag, timeout)block(tag, timeout)– I/O access I/O access read(temperature)read(temperature)

2727

Protection Domains for Tag Protection Domains for Tag SpaceSpace

Owner: SM that creates the tagOwner: SM that creates the tag Family: all SMs having a common ancestor Family: all SMs having a common ancestor

with the SM that owns the tagwith the SM that owns the tag Origin: all SMs created on the same node as Origin: all SMs created on the same node as

the family originator of the tag owner the family originator of the tag owner Code: all SMs carrying a specific code brickCode: all SMs carrying a specific code brick Others: all the other SMsOthers: all the other SMs

FamilyFamilyCodeCodeOriginOrigin

Others

OwnerOwner

2828

Access Control Example Access Control Example (Code-based protection domain) (Code-based protection domain)

Node5

Node3

Node4

SM2SM2

SM2C2 Cr

Node1

Node2Code

Bricks

TagOwner = SM1

[Hash(Cr), RW]

SM1C1 Cr

SM1 SM1SM3

C3C4

Access permission granted for SM2Access permission denied for SM3

Cr Same routing used by SM1 and SM2

2929

SM AdmissionSM Admission Ensures progress for all SMs in the networkEnsures progress for all SMs in the network Prevents SMs from migrating to nodes that Prevents SMs from migrating to nodes that

cannot provide enough resourcescannot provide enough resources SMs specify lower bounds for resource SMs specify lower bounds for resource

requirements (e.g., memory, bandwidth)requirements (e.g., memory, bandwidth) SMs accepted if the node can satisfy these SMs accepted if the node can satisfy these

requirementsrequirements More resources can be granted according to More resources can be granted according to

admission policyadmission policy– If not granted, SMs are allowed to migrateIf not granted, SMs are allowed to migrate

3030

Application ExampleApplication Example

n=0n=0while (n<NumTaxis)while (n<NumTaxis) migrate(Taxi);migrate(Taxi); if (readTag(Available))if (readTag(Available)) writeTag(Available, false);writeTag(Available, false); writeTag(Location, myLocation);writeTag(Location, myLocation); n++;n++;

n=n=00

TaxiTaxi TaxiTaxi

n=0n=0 n=0n=0 n=1n=1 n=1n=1 n=2n=2

needneed2 taxis2 taxis

data brickdata brickapplication application code brickcode brickrouting routing code brickcode brick

SMSM

3131

migrate()migrate() – migrates application to next node of interest– names nodes by tags– implements routing algorithm

sys_migrate()sys_migrate()– one hop migration– used by migrate to implement routing

SM MigrationSM Migrationmigrate(Taxi)migrate(Taxi)

sys_migrate(2)sys_migrate(2) sys_migrate(3)sys_migrate(3) sys_migrate(4)sys_migrate(4)

TaxiTaxiTaxiTaxi

11 22 33 44

3232

NetworkNetwork

Routing ExampleRouting Example 1 2 i

RouteToTaxi = 2

TaxiRouteToTaxi = ?RouteToTaxi = j

migrate(Taxi){ while(readTag(Taxi) == null) if (readTag(RouteToTaxi)) sys_migrate(readTag(RouteToTaxi)); else create_SM(DiscoverySM, Taxi); createTag(RouteToTaxi, lifetime, null); block_SM(RouteToTaxi, timeout);}

3333

SM Self-RoutingSM Self-Routing SMs carry the routing and execute it at each nodeSMs carry the routing and execute it at each node Routing information stored in tag spaceRouting information stored in tag space SMs control their routingSMs control their routing

– Select routing algorithm (migrate primitive)Select routing algorithm (migrate primitive) Multiple library implementationsMultiple library implementations Implement a new oneImplement a new one

– Change routing algorithm during execution in response Change routing algorithm during execution in response toto

Adverse network conditionsAdverse network conditions Application’s requirementsApplication’s requirements

3434

Self-Routing Simulation ResultsSelf-Routing Simulation Results

starting nodestarting nodenode of interestother nodeother node

Geographic + On-Demand Routing performs better than simple on-demand routing 25%-40% decrease in completion time 62%-92% reduction in amount of bytes sent in the network

- Less energy and bandwidth consumption

0

0.5

1

1.5

2

2.5

3

0 100 200 300 400 500 600 700 800

Region Radius (meters)

Com

plet

ion

Tim

e (s

ec)

On-Demand Routing Geographic+On-Demand Routing

0

500

1000

1500

2000

2500

3000

3500

4000

4500

0 100 200 300 400 500 600 700 800

Region Radius (meters)

Byte

s Se

nt in

the

Net

wor

k (K

Byt

es)

On-Demand Routing Geographic+On-Demand Routing

3535


3636

Spatial Programming Spatial Programming Implementation Using Smart Implementation Using Smart

Messages Messages SP application translates into an SMSP application translates into an SM

– Spatial reference access translates into an SM Spatial reference access translates into an SM migration to the mapped nodemigration to the mapped node

– Embedded system properties: TagsEmbedded system properties: Tags SM self-routes using geographical routing and SM self-routes using geographical routing and

content-based routing content-based routing Reference consistencyReference consistency− Unique addresses (stored in mapping table) are Unique addresses (stored in mapping table) are

unique tags created at nodesunique tags created at nodes− SM carries the mapping tableSM carries the mapping table

3737

Max_motion = {Left_Hill:motion[1], timeout}.detect;Max_motion = {Left_Hill:motion[1], timeout}.detect;

SP to SM Translation: ExampleSP to SM Translation: Example

Spatial Reference Spatial Reference AccessAccess

Smart MessageSmart Message

ret = migrate_geo(location, timeout);ret = migrate_geo(location, timeout);if (ret == LocationUnreachable)if (ret == LocationUnreachable) ret = migrate_tag(yU78GH5, timeout);ret = migrate_tag(yU78GH5, timeout);if ((ret == OK) && (location == Left_Hill))if ((ret == OK) && (location == Left_Hill)) return readTag(detect);return readTag(detect);else throw TimeoutException else throw TimeoutException

{Left_Hill,motion,1}{Left_Hill,motion,1} {yU78GH5,location}{yU78GH5,location}MappingMappingTableTable

CodeCodeBrickBrick


3838

SM Prototype ImplementationSM Prototype Implementation Modified version of Sun’s Java K Virtual Modified version of Sun’s Java K Virtual

MachineMachine– Small memory footprint (160KB)Small memory footprint (160KB)

SM and tag space primitives implemented SM and tag space primitives implemented inside virtual machine as native methods inside virtual machine as native methods (efficiency)(efficiency)

I/O tags: GPS location, neighbor discovery, I/O tags: GPS location, neighbor discovery, image capture, light sensor, system statusimage capture, light sensor, system status

Ad hoc networks ofHP iPAQ PDAs running Linux

3939

Lightweight Migration Lightweight Migration Traditional process migration difficultTraditional process migration difficult

– Strong coupling between execution entity and hostStrong coupling between execution entity and host– Needs to take care of operating system state (e.g., Needs to take care of operating system state (e.g.,

open sockets, file descriptors)open sockets, file descriptors) Tag space decouples the SM execution state Tag space decouples the SM execution state

from the operating system statefrom the operating system state SM migration transfers onlySM migration transfers only

– Data bricks explicitly specified by programmerData bricks explicitly specified by programmer– Minimal execution control state required to Minimal execution control state required to

resume the SM at the next instruction (e.g., resume the SM at the next instruction (e.g., instruction pointer, operand stack pointer) instruction pointer, operand stack pointer)

4040


4141

Micro-benchmark Results Micro-benchmark Results

0

1

2

3

4

5

6

7

2 4 8 16Size (KBytes)

Tme

(ms)

Java object arrayJava int array

Cost of data serializationCost of data serialization

0

5

10

15

20

25

30

1224 2236 4294 8383Code Size (Bytes)

Tim

e (m

s)

Code UncachedCode Cached

Cost of single hop migration

4242

Experimental Results for Simple Experimental Results for Simple Routing AlgorithmsRouting Algorithms

Completion Time

Routing algorithmCode not cached (ms)Code cached (ms)GeographicOn-demand

415.6 126.6506.6 314.7

user nodenode of interestintermediate node

8 HP iPAQs running Linux and using 8 HP iPAQs running Linux and using IEEE 802.11 for wireless communicationIEEE 802.11 for wireless communication

4343

SP Application: Intrusion SP Application: Intrusion DetectionDetection10 HP iPAQs with 802.11 cards and GPS devices10 HP iPAQs with 802.11 cards and GPS devices

user nodeuser nodelight sensorlight sensorcamera nodecamera noderegular noderegular node

monitoredmonitoredspacespace

4444

ConclusionsConclusions Spatial Programming makes outdoor Spatial Programming makes outdoor

distributed computing simpledistributed computing simple– Volatility, mobility, configuration dynamics, ad hoc Volatility, mobility, configuration dynamics, ad hoc

networking are hidden from programmernetworking are hidden from programmer Implementation on top of Smart Messages Implementation on top of Smart Messages

– Easy to deploy new applications in the networkEasy to deploy new applications in the network– Quick adaptation to highly dynamic network Quick adaptation to highly dynamic network

configurationsconfigurations Acknowledgments: Liviu Iftode, Ulrich Acknowledgments: Liviu Iftode, Ulrich

Kremer, Chalermek Intanagonwiwat, Porlin Kremer, Chalermek Intanagonwiwat, Porlin KangKang

4545

Other projects based on Smart Other projects based on Smart MessagesMessages

Portable Smart Messages: work over Portable Smart Messages: work over unmodified JVMs using bytecode unmodified JVMs using bytecode instrumentation (done by Nishkam)instrumentation (done by Nishkam)– Tested on iPAQs, Ericsson P900, and Nokia

Communicator Context-aware Migratory Services in Ad Hoc

Networks (Oriana will present it after this talk) Spatial Views: A location-aware, declarative

programming language (Uli Kremer will present it on Friday)

4646

Two of my current projectsTwo of my current projects Intelligent distributed transportation systems

– Sponsored by NSF, collaboration with Rutgers University (Pravin will talk about it in the afternoon)

– Design a vehicular network architecture and build a prototype for distributed computing over vehicular networks

SmartCampus– Sponsored by NSF, joint work with IS & ECE

departments at NJIT– Investigate mobile computing applications, protocols,

middleware over a campus-wide, location-aware, mobile community test-bed (100s of users with mobile devices)

4747

Thank you!Thank you!

http://www.cs.njit.edu/~borcea/http://www.cs.njit.edu/~borcea/

Documents

Outdoor Distributed Computing*