Bridging the Gap between Humans and the Physical World

Bridging the Gap between Humans and the Physical World

A Step Toward Reducing Energy Consumption and Increasing Comfort

Dr. Fred Jiang / 姜小凡In collaboration with Mike Liang, Jeff Hsu, Caiquan Liu, Jie Liu, and Feng Zhao

Mobile and Sensing Systems GroupMicrosoft Research Asia

National Taiwan University / 2011-4-1

MotivationWe interact with our environment in very limited ways

Simple time series data collectionActuation is limited because of lack of uniform interfacePhysical objects cannot proactively reach out to us

Result: waste in energy and suboptimal comfort

Example 1: environment (physical objects) aware of who is acting on it, collect usage / energy data in real-time (personal energy footprint)Example 2: alert me if something is not normalExample 3: objects interacting with objects

Related Work

Ambient intelligenceRoomwareSmart objectsLocalizationX10Beaver monitoringUbicomp literature

We Propose

Bi-direction interaction and information flow between humans and physical objectsFor humans to have real-time finer-grained knowledge and control over physical objectsEnvironment to be aware of humans, and make intelligent decisions, proactively and reactivelyBridging the gap between humans and their environmentOur vision of IoT (it’s not about simply putting a networking interface on everything)

Three Primary Components

Precise indoor localization primitiveReliable IPv6 based networkingUniform interface for abstracting services of physical objects

Virtual Overlay

Entrance to 4th floor

name: fridgestates: on : true temp : 5 door : opencommands: turn_on() turn_off()events: notifyDorOpn notifyPwrUsgchildren: []

Indoor Localization

Previous workWhat we needChallenges

Localization Options

Wifi radioBT radioInertial sensorOptical (Kinect)RFIDMagnetic (PKE)

Magneto-Induction

Slide courtesy of Andrew Markham (http://www.comlab.ox.ac.uk/people/andrew.markham/)

Magneto-Induction

Slide courtesy of Andrew Markham (http://www.comlab.ox.ac.uk/people/andrew.markham/)

Magneto-Induction

Slide courtesy of Andrew Markham (http://www.comlab.ox.ac.uk/people/andrew.markham/)

Challenges for Indoor

4m x 1m antenna -> 5m^3 trackingWe don’t have that much space!We use a 8cm x 1.5cm transmitter antennaAnd 3D coil IC (2cm x 1cm) as receiving antenna

Power supply issuesOutdoor vs indoorInterference issuesPhone magnetometer (digital compass)

Microsoft LivePulse

Mains Power

AD/DC/DC Converstion90-240VAC -> 12VDC -> 5VDC -> 3.3VDC

Mechanical Relay

Hall-effectSensor

MI MOSFET Driver

125kHz MI Transmitting Antenna

CC430 SoCEnergy Meter

IC

900MHzPCB

Antenna

Microsoft LivePulse• Precise indoor

localization

Range: ~ 3m omnidirectional

~10cm dropoff zone

Consistent over timeRobust against interference and multipathBlock by metal

• Wireless energy monitoring

Microsoft LiveLink

3D MI Receiving Antenna

Wakeup

Networking

RequirementsReliableGlobally addressableScalableLow power

Radio Consideration

• 802.15.4– 2.4GHz– 863 MHz to 928MHz– 100-150m range– US

• 802.15.4c (c for china)– 779-787 MHz– 250m range

• Core module choices– 60USD Epic core– 20USD SuRF core

• Energy

Networking Overview

One-hopIPv6 communication over 802.15.4c

JSON / RESTful services over Internet

Repository of virtual objects

Virtual Object Abstraction Layerapartment

bedroom office bathroom hallway

desk desktop room portable lamp computer lamp heater

coffee laptop piano pinballmachine computer keyboard machine

LCD computer monitor tower

• Hierarchical• Event based

Object Representation

StatesE.g. On/off

CommandsE.g. turn on / read power / set temp / count people

EventsE.g. door opened / light-off / over-threshold

ChildrenPtr to children objects

name: officestates: lightOn? : True doorOpen? : False roomOccupied? : True roomTemp? : 30C roomEnergy? : 200W commands: toggleLight numOccRoom lastTimeDoorOpened

events:

notifyDoorOpen notifyNewOccupant notifyHourlyEnergyUsechildren [ipv6]: coffeeMachine deskLamp laptopComputer desktopComputer pianoKeyboard roomLamp pinballMachine portableHeater

1. DISCOVERY_REQ

2. DISCOVERY_REPLY

to http://<ipv6OfOffice>

3. EVENT_SUBSCRIPTION [notifyNewOccupant]

4. NOTIFICATION

“let me know if someone enters room”

“someone just entered your room”

Open Source Standard

JSON is the object encodingSOAP / WSDL / Zigbee SE Profile – Nooo

REST/HTTP is the interchange schemeHTTP_GET / HTTP_GET / HTTP_RESPONSE

IPv6Interoperate with existing IP devices without understanding additional protocols

Applications

Personal energy footprintSocial networkingHealth care

Related EffortsSmart meters (via utilities)

Enables time-of-day billingGoogle PowerMeter

Visualization of whole-house usageDefining a API for energy data

Microsoft HohmComplimentary work (help learn signatures)House-level / NILM / can benefit from DB of appliance signatures

In the right direction, but not enoughOnly at the whole-house levelLack of actuation / control

Source: http://www.microsoft-hohm.com

Source: X. Jiang Dissertation

Individualized Energy Feedback and Control

• Real-time energy apportionment– Individualized energy

accounting / billing– Energy map / trail of occupants

• Alerts and abnormally detection– Via cellphone– Remote alerts

• Control and actuation– Automated control of devices– Scheduling

An Energy-Centric Ecosystem – Social Network of Energy

Social network based on real-time individual energy usagesWindows 7 GadgetWindows Mobile 7 AppFacebook appReward systems

Building managers have fine-grain control and view

mobile application

web application - synergy

Social Network of Energy

LocalizationLocation Status updates, kind of like Foursquare Checkins.• “Mike is now in the coffee room.”

ApportionmentNow that we have localization primitive, we can specifically attribute the use of public energy to individual users. Very useful.• “Attribute the last 3 minutes of coffee energy usage to Jeff.”Add this to personal monitoring of private appliances, and we now have a complete carbon footprint.

web application dashboard - map

Social Network of Energy

Map of User’s owned areas…and where energy usage is taking place the mostSocial Interaction

Compare Energy Usage of Specific Appliances, or aggregateGames• User VS. User [One-Month Challenge, percent

improvement?]• Team VS. Team

web application dashboard - energy

Social Network of Energy

Add Appliances [Home and Work]Energy Usage Trends

Day to Day Comparisons in a tableEnergy Usage Graphs [Compare different devices]

Email Summary of Energy Usage to User [each day, or week, or month]

web application – user profile

Social Networking “Wine Party” App

Frequently, it is very difficult to meet people you click with at social gatherings.Ex. You are at a company party. You are bored. You pull out your mobile phone. A map of the room you are in shows up, and there are markers on the map. Click on a marker, and you can see that person’s interests, details, hobbies, etc.See twitter feed, facebook profile, etc.When someone who match your interest pass by, your phone vibrates to alert you.

Health Care

Room is aware of status of elderly occupants 1. phone insert trigger / event2. object representing the room 3. room send a signal to alert

Conclusion• Bi-direction interaction

between humans and physical objects

• For humans to have real-time fine-grained knowledge and control over physical objects

• Environment to be aware of humans, and make intelligent decisions, proactively and reactively

• Precise indoor localization primitive

• Reliable IPv6 based networking

• Uniform virtual representation of physical objects

Thank you

Questions?

Feel free to contact me at fxjiang@microsoft.com http://research.microsoft.com/people/fxjiang