Manfred Linking the Real World

Copyright 2011 Digital Enterprise Research Institute. All rights reserved.

Digital Enterprise Research Institute www.deri.ie

Enabling networked knowledge

Linking the Real World

Manfred Hauswirth





DERI’s Mission

Enabling & exploiting

Networked Knowledge



About DERI

Founded June 2003 as a CSET (Centre for Science, Engineering and Technology).

Link scientists and engineers / academia and industry

Fundamental research

Development of Irish-based technology companies

Attract industry

Education & outreach

DERI Institute CSET

Commercialization, DAI

EU, EI, direct industry, IRCSET

DERI strategic plan responds to priorities

Local: University focus on Informatics, Physical & Computational Sciences

National: SMART Economy, Program for Government

International: EU Digital Agenda



About DERI

Number one in our core space

Research Publications > 1000

Participation in 17 standardisation groups (W3C, OASIS)

Approx. 140 members from 30 nations

57 PhDs /Masters

42 completed PhDs/Masters

Core Industrial Partners:

MNC’s: Cisco, Avaya, Bel-Labs, Ericsson…

SME’s: Storm, Celtrak, OpenLink……

Research: FBK

Total Research Grants: > €60 million

SFI, EU Framework, Enterprise Ireland, Industry

Currently 18 EU project running

Industry funded projects with Fujitsu Labs Japan, Cisco, Google, Renault, EADS, Fidelity,…



CSET Partners

Key Industry Collaborations

6

http://www.ericsson.com/

http://www.produktive-schweiz.ch/

http://www.cisco.com/web/SI/index.html



DERI Innovation Approach

Research

Excellence

Spin Outs

Open

Source

Prototypes

Commerci-

alisation

Standards

Industry

Collaboration

• Joint projects

• Patents and

Licensing

• DERI Applied

Innovation

• Drupal 7

• Semantic Desktop

• SIREn

• W3C

• OASIS

• SIOC

• VOID, DCAT

• schema.org

• OData

• Atom

• PEPPR

• IVEA

• Seevl

• Sindice.com

• Peracton



The DERI House

eBusiness

Financial Services

Health Care

Life Sciences

eLearning Green &

Sustainable IT

eGovernment

Information

Mining

and Retrieval

Natural

Language

Processing

Reasoning and

Querying

Cloud Data

Management

Knowledge

Discovery

Social Software

Service

Oriented

Architecture

Sensor

Middleware

DERI Applied

Research Commercialisation

Data

Visualisation

and Interaction

Cyber

Security

Cloud

Data

Analy

tics

Linked

Data

Security,

Privacy

& Trust

DERI is designed to provide an integrated solution



Solve which problem?



Interconnected

Universal

All encompassing

Assists humans,

organizations and systems in

problem solving

Enable global and local

collaboration

A Network of Knowledge



A Network of Knowledge

enabling innovation and

increased productivity

Interconnected

Universal

All encompassing

assists humans,

organizations and systems

with problem solving

Linked Data

• Search

• Collaboration

• Information Mining

• Middleware

• Application

Research Domains

• Commercialization


Enabling networked knowledge 12 of 46

Two Key Ingredients

1. RDF – Resource Description Framework

Graph based Data – nodes and arcs

Identifies objects (URIs)

Interlink information (Relationships)

2. Vocabularies (Ontologies)

provide shared understanding of a domain

organise knowledge in a machine-comprehensible way

give an exploitable meaning to the data



Why Graphs and Ontologies?

Cities:Dublin

84421km2

Geo:IslandOfIreland

EU:RepublicOfIreland

Geo:locatedOn

Geo:area

Geo:hasCapital

Geo:hasLargestCity

Wikipedia.org

Gov.ie

EU:RepublicOfIreland

Person:EndaKenny

Gov:hasTaoiseach

Gov:hasDepartment

IE:DepartmentOfFinance



Linked Open Data Cloud

14

2008 2007

2008 2008

2008

2009

2009 2010

14



Linked Data Domains

Over 200 open data sets with more than 25 billion facts,

interlinked by 400 million typed links, doubling every 10 month!

http://lod-cloud.net/

Linking Open Data cloud diagram, by Richard Cyganiak and Anja Jentzsch.

Media

Government

Geo

Publications

User-generated

Life sciences

Cross-domain

US government

UK government

BBC

New York Times

LinkedGeoData

15

BestBuy

Overstock.com

Facebook

Powered

by DERI!






Challenges of Big Data

“90% of the data in the world today

has been created in the last two

years alone” – IBM

The bringing together of a vast

amount of data from public and

private sources […] is what Big Data is

all about,” – IDC



Solutions required for

Management and Integration

Abstraction and Reasoning

Analytics and Visualization

Interaction and Collaboration

Domain Knowledge

and

Integration into a coherent

Framework!

How to exploit Big Data?



Knowledge Dashboard

Networked

Data

Abstraction

Reasoning

Analytics

Visualisation



Invests in

human and social capital

traditional/modern infrastructure

that

fuels sustainable economic development

and high quality of life

while

managing natural resources

through

participatory governance

What is a Smart City?

http://ideas.repec.org/p/dgr/vuarem/2009-48.html



A Smart City removes silos moving

towards a connected digital layer.



http://www.mckinsey.com/mgi/publications/ig_data/pdfs/MGI_big_data_full_report.pdf

Untapped Value Silos’ Value



A Smart City driver of change will be Data.





The Problem



Semantic description of sensors, streams,

events, observations, etc.

“Senso ergo sum” – semantic descriptions

down to the sensor level

Web protocols down to the sensor level

SPARQL-like access to streams and sensors

Infrastructure framework

Goal

Streams are just yet another

form/source of linked data



REST

Keep it simple, Stupid!

LOD

Application

Application := Data + Services



KISS revisited

Application

CQELS SPARQL REST

Linked Data

COAP

Sensors

Virtual

Sensors

Linked Streams



Where are we

right now?



Sensors, streams,

events, observations



Semantic Sensor Networks ontology to describe sensors

and sensor data

Semantic annotations for OGC’s SWE Sensor Model

Language

Motivations

No existing sensor ontology included all the basic concepts

Ease integration of (some) semantics in more spread languages

and standards (specifically SensorML)

W3C SSN XG



Relation to existing

standards



SSN-XG Ontology Structure



SSN-XG Ontology Structure



SSN Application: SPITFIRE

•DUL: DOLCE+DnS Ultralite

•EventF: Event-Model F

•SSN: SSN-XG

•CC: Contextualised-Cognitive

Concepts on sensor network topology and

devices

Concepts on sensor role, events, sensor project

Event

Datasets

Sensor Datasets

LOD Cloud



SPITFIRE Vocabulary

http://www.spitfire-project.eu

coalesenses



Size matters!

• OS + 6LowPAN + CoAP + Semantic description < 48kB?

• Processing power?



Storage requirements



OK, we can

describe sensors

and their data now



Standardisation

Physical: 802.15.4

Network: IEEE 6LoWPAN, ROLL

Service layer:

– IETF CoRE (Constrained RESTful Environments):

CoAP protocol + extensions (very recent)

– Encoding (Extensible XML interchange - EXI, SensorML)

– Ontologies

CoAP = Constrained Application Protocol

IETF draft, http://tools.ietf.org/id/coap

Core proposal + > 17 extensions

RESTful sensor

interfaces



CoAP = HTTP for sensors



Accessing sensors from we browser using HTTP-CoAP

proxying – SPITFIRE Smart Service Proxy (SSP)

CoAP Example



OK, we can access

sensors via RESTful

interfaces now



KISS revisited

Application

CQELS SPARQL REST

Linked Data

COAP

Sensors

Virtual

Sensors

Linked Streams



CQELS

Continuous Query Evaluation over

Linked Streams

Scalable processing model for unified

Linked Stream Data and Linked Open

Data

Combines data pre-processing and an

adaptive cost-based query

optimization algorithm

Experimental evaluation shows great

performance (w.r.t. response time

and scalability)



Query rewriter

Orchestrator

Data transformation

SPA

RQ

L-like

Optimizer

Operator implementations

Access methods

Executor

Query

Execution

Overhead

Black Box Approach



Orchestrator

Query Rewriter

SPARQL

Engine

Data

transformation

ESPER

Query Rewriter

Data

transformation

C-SPA

RQ

L

CSPARQL to SPARQL

CSPARQL to EPSER EPL

Orchestrator

Query Rewriter

Prolog Engine

Data

transformation

EP-SPARQL to Prolog

EP-SPARQL

C-SPARQL

RDF to prolog facts

RDF to Java objects

EP-SPA

RQ

L

EP-SPARQL and C-SPARQL



Adaptive Optimizer


Native Access methods

Adaptive Executor

Query

Adaptive Execution

RDF

dataset

Linked

datastream

White Box Approach



Incoming data will continuously change the costs of query plans

➥Data elements are adaptively routed to processing operators on

equivalent data flows (routing policies)

Enabling adaptivity



Example



Triple-based window operators extracts triples

from RDF stream or dataset that

match a given triple pattern

are valid within in a time window

Relational operators enable employing relational

algebras in the processing model

Streaming operators generate new streams from

output of other operators based on graph

templates

Processing Model:

Operators



Continuous Query Evaluation over

Linked Streams (CQELS)

Adaptive Optimizer


Native Access methods

Adaptive Executor

Query

Adaptive Execution

RDF

dataset

Linked

datastream

CQELS language (an extension of SPARQL 1.1)

Dictionary

SPO index scheme

Ring Triple-based

indices for

windows

Caching and Indexing

Dynamic Routing Policy



Dictionary encoding

Smaller memory for representing triples

Avoid lookup & decoding overhead for numeric RDF nodes

Caching and Indexing

Caching: avoid re-computing of intermediate results of sub-

queries over non-stream data.

Indexing: facilitate faster access on caches and window data.

Dynamic Routing Policy

Incoming data can be evaluated in multiple equivalent data

flows adaptive to changes

Easy & flexible support to implement routing policies

Techniques



Stream pattern Construct new RDF stream

CQELS Language – an extension to SPARQL 1.1

CQELS query language



Conference scenario: combine linked stream from RFID tags

(physical relationships) with DBLP data (social relationships)

Setup

Systems: CQELS vs ETALIS and C-SPARQL

Datasets

– Replayed RFID data from Open Beacon deployments

– Simulated DBLP by SP2Bench

Queries: 5 query templates with different complexities

– Q1: selection,

– Q2: stream joins, Q3,Q4: Stream and non-stream joins

– Q5: aggregation

Experiments

– Single query: generate 10 query instances of each template by varying the constants

– Vary size of DBLP dataset (104-10

7triples)

– Multiple queries: register 2M parallel instances (0≤M≤10)

Experimental setup



CQELS performs faster by orders of magnitude

Experiment results - Query

execution time

Query 1 Query 2 Query 3 Query 4 Query5

CQELS 0.47 3.90 0.51 0.53 21.83

C-SPARQL 332.46 99.84 331.68 395.18 322.64

ETALIS 0.06 27.47 79.95 469.23 160.83

Simple selection: ETALIS performs best

Stream join:

25 times faster than C-SPARQL

8 times faster than ETALIS

Stream and non-stream joins:

>600 times faster than C-SPARQL

150-850 times faster than ETALIS

Aggregation:

15 times faster than C-SPARQL

8 times faster than ETALIS



Scalability: Static data size



Scalability: # of queries



Optimization

Adaptive cost-based query optimization

Inter-query optimization

Smart and dynamic caching

Adaptive caching

Materialized view maintenance for dynamic data

Scalability: clusters and cloud

Next steps



OK, now we can also

process Linked

Streams and integrate

Linked Data efficiently



KISS revisited

Application

CQELS SPARQL REST

Linked Data

COAP

Sensors

Virtual

Sensors

Linked Streams



A single, integrated mobile phone data store for all

applications

Intrinsically integrated with the Web (Linked Data)

RDF-on-the-go

http://rdfonthego.googlecode.com/



RDF-on-the-go

http://rdfonthego.googlecode.com/



Business Card Demo

URI to FOAF file



OK, we can do all

that on mobile

phones too



KISS revisited

Application

CQELS SPARQL REST

Linked Data

COAP

Sensors

Virtual

Sensors

Linked Streams



Global Sensor Networks

Heterogeneous platforms

Heterogeneous data

No common abstractions

Very large scale

Decrease the cost and complexity of sensor

network deployment

Abstraction level

Abstract from data

producers

Data level

Semantic description of

sensors and sensor data

Data integration

Large scale

Distributed query

processing and reasoning



Where are we now?

• Uniform, declarative abstractions

• Simple semantic descriptions

• Supports all major platforms

• Fast and simple deployment

• Plug & play

• Zero-programming

• Efficient query processing

Global Sensor Networks

http://gsn.sourceforge.net/

GSN

SOA

WS



GSN’s view of the world

Sensor network +

base computer =

sensor node

Many sensor nodes

produce a lot of data on

the Internet

Questions:

Deployment

Description

Discovery

Integration

Distributed processing



Central abstraction:

Virtual sensors

A virtual sensor can be any kind of data producer

a real sensor, a wireless camera, a mobile phone, etc.

a combination of other local or

remote virtual sensors

Specification

simple semantic descriptions of sensors and data streams

declarative SQL-based specification of the data stream processing

functional properties related to stream quality management, etc.

N input streams

processing

1 (structured) output stream



<virtual-sensor name="room-monitor" priority="11">

<addressing>

<predicate key="geographical">BC143</predicate>

<predicate key="usage">room monitoring</predicate>

</addressing>

<life-cycle pool-size="10" />

<storage permanent="true" history-size="10h" />

<output-structure>

<field name="image" type="binary:jpeg" />

<field name="temp" type="int" />

</output-structure>

<input-streams>

<input-stream name="cam">

<stream-source alias="cam" storage-size="1"

disconnect-buffer-size="10">

<address wrapper="remote">

<predicate key="geographical">BC143</predicate>

<predicate key="type">Camera</predicate>

</address>

<query>select * from WRAPPER</query>

</stream-source>

<stream-source alias="temperature1“

storage-size="1m"



<predicate key="type">temperature</predicate>

<predicate key="geographical">

BC143-N

</predicate>

</address>

Virtual sensor definition:

XML + SQL

70 of 58

<query>

select AVG(temp1) as T1 from WRAPPER

</query>

</stream-source>

<stream-source alias="temperature2“

storage-size="1m"



<predicate key="type">

temperature

</predicate>

<predicate key="geographical">

BC143-S

</predicate>

</address>

<query>

select AVG(temp2) as T2

from WRAPPER

</query>

</stream-source>

<query>

select cam.picture as image, temperature.T1

as temp

from cam, temperature1

where temperature1.T1 > 30 AND

temperature1.T1 = temperature2.T2

</query>

</input-stream>

</input-streams>

</virtual-sensor>

Input stream 3:

Temperature

Input stream 2:

Temperature

Input stream 1:

Camera images

Meta-data

System resources

to assign

Structure / data type

of output stream

Query over input

streams to produce

output stream of

the Virtual Sensor



HTTP Generic Wrapper

HTTP GET or POST requests

Serial Forwarder Wrapper

TinyOS compatible motes

USB Camera Wrapper

Local USB connection

Bluetooth Wrapper

MAC and RFCOMM Bluetooth

GPSD Wrapper

More than 60 NMEA-compliant GPS devices

Accessing sensors:

Wrappers

Generic UDP Wrapper

UDP connections

Generic Serial Wrapper

Local RS-232 connections

TI-RFID Wrapper

Texas Instruments Series 6000 S6700 multi-protocol RFID readers

Generic RSS/XML Wrapper

COAP Wrapper

RESTful interface to sensors

Contiki, Coalesenses



Coding efforts

50 RFID reader (TI)

60 Wireless camera (HTTP)

300 Wired camera

180 Generic serial

45 Generic UDP

75 WiseNode

120 TinyOS

Lines of code Wrapper type



Selected Features

SafeStorage

- Safe data backups

Workflow Editor

- Web-based design of

Virtual Sensors



Plug and Play: Zero

Programming

An IEEE 1451-compliant sensor provides a Transducer

Electronic Data Sheet (TEDS) which is stored inside the sensor

TEDS provides a simple semantic description of the sensor

the sensor's properties and measurement characteristic

GSN uses the TEDS self-description feature for dynamic

generation and deployment of virtual sensor descriptions

Next step: store queries not only data in TEDS or RFID tags

New level of data processing in terms of flexibility



Does it really work?



Experimental setup

5 desktop PCs

Pentium 4, 3.2GHz with 1MB

cache, 1GB memory, 100Mbit

Ethernet, Debian 3.1

Linux kernel 2.4.27, MySQL

5.18

SN-1: 10 Mica2 motes with light and

temperature sensors, packet size 15

Bytes, TinyOS

SN-2: 8 Mica2 motes with light,

temperature, acceleration, and

sound sensors, packet size 100

Bytes, TinyOS

SN-3: 4 Shockfish Tiny-Nodes with a

light and two temperature sensors

packet size 29 Bytes, TinyOS

SN-4: 15 wireless 8002.11b cameras

(AXIS 206W), 640x480 JPEG, 5 with

16kB average image size, 5 with

32kB, 5 with 75kB

SN-5: TI Series 6000 S6700 multi-

protocol RFID reader with three

different kind of tags (up to 8KB of

data)



Experimental setup

2 1.8 GHz Centrino laptops with

1GB memory as observers

Each ran up to 250 lightweight

GSN instances.

Each instance produced random

queries with varying table names,

varying filtering condition

complexity, and varying

configuration parameters

3 filtering predicates in the

WHERE clause on average, using

random history sizes from 1

second up to 30 minutes and

uniformly distributed random

sampling rates (seconds) [0.01, 1]

Motes produce random bursts (1-

100 data items) with 25%

probability



Processing time per

client



Scalability in the number of

clients

79 of 58



OK, now we also have a

nice middleware /

database abstraction

for any sensor type



KISS revisited

Application

CQELS SPARQL REST

Linked Data

COAP

Sensors

Virtual

Sensors

Linked Streams



Putting it all together

Semantics / Linked Data / Real-Time / Streams / GIS



Linking the Real World

QR code points to FOAF profile

Associated RFID tag

Associated mobile

Available Not Available

RFID tag of person is identified

FOAF info is displayed

Availability based on haptic

phone interface

Position of other people

In the demo room

People in DERI scan QR code

with mobile to check into room

Position of people in DERI



Behind the Scenes

Demo roomRFID base station

Screen

where.deri.ie

REST

(position, availability)

REST

(availability)

www.deri.ie

gsn.deri.ie

REST

(position)REST

(position)

REST

(FOAF)

REST

(FOAF)



OK, now let’s make

it bigger and

general-purpose!



CQELS SPARQL REST

Linked Data

COAP

Sensors

Virtual

Sensors

Linked Streams

KISS re-re-visited

Application

Middleware



Linked Sensor

Middleware

Live data

Middleware for the semantic integration of live

real-world data

SPARQL endpoint for querying unified Linked Stream Data

and Linked Data

Sensor mashup composer

Wrappers for collecting and enriching

real-time / stream (sensor) data

Web interface for data

exploration, annotation and

visualisation

Mobile phone applications



LSM Architecture



:dublinAirport

:aHumidity

:aTemperature

:weatherStation

:latestWeather

:readings

:humidValue :tempValue

“18”^xsd:fl

oat “Celcius”

“60”^xsd:fl

oat “%”

ssn:featureOfInterest

ssn:observedBy

ssn:observes

ssn:observes

ssn:isPropertyOf ssn:isPropertyOf

ssm:observedPropery ssm:observedPropery

ssm:value ssm:value ssm:unit ssm:unit

ssn:hasValue ssn:hasValue

ssn:observationResult

Sensor metadata

Stream data snapshot at 2011-07-08T21:32:52

Linked Stream Model



Webcams:24570

Traffic:469 (London,

Ohio)

Roadactivity:575 (Ohio)

Flights: >1000

Railway stations:251

(London)

Bike hire:421(London)

Weather: 82365

Snowfall: 2639

Snow depth: 377

Sea level: 45

Radar:1

Satellite: 12

Over 110,000 live data

sources

… and growing!!!



Deployment

Stream Sources

Data Bus

CQELS

(Stream

Proc.)

Virtuoso Web

server



Mobile Applications

LSM

SPARQL,CQELS

SPARQL-XML/RDF

100-200 lines of code



LSM: Live flights info



LSM: Live train info



LSM: Live traffic info



A demo is worth a thousand words

http://lsm.deri.ie



LSM Example

Application



LSM Example

Application



LSM Example

Application



OK, what’s next?



Across Research Areas

Internet of Things

Cloud Mobile

Semantic Web Linked Data



Strategic Application

Domains

Enterprise Environments

Telehealth

Smart Cities



Sensor data management in the Cloud

Sensor data annotation and sharing (portals,

community-based)

Social network analysis (online, mobile, real-world)

Social and opportunistic sensing (mobile phone)

Distributed query processing

Integrating business processes and sensors

Upcoming Research

Areas

10

3



Conclusions

“Linking the Real World”

requires cross-domain /

cross-layer research

Non-trivial, open research

questions knowledge

management, Semantic

Web, databases, Cloud,

sensor networks, etc.

Running systems and

experiments!



Danh Le-Phuoc

Anh Le Tuan

Myriam Leggieri

Hoan Nguyen Mau Quoc

Josiane Xavier Parreira

Martin Serrano

Christian von der Weth

Acknowledgements