Private Public Partnership Project (PPP) - FIWARE · Private Public Partnership Project (PPP) Large-scale Integrated Project (IP) D.14.1.1: Contribution to FIWARE Reference Architecture

Private Public Partnership Project (PPP)

Large-scale Integrated Project (IP)

D.14.1.1: Contribution to FIWARE Reference Architecture (IoT)

Project acronym: FI-Core Project full title: Future Internet - Core Contract No.: 632893 Strategic Objective: FI.ICT-2011.1.7 Technology foundation: Future Internet Core Platform Project Document Number: ICT-2013-FI-632893-WP14-D.14.1.1 Project Document Date: 2015-06-16 Deliverable Type and Security: PU Author: Carlos Ralli Ucendo (TID) Contributors: All partners in IoT Chapter.

https://forge.fi-ware.org/plugins/mediawiki/wiki/fiware/index.php/File:FP7Portrait_logo.jpg

https://forge.fi-ware.org/plugins/mediawiki/wiki/fiware/index.php/File:FI-WARE_logo.png

Future Internet Core Platform

D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 2

Preface

Executive summary

This document describes the FIWARE Internet of Things enablement chapter Reference Architecture,

comprising a set of Generic Enablers that provide the various FIWARE IoT capabilities.

Structure of this Document

The document is generated out of resources publicly provided in the FIWARE wiki. For the current

version of the documents, please visit the public wiki at

https://wiki.fiware.org/Internet_of_Things_(IoT)_Services_Enablement_Architecture

Change History

Release Major changes description Date Editor

v0.1 Initial contributions 2015.04.12 Carlos Ralli Ucendo, All

V0.2 Reviewed version 2015.05.20 All partners

V0.3 Updated based on contributions by partners 2015.05.21 Carlos Ralli Ucendo

V0.4 Review by Security Chapter’s architect 2015.05.29 Cyril Dangerville

Table of Contents

Preface .......................................................................................................................................................... 2

Executive summary ................................................................................................................................... 2

Structure of this Document ...................................................................................................................... 2

Change History .......................................................................................................................................... 2

Table of Contents ...................................................................................................................................... 2

1 Internet of Things enablement Architecture ........................................................................................ 7

1.1 Introduction .................................................................................................................................. 7

https://wiki.fiware.org/Internet_of_Things_(IoT)_Services_Enablement_Architecture



1.2 Architecture Overview .................................................................................................................. 7

1.2.1 Complete Architecture .......................................................................................................... 8

1.2.2 Typical IoT use-case Scenarios (I): Common Simple scenario ............................................... 9

1.2.3 Typical IoT use-case Scenarios (II): NGSI Native Scenario enabling Devices composition .. 11

1.2.4 Typical IoT use-case Scenarios (III): Full Scenario enabling Devices composition .............. 12

1.3 Basic Concepts ............................................................................................................................ 13

1.3.1 Device, IoT-end node, IoT Resource, Thing and IoT NGSI Context Entity ........................... 13

1.3.2 Interface Abstraction Levels ............................................................................................... 13

1.3.3 OMA NGSI – Context Management .................................................................................... 16

1.4 Architecture description of GEs .................................................................................................. 16

1.4.1 Backend ............................................................................................................................... 16

1.4.2 IoT Edge ............................................................................................................................... 17

1.5 Interfaces to other Chapters ....................................................................................................... 17

1.5.1 Data ..................................................................................................................................... 17

1.5.2 Security ............................................................................................................................... 17

1.5.3 I2ND .................................................................................................................................... 17

2 IoT Backend Device Management ...................................................................................................... 19

2.1 Copyright ..................................................................................................................................... 19

2.2 Overview ..................................................................................................................................... 19

2.2.1 IoT SDKs ............................................................................................................................... 21

2.2.2 IoT Agent Manager.............................................................................................................. 22

2.2.3 IoT Edge Management ........................................................................................................ 22

2.3 Basic Concepts ............................................................................................................................ 22

2.3.1 Ultralight2.0 ........................................................................................................................ 22

2.3.2 OMA LWM2M ..................................................................................................................... 22

2.3.3 MQTT ................................................................................................................................... 22

2.3.4 IETF CoAP ............................................................................................................................ 23

2.3.5 FIWARE NGSI ....................................................................................................................... 23

2.4 Main Interactions ........................................................................................................................ 23

2.4.1 IoT Agent: Ultralight2.0/http .............................................................................................. 23



2.4.2 IoT Agent: Ultralight2.0/MQTT ........................................................................................... 26

2.4.3 IoT Agent: LWM2M/CoAP ................................................................................................... 27

2.5 Basic Design Principles ................................................................................................................ 31

3 IoT Backend IoT Broker ....................................................................................................................... 33

3.1 Copyright ..................................................................................................................................... 33

3.2 Overview ..................................................................................................................................... 33

3.2.1 Role of IoT Broker GE .......................................................................................................... 33

3.2.2 Data model and interface outline ....................................................................................... 33

3.2.3 Functionality outline ........................................................................................................... 35

3.3 Main Concepts ............................................................................................................................ 36

3.3.1 Basic Concepts .................................................................................................................... 36

3.3.2 Additional Concepts ............................................................................................................ 37


3.4.1 Query Handling ................................................................................................................... 38

3.4.2 Subscription Handling ......................................................................................................... 39

3.4.3 Update ................................................................................................................................. 40

3.4.4 Notification ......................................................................................................................... 41

3.4.5 Availability Notification ....................................................................................................... 42


3.6 References .................................................................................................................................. 44

4 IoT Backend - IoT Discovery ................................................................................................................ 45

4.1 Copyright ..................................................................................................................................... 45

4.2 Overview ..................................................................................................................................... 45

4.2.1 Data model outline ............................................................................................................. 45

4.2.2 Functionality outline ........................................................................................................... 45

4.3 Basic Concepts ............................................................................................................................ 46

4.3.1 FIWARE NGSI ....................................................................................................................... 46

4.3.2 GE Internal Architecture ..................................................................................................... 47

4.4 Additional Concepts .................................................................................................................... 49

4.4.1 Associations in FIWARE NGSI-9 ........................................................................................... 49



4.4.2 Optional Features for Discovery ......................................................................................... 49


4.5.1 Registration ......................................................................................................................... 50

4.5.2 Discovery ............................................................................................................................. 52

4.5.3 Subscription ........................................................................................................................ 53

4.5.4 Notification ......................................................................................................................... 53

4.6 Added-value (Optional) Features ................................................................................................ 54

4.6.1 Geographical Discovery ....................................................................................................... 54

4.6.2 Probabilistic Discovery ........................................................................................................ 54

4.6.3 Semantic Discovery ............................................................................................................. 60

5 FIWARE.ArchitectureDescription.IoT.Gateway.DataHandling ............................................................ 65

5.1 Copyright ..................................................................................................................................... 65

5.2 Overview ..................................................................................................................................... 65

5.2.1 Internal architecture components diagram ........................................................................ 66

5.2.2 Main interfaces ................................................................................................................... 68

5.2.3 Main components ............................................................................................................... 69

5.3 Basic Concepts ............................................................................................................................ 72

5.3.1 Event concepts .................................................................................................................... 72

5.3.2 Complex Event Processing concepts ................................................................................... 73

5.3.3 Choosing the right implementation for you ....................................................................... 73

5.3.4 EspR4FastData implementation .......................................................................................... 74

5.3.5 SOL/CEP implementation .................................................................................................... 75

5.3.6 NGSI ..................................................................................................................................... 79

5.3.7 ETSI M2M ............................................................................................................................ 79



5.6 Re-utilised Technologies/Specifications ..................................................................................... 81

6 IoT Gateway – Protocol Adapter ........................................................................................................ 82

6.1 Copyright ..................................................................................................................................... 82

6.2 Overview ..................................................................................................................................... 82



6.3 Basic Concepts ............................................................................................................................ 82

6.3.1 Base Driver .......................................................................................................................... 86

6.3.2 Protocol Adaptation ............................................................................................................ 86

6.3.3 Generic Device Access API .................................................................................................. 87



7 IoT Gateway Device Management ...................................................................................................... 90

8 Glossary ............................................................................................................................................... 91



1 Internet of Things enablement Architecture

1.1 Introduction

Internet of Things chapter provides the Generic Enablers that allow Things to become available,

searchable, accessible, and usable context resources, thus fostering interactions between FIWARE-based

Applications and real-life objects.

In this context, Things mean any physical object, living organism, person or concept interesting from the

perspective of an application and whose parameters are totally or partially tied to sensors, actuators or

combinations of them.

The successful innovative idea behind FIWARE IoT is that all Things or IoT resources are exposed to

FIWARE Application developers just as other NGSI Context Entities. Therefore, developers will not have

to deal at all with today's complexity and high fragmentation of IoT technologies and deployment

scenarios. On the contrary, Application developers will just need to learn and use the same NGSI Context

Broker API used in FIWARE to represent all Context information.

Developers will just read Entity attributes to gather sensors information and will just modify (update)

specific attributes to trigger commands that will be delivered to actuators.

Normally, during FIWARE Releases (R1 to R6), the Backend will refer to the IoT Backend enablers

instances running in the FIWARE Lab, as described in the FIWARE Catalogue.

1.2 Architecture Overview

IoT chapter Architecture deployment varies from simple scenarios – connecting few devices using

standard IoT communication protocols to the Data chapter’s Context Broker – to more complex

scenarios distributed across a large number of IoT networks connecting IoT Gateways and IoT nodes and

providing advanced composition and discovery functions.

IoT GEs are spread over two different domains:

IoT Backend. It comprises the set of functions, logical resources and services hosted in a Cloud

data enter. Up north, it is connected to the Data chapter’s Context Broker, so IoT resources are

translated into NGSI Context Entities. South-wise the IoT backend is connected to the IoT edge

elements that make up the physical IoT infrastructure.



IoT Edge. It is made of all on-field IoT infrastructure elements needed to connect physical

devices to FIWARE Apps. Typically, it comprises: IoT end-nodes, IoT gateways and IoT networks

(connectivity). The IoT Edge and its related APIs will facilitate the integration of new types of

gateways and devices, which are under definition in many innovative research projects, and

warranty the openness of FIWARE IoT architecture.

1.2.1 Complete Architecture

This section depicts a complete scenario with all FIWARE IoT GEs (Backend and IoT Edge) and their

relationship in terms of APIs and communication protocols.

Please, to avoid unnecessary complexity, check first our current and planned typical scenarios. Check

them at the next subsections "Typical IoT use-case Scenarios".



Figure 1: IoT Architecture Overview

1.2.2 Typical IoT use-case Scenarios (I): Common Simple scenario

This scenario stands for a simple integration of IoT devices into the Data chapter Context Broker so that

FIWARE Application developers can interact with them. This use-case scenario is currently working and

available to FIWARE users at the FIWARE Lab.

The mandatory GEs for this scenario are:

Backend Device Management GE. That will translate the Device or Gateway specific

communication protocol into NGSI. It will always handle sensor notifications from the device

towards the Context Broker and for some specific protocols actuation/command messages from

the Context Broker to the device. Please refer to this GE architecture and documentation for

more information.



Data Chapter's Context Broker GE. It will handle all Context Entities representing the IoT

devices. It is the natural interface for FIWARE Application Developers for reading IoT sensing

information (Entity attributes) and, if implemented for the Device communication protocol, also

to trigger commands (Entity specific attributes).

Optional GEs and functions are:

Gateway Logic GE. It will handle the IoT Edge management API and functions plus the gateway-

to-gateway API and functions. For the first (Edge Management) it needs the correspondent

function/module to be activated and configured in the BE Device Management GE.

Figure 2: IoT Common Simple Scenario



1.2.3 Typical IoT use-case Scenarios (II): NGSI Native Scenario enabling Devices

composition

This scenario describes a use case where IoT end nodes or at least intermediate Gateways implement

the NGSI protocol. This use-case scenario is currently under experimentation and will be available soon

to FIWARE users at the FIWARE Lab.

If devices are NGSI native they might be directly connected to the Data Context Broker, but also they

may be connected to the Gateway Data Handling GE to perform events classification and/or

composition (Complex Event Processing) at the Gateway level.

Data Handling GE may push NGSI notifications directly to the Data Context Broker or might be pushing

them to the IoT Broker GE that combined with the IoT Discovery GE will be able to provide Things

composition and discovery functions.

Backend Device Management GE is optional, as it is only needed in this scenario if IoT Edge

Management functions are offered to IoT integrators.



Figure 3: IoT NGSI Native Scenario

1.2.4 Typical IoT use-case Scenarios (III): Full Scenario enabling Devices composition

This scenario includes all IoT enablers and it is depicted by the complete diagram shown at the

beginning of this section (figure 3). It is able to handle and combine (i.e. build Things from devices)

native NGSI devices/Gateways and also any other kind of IoT devices/gateways (via the BE Device

Management IoT Agents).

This use-case scenario is currently under experimentation and will be available at the end of Release 5 to

FIWARE users at the FIWARE Lab.



1.3 Basic Concepts

1.3.1 Device, IoT-end node, IoT Resource, Thing and IoT NGSI Context Entity

A device or IoT end-node is a hardware entity, component or system that either measures or influences

the properties of a thing/group of things or performs both activities concurrently. Sensors and actuators

are devices. Normally, we will use the term IoT-end node for complex physical devices with several

sensors and actuators (e.g. Arduino-based complex system. This means several boards -Arduino or not-

connected to an Arduino board for communications). Devices might use standard or proprietary

communication protocols that are natively pushed to the Backend GEs or might be translated into any

other standard or proprietary communication protocol at IoT Gateways. A particular case of

communication protocol is OMA NGSI. However, we do not expect devices and gateways to implement

it and therefore FIWARE GEs provide a way to translate these protocols into NGSI.

An IoT resource is a computational element providing access to sensor/actuator devices. An information

model for the description of IoT resources can include context data like location, accuracy, status

information, etc. IoT resource-level data consists not only of the measured data, but also context

information like the data type, a time stamp, accuracy of measurement, and the sensor by which the

measurement has been performed. IoT resources can be addressed using a uniform addressing scheme.

The resource is usually hosted on the device but it has a logical representation in the backend as well.

A thing can be any object, person, or place in the real world. Things are represented as virtual things

having an entity ID, a type and several attributes. Sensors can be modelled as virtual things, but other

real-world things like rooms, persons, etc. can be modelled as virtual things as well. So thing-level data

consists of descriptions of things and their attributes, while information on how the data has been

obtained might be contained as meta data, but is in general out of scope.

In FIWARE, all Things, IoT resources or IoT end-nodes are represented in the end as IoT NGSI Context

Entities in the Data chapter’s Context Broker GE.

1.3.2 Interface Abstraction Levels

This section explains in detail the functionalities and interfaces provided at each one of the IoT domains

or levels.

1.3.2.1 Backend Level

From a functionality point of view, IoT backend GEs perform the following functions:



Provision of Things NGSI Context entities. This means to create an NGSI Context Entity per each

one of the IoT resources. It is typically carried out by the Backend Device Management GE,

although in some advanced scenarios it might be alternatively implemented by the IoT Broker

GE. The GE in charge of this function plays the role of Context Producer for all Entity attributes

related to IoT resources sensing/observations. On the other hand, it registers itself as Context

Provider for those Entity Attributes related to actuation capabilities. This way, whenever a

FIWARE developer modifies (updates) one attribute related to actuation, the Context Broker will

push back the notification to this enabler that will in turn trigger a command to the device.

IoT Southbound protocol adaptation. The variety (fragmentation) of communication protocols

for IoT devices and/or gateways is extremely high today. FIWARE implements the most reputed

open standards in the market today to ease the connection of any kind of IoT devices.

Moreover, it provides an easy way to extend the number of supported communication protocols

regardless they are based on open or proprietary specifications. This function (in the backend) is

carried out by the Backend Device Manager GE so it handles the translation of IoT southbound

protocols (sensing and actuation) into/from the OMA NGSI protocol. For some specific

communication protocols, some tools (SDK) for specific or generic hardware platforms are also

provided for IoT integrators convenience.

IoT Edge management. Some configuration, operation and monitoring functions regarding IoT

Edge elements (Connectivity/Networks, Gateways, End-nodes) might be controlled from the

Backend Device Management GE and thus exposing a convenient API to IoT integrators in

addition to the NGSI API for Context Broker interconnection.

IoT Devices composition and discovery. IoT resources might be combined into new IoT

resources representing a more abstract concept. For instance, all temperature sensors in a room

might be combined (e.g. average of all observations) to create an improved "Room-

A_Temperature" thing with its related Context entity. Discovery facilities are provided for

atomic and composed IoT resources. These functions are provided by the IoT Broker GE in

combination with the IoT Discovery GE.

1.3.2.2 IoT Edge level

On the other hand, IoT Edge GEs provide the following functionalities:

IoT Southbound protocol adaptation. This functionality is similar but pretty much reduced

compared to the one provided at the backend, it just means to translate specific protocols into

NGSI within a gateway device so that native NGSI entities might be pushed directly to the



backend enablers (typically the IoT Broker or directly the Data chapter Context Broker). This

function is provided by the Protocol Adaptor GE. Current supported technologies are ZigBee and

RFID tags.

Complex (NGSI) Event Processing. Although there is one specific GE in the Cloud for this

functionality (Data Chapter’s CEP GE), for some scenarios it might be useful to reduce the

network traffic exchanged between the IoT edge elements and the Cloud infrastructure. It is

provided by the Data Handling GE within a Gateway device. Data Handling GE consumes,

processes and delivers NGSI events so it is typically used together with the Protocol Adaptor in

the same gateway device.

Gateway Logic. This function handles a gateway-to-gateway API (currently under definition) and

the IoT Edge configuration API at the gateway level (currently under definition too). For the IoT

Edge configuration of gateways existing protocols such as OMA-LWM2M management

interfaces, OMA-DM or BBF TR.69 are to be considered. It will be provided by the Device

Management GE.

IoT end-nodes Configuration. Some IP-capable nodes will not traverse gateways and therefore

they might be operated and monitored directly. This means they will implement themselves an

IoT Edge configuration API at the IoT-end node level. In this specific case mainly the

management interfaces of OMA-LWM2M are expected to be considered due to the constrained

nature of IoT-end nodes.

IoT Networks Configuration. In complex scenarios such as smart cities, IoT-end nodes are not

expected to be connected over a unique connectivity network. On the other hand, several IP

alternatives are expected to co-exist, namely: cellular (2G, 3G, 4G and soon 5G), meshed-radio

networks (6LowPAN/IEEE.802.15.4), IP/BLE, LPWA, etc. It is important to note that this control

plane is not the IoT devices or end-nodes control plane, but a different one that may include

also interaction with network APIs. This function will be provided at the IoT Edge module in the

Backend Management GE, the Gateway Logic module at Gateways and in a similar module at

IoT-end nodes.

A key design statement is that, whenever present, IoT Gateways are not expected to be permanently

connected to the Backend as per communications design or because of failures. Another relevant

remark is that IoT Gateways are expected to be constrained devices in some scenarios. Therefore, light-

weight implementations of the same GEs as in the IoT Backend, plus additional GEs interfaces helping to

save unnecessary features/GEs might be considered in the Gateway domain.



1.3.3 OMA NGSI – Context Management

The OMA NGSI Context Management standard provides the NGSI-9 and NGSI-10 interfaces to manage

and exchange Context Information about Context Entities. A Context Entity is any entity which has a

state. Values of attributes of defined entities become the Context Information that applications have to

be aware of in order to support context-awareness. Context Information is any volatile or persistent

information, which describes a state of a Context Entity. Of course, Context Entities could be users,

devices, places, buildings, therefore “things” as defined in the FIWARE IoT chapter. Context Information

related to things can be measured by sensors, and combined with other context information manually

set by humans, derived from interactions with the user, operations on handsets or terminals, inferred

from other information, or requested from databases. Adoption of OMA NGSI in the FIWARE IoT chapter

enables to manage the configuration of, and the data associated to, arbitrary physical objects in the real

world, (Devices and Things). Interestingly, it enables this at a level of abstraction that allows getting rid

of the complexity of managing connections with gateways and devices. Actually, updates on the state

and configuration of those physical objects will come as updates on the Context Information (i.e.,

updates on attributes of Context Entities representing those physical objects) and Configuration (i.e.,

updates on information about available Context Entities). The purpose of the NGSI-9 interface is to

exchange information about the availability of Context Information and Context Entities, while NGSI-10

is designed for exchanging the Context Information itself.

References:

NGSI Context Management - Open Mobile Alliance V1.0 May 29, 2012

FIWARE NGSI 9/10 Open API Specification

1.4 Architecture description of GEs

1.4.1 Backend

IoT Edge Generic Enablers (GEs) are:

FIWARE.ArchitectureDescription.IoT.Backend.DeviceManagement

FIWARE.ArchitectureDescription.IoT.Backend.IoTBroker

FIWARE.ArchitectureDescription.IoT.Backend.IoTDiscovery

http://technical.openmobilealliance.org/Technical/release_program/docs/NGSI/V1_0-20120529-A/OMA-TS-NGSI_Context_Management-V1_0-20120529-A.pdf

https://forge.fi-ware.eu/plugins/mediawiki/wiki/fiware/index.php/FI-WARE_NGSI_Open_RESTful_API_Specification

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FIWARE.ArchitectureDescription.IoT.Backend.DeviceManagement

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FIWARE.ArchitectureDescription.IoT.Backend.IoTBroker

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FIWARE.ArchitectureDescription.IoT.Backend.IoTDiscovery



1.4.2 IoT Edge

FIWARE.ArchitectureDescription.IoT.Gateway.DataHandling

FIWARE.ArchitectureDescription.IoT.Gateway.ProtocolAdapter

FIWARE.ArchitectureDescription.IoT.Gateway.DeviceManagement. This component will be

renamed to Gateway Logic in forthcoming Releases.

1.5 Interfaces to other Chapters

1.5.1 Data

IoT chapter mainly interacts with the Data chapter's Context Broker GE as long as all IoT resources

(Things) are translated into NGSI Context Entities that are provisioned in a specific Context Broker. It

uses both NGSI10 (for sensing and actuation attributes of Thing entities) and NGSI9 (just for actuation

attributes, if implemented). Please refer to Context Broker GE documentation for more information.

1.5.2 Security

IoT chapter GEs are secured by hiding themselves behind a Security Chapter’s PEP proxy GE that

provides OAuth2.0-token-based access control (including users permissions and roles plus the concept

of FIWARE service-path and subservice-path) using the Security Chapter’s Identity Management GE as

Authentication server, aka OAuth Provider in the context of OAuth. Please refer to FIWARE deliverable

D.17.1.1 for more information.

Security at the Backend domain APIs will be provided by the overall security access mechanism based on

Oauth2.0, developed in the FIWARE Security chapter and used as a reference in the FIWARE testbed.

1.5.3 I2ND

IoT chapter will interact with Robots included in the I2ND chapter. The typical scenario will be Robots

based on ROS architecture interacting with IoT resources represented as NGSI Entities in a Context

Broker. However, straight interactions with IoT elements might be envisaged. Please refer to I2ND

Robotics section for more information.

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FIWARE.ArchitectureDescription.IoT.Gateway.DataHandling

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FIWARE.ArchitectureDescription.IoT.Gateway.ProtocolAdapter

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FIWARE.ArchitectureDescription.IoT.Gateway.DeviceManagement



.



2 IoT Backend Device Management

2.1 Copyright

Copyright © 2015 by Telefonica I+D

2.2 Overview

The Backend Device Management GE is normally the central enabler at the IoT backend for most

common scenarios.

Basically, it provides the following features:

Connect physical devices to a FIWARE platform. Devices and/or gateways may use different

standard or proprietary communication protocols and API. The following sections provide

details on currently/planned supported protocols, how to configure and scale up the translators

(IoT-Agents) plus information on extensibility.

Manages IoT-related NGSI Context Entities. It handles the connection (northbound) to a Data

chapter Context Broker to create one Context Entity per physical connected device. For most

cases, FIWARE Application developers will only interact with those NGSI Entities.

IoT Edge Management. This functionality is provided by the "IoT Edge Manager" module, which

is planned for FIWARE Release 6. It will provide IoT Integrators with the ability of configure,

operate and monitor IoT end-nodes, IoT Gateways and IoT networks.

The following diagram depicts its overall planned architecture.

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/TID



Figure 4: Southbound IoT protocols

3.1 Main Components

3.1.1 IoT Agents

The IoT Agents are the software modules handling South IoT Specific protocols and North OMA NGSI

interaction. The minimum configuration of a Backend Device Management GE in a FIWARE ecosystem

includes at least one IoT Agent.

IoT Agents must:

Handle the creation of an NGSI Context Entity in a Context Broker (at its northbound) per each

one of the connected IoT Devices.

Act as Context Producer for those attributes related to sensing capabilities or observations.

Provide an Administration/Configuration API.



IoT Agents may:

Perform Commands triggering whenever a developer updates specific attributes in a

Northbound Context Broker. In this case, the IoT Agent will register itself and play the role of

Context Provider of those specific actuation attributes. If this functionality is not provided, the

IoT Agent must provide an API to receive commands and deliver them to devices.

Currently, the following IoT Agents are specified in the roadmap:

UL2.0/HTTP. Ultralight2.0 is an open specification for devices semantics (capabilities,

observations, commands). This IoT Agent handles devices using this representation over

HTTP/IPv4 connections.

UL2.0/MQTT. This IoT Agent handles devices using this representation over MQTT topics.

LWM2M/CoAP. This IoT Agent connects devices compliant with the ETSIM2M and OneM2M

southbound proposition: OMA Lightweight M2M specification over the REST protocol for

constraint devices/networks (IETF CoAP).

Generic IoT Agent. A skeleton of a generic IoT agent is provided to help IoT integrators to cope

with new standards or proprietary protocols.

2.2.1 IoT SDKs

For some specific hardware Devices/IoT-end nodes and protocols/APIs, SDKs (client/device side tools)

might be optionally provided for certain Hardware platforms in order to facilitate testing and integration

activities.

Those SDKs are not installed in the Backend but at the devices or gateways to be connected. However,

SDKs depend on specific communication protocols and thus specific IoT Agents, so they are provided

together with them and not as IoT Edge GEs.

For FIWARE Release 4, an SDK related to Ultralight2.0/HTTP Agent is provided.



2.2.2 IoT Agent Manager

The IoT Agent Manager is an optional module that will interface with all the IoT Agents installed in a

data enter throughout their Administration/Configuration API. This will enable a single point to launch,

configure, operate and monitor all IoT-Agents in a FIWARE Ecosystem.

This module will be specified in FIWARE Release 5.

2.2.3 IoT Edge Management

The IoT Edge Manager is an optional module that will interface with IoT end-nodes, IoT Gateways and

IoT network APIs throughout their IoT Edge API in order to operate and monitor the IoT Edge

infrastructure, i.e. connectivity, gateways and devices.

This module will be specified in FIWARE Release 6.

2.3 Basic Concepts

The Backend Device Management GE is based on the usage of the following protocols: FIWARE NGSI,

Ultralight2.0 and OMA-LWM2M.

2.3.1 Ultralight2.0

Ultralight2.0 Specs. (HTTP & MQTT Transport)

2.3.2 OMA LWM2M

OMA LightweightM2M v1.1

2.3.3 MQTT

MQTT 3.1 Specification

https://forge.fiware.org/docman/admin/index.php?editdoc=1&docid=5084&group_id=11

http://technical.openmobilealliance.org/Technical/technical-information/release-program/current-releases/oma-lightweightm2m-v1-0

http://mqtt.org/documentation



2.3.4 IETF CoAP

IETF Constraint Application Protocol: RFC 7252

2.3.5 FIWARE NGSI

NGSI Context Management - Open Mobile Alliance V1.0 Aug 3, 2010

FIWARE NGSI Open API Specification

Context Information interface OMA_NGSI_10

2.4 Main Interactions

2.4.1 IoT Agent: Ultralight2.0/http

This IoT Agent connects devices (or Gateways grouping devices) described with the open description

Ultralight2.0 over the HTTP protocol.

From FIWARE Release 4 onwards, commands will be triggered from specific attributes in the related

NGSI Context Entity.

The following subsections detail the main interactions needed to work with this specific IoT Agent.

2.4.1.1 Service Creation

This operation is thought for private instances of the component or Admin users of the FIWARE Lab

public instance.

Each tenant (customer or group of users) is expected to use a different service. This is the way this GE

enables multitenancy.

A Service is tied to a shared secret, called API-KEY that all devices (or gateways) need to know to be able

to send measurements or receive commands to/from the IoT Agent. This security mechanism is

complementary to the overall IDM architecture for all FIWARE GEs.

Devices sending observations with a FIWARE-Service "S1" and FIWARE-Service-Path "PATH" headers will

be represented up north in the Context Broker as NGSI Entities with the same FIWARE-Service and

FIWARE-Service-Path.

http://tools.ietf.org/html/rfc7252

http://www.openmobilealliance.org/Technical/release_program/docs/NGSI/V1_0-20101207-C/OMA-TS-NGSI_Context_Management-V1_0-20100803-C.pdf


http://forge.fi-ware.eu/plugins/mediawiki/wiki/fiware/index.php/OMA_NGSI_10



In order to create a Service, an admin will run a specific HTTP POST request. HTTP GET operation at the

same URL will return the existing services.

Figure 5: IoT Agent API – Service Creation

2.4.1.2 Device Registration/Creation

Before a device sends observations it needs to be registered. This registry will include the Device ID,

Entity ID, Entity type, attributes, static attributes and command parameters related to the device.

Depending on the specific service agreements, device registration might be carried out by an Admin,

regular users or both.

To register a device, an HTTP POST request will be run.

Figure 6: IoT Agent API - Registration



2.4.1.3 Device Observation

Once a device is registered, observations might be sent with a simple HTTP POST request (executed from

the Device or a gateway) that includes the Device ID and the API-KEY in the URL.

Figure 7: IoT Agent API - Observation

2.4.1.4 Device Commands

There are two ways to send UL2.0 commands: Push and Pooling modes.

Push commands are sent to devices in a synchronous way. Therefore it assumes always-on

reachable devices (or gateways) implementing an HTTP server.

Pooling commands are queued in the IoT Agent until a Device asks for them. The Administration

API provides a method to check out the status of commands addressed to a specific device.

For the case of Pooling commands, Devices (or gateways) will run an HTTP request periodically and get

the full queue of pending commands. Push devices will run an HTTP server to synchronously receive the

command once delivered to the IoT-Agent.



Figure 8: IoT Agent API - Commands

2.4.1.5 Administration API

The current Administration API of this IoT Agent comprises the above-described Functions "Service

Creation" and "Device Register".

2.4.2 IoT Agent: Ultralight2.0/MQTT

To be provided in the next Architecture release.



2.4.3 IoT Agent: LWM2M/CoAP

This IoT Agent connects Lightweight M2M Clients (devices), communicating over COAP, to the NGSI

Context Broker in the Data Chapter. In this release, the IOT Agent accepts writes to device attributes

through writes in the Context Entities, but it doesn’t support full commands.

The following sections detail the main interactions with the agent (without entering in low-level details).

2.4.3.1 Device Provisioning

In order for the system to recognize the device and do the appropriate mapping to NGSI context

elements, one of two things must happen: either the device is provisioned in advance into the system,

or a service is provisioned in the system and the device assigned to it. In the former case, the DeviceID

will be used to identify the device, using it as the Endpoint name in LWM2M. The following pieces of

data can be specified in provisioning: device ID, security information, NGSI entity mappings and types of

attributes provided.

Figure 9: IoT Agent API - Provisioning

2.4.3.2 Service creation



For those cases where the devices are not specified individually, services can be provisioned as a whole.

A service is identified by a pair of (resource, API Key) attributes, and can contain roughly the same kind

of information as the device provisioning requests. The specified resource corresponds to a LWM2M

server endpoint where the clients will send their requests (the API Key will be used in southbound

security interactions, not implemented in the first version of the Agent GEri).

Each time a device arrives to the specified endpoint, it will be assigned to the proper service based on

the endpoint.

Figure 10: IoT Agent API – Service Creation

2.4.3.3 Device Registration

Every device in LWM2M must register to the LWM2M server before starting any interaction. In this

registration the LWM2M client has to indicate the following information to the server: Endpoint name

(DeviceID) and supported objects (a list of links for every OMA LWM2M object that can be accessed by

the server). The LWM2M server uses this information to assign the device to a service (or to retrieve the

device information in case it has been individually provisioned).

Based on the service or device information, the server will decide what attributes of the objects

provisioned by the device are active, lazy or commands, and it will register itself in the Context Broker as

the Context Provider of those in the two last categories (lazy and commands).



Figure 11: IoT Agent API - Registration

For those attributes defined as active, the server will emit an Observe request (Information Reporting

interface in LWM2M), indicating the desired timeout between measures to be received (using the

WriteAttributes operation).

2.4.3.4 Device lazy observation

For those attributes of the devices that marked as lazy, the updates and reads of Context Entities in the

Context Broker GE will be mapped to Read and Write operations from the Device Management Interface

in LWM2M.



Figure 12: IoT Agent API – Lazy Observation

2.4.3.5 Device active observation

For those attributes of the devices marked as active, an Observe operation is issued from the server

upon registration, expecting subsequent measures to be issued as Notify responses to the Observe

message (as stated in the Information Reporting Interface of the LWM2M Specification).

Figure 13: IoT Agent API – Active Observation

2.4.3.6 Device command



Commands are issued as changes in a particular value in the Context Entity, and are mapped to Execute

operations from the server to the client. The status of the execution is maintained in a special attribute

in the Context Entity that is updated with any upcoming information to the server.

Figure 14: IoT Agent API – Commands

2.5 Basic Design Principles

Face IoT fragmentation: Today's IoT framework considers a number of vertical silos exploiting

different M2M technologies and protocols. The Backend Device Management GE aims to offer a

simple and flexible solution to connect any kind of IoT devices to a FIWARE ecosystem.

Modular design: Functionalities (including supporting different IoT southbound protocols) are

provided as modules. The minimum configuration of a Backend Device Management GE in a

Data enter comprises one single IoT-Agent. Other configurations may add other IoT Agents, the

IoT Agent Manager or the IoT Edge Manager. All those modules might be installed in the same

or different virtual machines for the sake of scalability and redundancy.



Easy extensibility: An skeleton of an IoT-Agent should be provided to allow IoT integrators to

create new IoT Agent modules for non supported standards or proprietary device/gateway

protocols.



3 IoT Backend IoT Broker

3.1 Copyright

Copyright © 2015 by NEC

3.2 Overview

3.2.1 Role of IoT Broker GE

The IoT Broker GE is an IoT Backend enabler. It is foreseen to run on a machine in a data center, where it

serves as a middleware which enables fast and easy access to Internet-of-Things data. Instead of having

to deal with the technical details of existing FIWARE IoT installations, application developers only need

to set up their application to communicate with the IoT Broker in order to retrieve the data they need.

The IoT Broker takes care of the task of communicating with the different IoT Devices and Gateways to

retrieve the needed information on behalf of the applications.

3.2.2 Data model and interface outline

The main interface exposed by the IoT Broker GE is FIWARE NGSI. This API has been developed by the

FIWARE community as a binding of the OMA NGSI (Next Generation Service Interface) Context

Management standard (OMA NGSI 9/10). This interface is exposed both southbound towards IoT

Gateways and Devices and northbound towards IoT Applications. In other words, the IoT Broker GE

retrieves information from IoT Gateways and Devices via the FIWARE NGSI protocol, while the same

protocol is used by applications to retrieve information from the IoT Broker GE. The IoT Broker

aggregates, filters, translates, and enriches the FIWARE NGSI based information before passing it to the

applications.

FIWARE NGSI consists of an information model and two distinct interfaces for information exchange.

The information model is centered on the concept of so-called context entities. Context entities

represent arbitrary objects of the real world, and the state of such objects is described in terms of the

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/NEC



values of attributes. In addition, metadata can be used to describe properties of attribute values. In the

context of the Internet-of-Things, context entities are used for representing devices like sensors and the

values they measure. Additionally, and even more importantly, arbitrary physical objects (Things) like

rooms, vehicles, or persons and their attributes like temperature or location, are as well expressed by

FIWARE NGSI context entities.

The two FIWARE NGSI context management interfaces distinguish between two types of information.

The first type is called context information and consists of attribute values and associated metadata as

described above. In other words, context information is information about the state of context entities.

This kind of information is exchanged using the operations that have been defined in OMA NGSI 10 and

adopted by FIWARE NGSI. The second type of information is context availability information, i.e.,

information on where context information can be retrieved by OMA NGSI 10 operations. This kind of

information is exchanged using the operations defined in OMA NGSI 9. More details and references on

these interfaces can be found in the next sections.



Figure 15: IoT Broker interfaces

3.2.3 Functionality outline

The IoT Broker GE is the point of contact for accessing information about things and their attributes (see

data model above). Applications can access this information using the NGSI 10 interface of the IoT

Broker. In the overall FIWARE reference architecture [7], the Context Broker GE from the Data & Context

Chapter enables up-to-date access to Context Information (not only things) by any application. Such

Context Information can be of any nature and comprises, but is not limited to, the Context Information

about things that is made available by the IoT Broker GE. However, the IoT Broker GE can as well be

accessed by any number of applications directly via its NGSI 10 interface. This enables a simpler setup

when applications only use IoT information.

For basic usage, the IoT Broker is a stateless component in the sense that it neither stores context

information (i.e. attribute values) nor context availability information. Instead, it interacts with the

whole IoT deployment in order to satisfy the requests it receives from the Context Broker GE or final

applications: when receiving a query on Context Information, e.g. attributes of Context entities, the IoT

Broker GE first contacts the IoT Discovery GE to perform a discovery. Using the resulting availability

information, the IoT Broker then contacts the information providers using NGSI 10 to receive the actual

attribute values. Note that the number of information providers to be contacted can be arbitrary. The

information is then aggregated and returned to the application.

The NGSI context management interface describes three types of operations for exchanging context

information. The first and most basic kind of operation is a simple query. When an application invokes

the query, it expects to receive context information as the response. The second kind of operation is a

subscription. When an application subscribes to certain context information, it just receives a

subscription ID as the response. Context information is then sent to the application in the form of

notifications. Depending on the kind of subscription, context information can be sent whenever

attribute values change, whenever attribute values exceed or are inside some range, or simply at fixed

time intervals.

Finally, there is a mode of information exchange defined in the NGSI context interface where

information updates are sent without the need to ask for them. When the IoT Broker receives such

updates, it forwards the information to some default application that is responsible for further

processing and/or storing such updates. In the FIWARE reference architecture, this application is the

Context Broker GE.

https://forge.fi-ware.eu/plugins/mediawiki/wiki/fiware/index.php/FI-WARE_Architecture



In case of advanced usage IoT Broker keeps certain kinds of states as represented by the two

repositories in the architecture picture. Firstly, it needs to keep track of existing subscriptions using the

Subscription Storage. This includes both subscriptions received from applications and subscription

issued to IoT Agents (i.e. Gateway Data Handling GE) by the IoT Broker. Secondly, the IoT Broker

includes a repository of context registrations, that is, information about where certain context

information can be retrieved. The latter is a new feature since version 4.2, and its purpose is to enable

usage of the IoT Broker without an external IoT Discovery GE in simple scenarios.

3.3 Main Concepts

3.3.1 Basic Concepts

The IoT Broker GE is based on the OMA NGSI context data model [1].

3.3.1.1 FIWARE NGSI

The IoT Broker GE provides FIWARE's RESTful binding specification of the OMA NGSI context

management interface (FIWARE NGSI specifications for short). More specifically, the GE provides the full

set of NGSI-10 operations, where it acts both as a client and as a server. As for NGSI-9, the IoT Broker GE

acts as a client and provides only the server function required for receiving context availability

notifications.

FIWARE NGSI Context Management specifications are based on the NGSI Context Management

specifications defined by OMA (Open Mobile Alliance) [2]. They take the form of a RESTful binding

specification of the two context management interfaces defined in the OMA NGSI Context Management

specifications, namely NGSI-9 [3] and NGSI-10 [4]. They also solve some ambiguities in the OMA specs

and extend them when necessary to implement the FIWARE Vision.

You can visit the FIWARE NGSI Open RESTful API Specification [5] to learn the main concepts underlying

FIWARE NGSI Context Management specifications. Please also refer to the e-learning course [9] on the

IoT Broker Generic Enabler.

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/NGSI_9/10_information_model



https://forge.fi-ware.eu/plugins/mediawiki/wiki/fiware/index.php/FI-WARE_NGSI-9_Open_RESTful_API_Specification_%28PRELIMINARY%29


https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FI-WARE_NGSI_Open_RESTful_API_Specification

http://edu.fi-ware.eu/course/view.php?id=33



3.3.2 Additional Concepts

3.3.2.1 Associations in FIWARE NGSI

One of the central features of the FIWARE IoT Backend is its ability to derive information about arbitrary

things (cars, people, houses, etc.) from device-level information. The link between these two levels of

abstraction is provided by the association concept defined for FIWARE NGSI. A definition of the concept

and its representation in NGSI-9 messages can be found on the NGSI association definition page [6]. An

association is an ordered pair consisting of a sensor-level entity (possibly with an attribute name) and a

thing-level entity (possibly enhanced with an attribute name). The interpretation of such a pair is that

the sensor provides information about the thing.

Associations potentially play a role for every operation supported by the IoT Broker.

The IoT Broker GE queries the IoT Discovery GE not only for context providers, but also for

associations.

For resolving a queryContext operation, the IoT Broker uses the association concept to find out

which lower-level entities, typically device-level entities, it has to retrieve from the context

providers. For example, for finding out the speed of a certain car, the IoT Broker might ask for

the measurement of a speedometer.

For the translation of thing-level subscriptions into device-level subscriptions the association

concept is used in the same way as in the query case.

When context providers call the update operation of the IoT Broker GE, the latter analyzes

associations in order to find out if updates of attribute values of Thing-level entities have to be

triggered.


The IoT Broker Generic Enabler is a middleware used for setting up and maintaining the data flows in

IoT deployments. It is designed to interact with large numbers of IoT data providers and data

consumers. On behalf of the consumers, the IoT Broker retrieves, assembles, and processes information

from the providers, offering the consumers a simple interface and masking the complexity and

heterogeneity of the Internet of Things. For this reason, the IoT Broker GE interacts potentially with a

large number of Gateways, other Backend instances, Devices, and of course data consumers.

Furthermore, the IoT Broker GE typically interacts with at least one instance of the IoT Discovery GE (but

there is also a standalone mode since release 4.2). The latter is where the IoT Broker GE retrieves

https://forge.fi-ware.eu/plugins/mediawiki/wiki/fiware/index.php/NGSI_association



information about where information is available in the IoT installation. Please note that due to a recent

architectural change the IoT Discovery GE is still referred to as "ConfMan GE" in the Flow Charts below.

In the FIWARE architecture, the role of the data consumer is played by a Generic Enabler in the Data &

Context Chapter: the Context Broker. The IoT Broker GE communicates with the Context Broker GE via

the Northbound Interface. The Southbound interface is used to communicate with the so-called IoT

Agents, which are providing data. The role of IoT Agent can be played by either the Backend Device

Management GE, or by the Gateway Data Handling GE.

3.4.1 Query Handling

Queries are one-time requests for information. They are realized by the queryContext operation of OMA

NGSI-10. With NGSI-10, GEs or applications can query for Thing-level information. In reaction to a query,

the IoT Broker determines the set of IoT Agents that can provide the requested information. This is done

by sending a discoverContextAvailability request to the IoT Discovery GE, which returns relevant

associations between Thing-level entities/attributes and Device-level entities/attributes, and provides

addresses of the IoT Agents who can provide the required information. After this step, the IoT Broker GE

queries the identified IoT agents, it aggregates the results, it maps Device-level entities to Thing-level

entities and it forwards them to the GE or application that issued the query.



Figure 16: IoT Broker – Query handling

3.4.2 Subscription Handling

Subscriptions are requests for information updates the issuer wishes to receive, under certain

conditions to be specified in the request message. The picture 17 shows the interaction diagram for the

OMA NGSI-10 subscribeContext operation, but the same interaction pattern also applies to the

updateContextSubscription and to the unsubscribeContext operations. Also in these interactions, like in

Query Handling, the role of the IoT Discovery GE is to provide the relevant associations between the

Device-level and Thing-level entities and the addresses of the relevant information sources (i.e. the IoT

agents).



Figure 17: IoT Broker – Subscription handling

3.4.3 Update

In general, updates received by the IoT Broker GE are forwarded to the IoT Discovery GE. Before

forwarding, the IoT Broker discovers the Thing-level entities associated to the Device-level entities about

which it has received an update. For that reason the IoT Broker contacts the IoT Discovery GE using a

discoverContextAvailability request. Note that a single received update may translate into several

updates because attributes of several things may need to be updated as a result of an update of a



device-level entity (e.g., the temperature measured by a given thermometer may lead to updates on a

thing representing a room close to the thermometer, as well as updates on things representing the floor

and the building where the room is located).

Figure 18: IoT Broker – Update handling

3.4.4 Notification

Notifications are the counterpart of subscriptions. A notification is sent whenever the condition that has

been specified in the subscription is satisfied. Like in the update operation, also here the IoT Broker

needs to resolve the relevant associations.



Figure 19: IoT Broker – Notification handling

3.4.5 Availability Notification

When a new IoT Agent having information that is relevant for an existing subscription becomes

available, the IoT Broker is notified about this in an availability notification, so that it can then make a

subscription to the new IoT Agent.



Figure 20: IoT Broker – Availability Notification handling


Stateless Design: The IoT Broker GE is stateless insofar as no context information is stored by it.

Its role in the Internet of Things is not to serve as a central data repository, but as a central point

which retrieves and aggregates data from various sources on behalf of applications. The absence

of state enables easy replication of IoT Broker GE instances, or federated deployments where

multiple instances of the IoT Broker GE are responsible for different parts or domains of the

Internet of Things. In such cases each IoT Broker GE instance serves as an IoT Agent for the other

instances, and each IoT Broker GE can in principle retrieve any piece of information available in

the IoT deployment.



Silent System: The IoT Broker enables IoT systems whose level of activity is proportional to the

number and complexity of the requests by applications. This is due to the fact that the IoT

Broker only queries IoT Agents when there are requests from applications. The principle of silent

systems is particularly helpful when working with battery constrained devices. However, also

systems that continuously produce data can be built using the IoT Broker GE. In this case, IoT

Agents have to be configured to use the NGSI 10 update operation in order to deliver data

independently of application requests.

3.6 References

[1] https://forge.FIWARE.eu/plugins/mediawiki/wiki/fiware/index.php/NGSI_9/10_information_model

[2] http://technical.openmobilealliance.org/Technical/release_program/docs/NGSI/V1_0-20120529-A/OMA-

TS-NGSI_Context_Management-V1_0-20120529-A.pdf

[3] https://forge.FIWARE.eu/plugins/mediawiki/wiki/fiware/index.php/FIWARE_NGSI-

9_Open_RESTful_API_Specification_%28PRELIMINARY%29

[4] https://forge.FIWARE.eu/plugins/mediawiki/wiki/fiware/index.php/FIWARE_NGSI-

10_Open_RESTful_API_Specification_%28PRELIMINARY%29

[5] https://forge.FIWARE.eu/plugins/mediawiki/wiki/fiware/index.php/FIWARE_NGSI_Open_RESTful_API_Spe

cification

[6] https://forge.FIWARE.eu/plugins/mediawiki/wiki/fiware/index.php/NGSI_association

[7] https://forge.FIWARE.eu/plugins/mediawiki/wiki/fiware/index.php/FIWARE_Architecture

[8] https://forge.FIWARE.eu/plugins/mediawiki/wiki/fiware/index.php/FIWARE_Open_Specifications_Legal_N

otice

[9] http://edu.FIWARE.eu/course/view.php?id=33

https://forge.fi-ware.eu/plugins/mediawiki/wiki/fiware/index.php/NGSI_9/10_information_model









https://forge.fi-ware.eu/plugins/mediawiki/wiki/fiware/index.php/NGSI_association

https://forge.fi-ware.eu/plugins/mediawiki/wiki/fiware/index.php/FI-WARE_Architecture

https://forge.fi-ware.eu/plugins/mediawiki/wiki/fiware/index.php/FI-WARE_Open_Specifications_Legal_Notice

https://forge.fi-ware.eu/plugins/mediawiki/wiki/fiware/index.php/FI-WARE_Open_Specifications_Legal_Notice

http://edu.fi-ware.eu/course/view.php?id=33



4 IoT Backend - IoT Discovery

4.1 Copyright

Copyright © 2014-2015 by University of Surrey

4.2 Overview

4.2.1 Data model outline

The IoT-Discovery GE is the part of the Backend tier of the IoT Architecture which is responsible for

context source availability management. The underlying data model of this GE is based on the OMA

NGSI-9 Context Management Information Model. This model relies on the concept of context entities,

which are generic entities whose state is described by the means of values of attributes and associated

metadata. In the context of IoT, context entities and context entity attributes can be used to model IoT

resources and the variables they measure, respectively. Additionally - and more importantly -arbitrary

physical objects (Things) like rooms, people, cars, etc. and their attributes like temperature, geo-

location, etc. can be used as well.

4.2.2 Functionality outline

The IoT-Discovery GE is responsible for the context availability registrations from IoT Agents, thus

making it the access point for information about entities and their attributes. In particular, the context

availability information provided by the IoT-Discovery GE is either forwarded from IoT Agents exposing

the FIWARE NGSI-9/10 interfaces. The role of IoT Agents can be played by either the Data Handling GE in

IoT Gateways, or the Backend Device Management GE. It is also possible that the context information is

provided by other IoT Backend systems.

Information about context source availability is typically forwarded to the FIWARE Context Broker GE so

that context information about Things becomes accessible to applications. However, it can be the case

that the Context Broker GE may manage context availability information that is not necessarily provided

by the IoT-Discovery GE, therefore linked to the Internet of Things, but gathered from other parts of the

application.

http://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/UNIS



More precisely, using the FIWARE NGSI-9 interface that the IoT-Discovery GE provides, applications and

services will be able to register, discover and subscribe to updates on context availability information

such as:

Information about IoT resources and the variables they measure

Information about Things and their attributes

Information about Associations between entities, whereby the attributes of a Thing can be

derived from attributes of other Things, or from attributes measured by IoT resources.

Optionally, the IoT-Discovery GE may provide different flavors of discovery mechanisms, along with the

main mode of discovery. These optional features include (but are not limited to):

Geographic: where entities can be discovered based on their location.

Probabilistic: where entities are discovered by similarity in syntax.

Semantic: where entities can be semantically-annotated and linked to well-known ontology

models that can provide more detailed information about a particular attribute, and increase

their interoperability with other information on the Web.

4.3 Basic Concepts

The IoT-Discovery GE is based on the OMA NGSI context data model.

4.3.1 FIWARE NGSI

The IoT-Discovery GE implements the RESTful FIWARE NGSI-9 binding specification. FIWARE NGSI

Context Management specifications are based on the NGSI Context Management specifications defined

by OMA (Open Mobile Alliance). They include two RESTful context management interfaces NGSI-9 and

NGSI-10 (see Open RESTful API Specification), which also solve some ambiguities in the OMA

specification for NGSI and extends them where necessary to realize the FIWARE Vision.

You can visit the FIWARE NGSI Context Management tutorial to learn the main concepts underlying

FIWARE NGSI Context Management specifications.


http://www.openmobilealliance.org/Technical/release_program/docs/CopyrightClick.aspx?pck=NGSI&file=V1_0-20101207-C/OMA-TS-NGSI_Context_Management-V1_0-20100803-C.pdf

http://www.openmobilealliance.org/Technical/release_program/docs/CopyrightClick.aspx?pck=NGSI&file=V1_0-20101207-C/OMA-TS-NGSI_Context_Management-V1_0-20100803-C.pdf

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FIWARE.ArchitectureDescription.IoT.Backend.ThingsManagement#Open_API_SpecificationsFIWARE_NGSI

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FI-WARE_NGSI_Context_Management_tutorial



4.3.2 GE Internal Architecture

The internal architecture of IoT-Discovery consists of two main components, the Configuration Manager

and the Configuration Repository.

Figure 21: IoT Discovery – Architecture

4.3.2.1 Configuration Manager

The Configuration Manager component consists of a context information registry in which context

provider applications can be registered. In addition, components interacting with the Configuration



Manager can perform discovery operations on that context registration information or subscribe to

changes on it.

It detail, the component provides the following functionality:

Allow NGSI-9 clients, such as the IoT Broker to discover and subscribe to context information,

through the NGSI-9 interface exposed by Iot Discovery.

Announce the availability of context information towards the Context Broker GE (or any other

component supporting the NGSI-9 interface) through the northbound NGSI-9 interface. The

Configuration Management component stores the context information in a Configuration

Repository described in next subsection. This repository is used to answer NGSI-9 request in the

exposed interfaces described above.

Receive registrations from IoT Gateways and the Thing-level Adapter through the NGSI-9

southbound interface and store this information in the Configuration Repository, thus updating

the context information registry.

4.3.2.2 Configuration Repository

The Configuration Repository stores information on the availability of context information and can be

accessed through the Configuration Management. When a NGSI-9 client sends a

discoverContextAvailabilityRequest operation in order to find out where the desired context

information can be found, the Configuration Manager returns zero or more ContextRegistration

instances that match the client's request.

Results can also include associated context sources if specified in the operation scope field in the

request. With this approach, IoT-Discovery can maintain higher-level context information that is not

available in the IoT Gateways or Devices. For example, a Gateway might not know the concept of cars,

but only maintains a list of sensors and their measurements. The information about which specific

sensors provide information about same car could be maintained only by the IoT-Discovery GE.



4.4 Additional Concepts

4.4.1 Associations in FIWARE NGSI-9

Figure 21: IoT Discovery – Associations

IoT-Discovery enables enhanced associations. See section 5.7.3.3 to have more information on this.

4.4.2 Optional Features for Discovery

When discovering and subscribing to notification for specific entities, the Operation Scope field in the

respective operation i.e. discoverContextAvailability and subscribeContextAvailability can be used to

specify a particular type of discovery, which can be:

By Association: using the type includeAssociations in the scope field

Geographical: using the type FIWARE::Location.



Probabilistic: using the type search.

There is also an option for discovering context sources via semantic query methods such as SPARQL. This

requires registrations to be available in semantic form. An NGSI-9 message can link to a corresponding

semantic description. This can be done by the client by specifying the link to the semantic description

the in the registration metadata using the type linked-data.


IoT-Discovery has a number of interfaces for interacting with applications, users, and other GEs within

the FIWARE architecture. For example, the GE communicates with the Context Broker GE via the

Northbound Interface and with the IoT Agents on the Southbound Interface. With respect to GEs, the

role of IoT Agent can be played by either the Backend Device Management GE, or by the Gateway Data

Handling GE. There is also an interface between the IoT Broker GE and the IoT-Discovery GE. In the

context of NGSI, IoT Discovery plays the role of the server, and any other actors interacting with it play

the role of the client. To illustrate this, in the following diagrams all IoT Agents, GEs, and applications are

abstracted using "NGSI-9 Client".

4.5.1 Registration

In order for their information on Things and/or Resources to be available at the Backend, IoT Agents

(which play the role of "NGSI9 Client") need to register their information to the IoT-Discovery GE. This is

done via the RegisterContext operation. Note that the registration can be performed at various levels of

granularity: registering specific Entity/Attribute combinations, registering that information about certain

entity type is available, or even in general registering that some entity information is available (without

getting more specific) are all possible registrations. Also note that in absence of a Context Broker GE

instance the registration is still stored in the IoT-Discovery GE.

This sequence is used for both new registrations and registrations updates.



Figure 22: IoT Discovery – Registration

4.5.1.1 Registration Update

A context provider can update a registration by using the registration ID that is returned from the

previous registration response.



Figure 23: IoT Discovery – Registration Update

4.5.2 Discovery

The entities that typically play the "NGSI-9 Client" role in this case are the IoT Broker GE (when

requesting context availability information to process NGSI-10 query/updates) or any other NGSI

application that wants to know the context availability associated with a given entity/attribute.



Figure 23: IoT Discovery – Discovery

4.5.3 Subscription

The entities that potentially could play the "NGSI-9 Client" role are any application that needs to be

aware of changes in context availability information. The "Subscribed App" could be the "NGSI-9 Client

itself".

Figure 24: IoT Discovery – Subscription

4.5.4 Notification

The entities that potentially could play the "NGSI-9 Client" role are IoT Agents, while "Subscribed App"

are applications subscribed to context availability information changed by the registration issued by the

"NGSI-10 Client" (we assume that such Subscribed Application are previously subscribed, as shown in

the sequence diagram in previous section).



Figure 25: IoT Discovery – Notification

4.6 Added-value (Optional) Features

The following sections describe additional features that IoT-Discovery may implement. However, they

are not mandatory for a basic implementation of this GE.

4.6.1 Geographical Discovery

As an optional feature, the IoT-Discovery GE supports the usage of geographic scopes in the operations

discoverContextAvailability and subscribeContextAvailability. In particular, the geographic scopes

defined under keyword SimpleGeoLocation in Appendix D of the OMA NGSI 9/10 specification [1] will be

supported.

For specifying the geographic location of entities and/or context providers, a new type of registration

metadata will be defined and implemented. The exact syntax and semantics of this metadata type is

under discussion.

4.6.2 Probabilistic Discovery

The Probabilistic Search Engine is a component that uses machine learning (ML) techniques for resolving

IoT Descriptions (whether plain XML or RDF). It uses unsupervised probabilistic ML techniques to group




similar IoT Descriptions. Its purpose is to allow automated discovery, ranking and recommendation of

IoT Descriptions. It provides an efficient mechanism for the search and discovery of registered IoT

Descriptions by automatically clustering all the IoT Descriptions and hence allowing the IoT Search

Engine to be scalable to large object repositories. It also automates the ranking of results in order of

relevance.

Figure 26: IoT Discovery – Probabilistic Search

4.6.2.1 Training the Engine

For the Probabilistic Search Engine to effectively refine the searching process, it goes through several

stages. First it involves a training process whereby all the Descriptions for each model is retrieved from

the repository. It will then extract "textual concepts" from the Descriptions using a parsing mechanism.

The concepts are essentially the elements, attributes and values of a Description. The extracted textual

concepts will then be used to create a "Document-term matrix", which is a mathematical matrix that is



used to reflect the frequency of textual concepts in a particular IoT Description, whereby the rows

represent the number of IoT Descriptions and the columns represent the numbers of concepts.

Figure 26: IoT Discovery – Concept Extraction

The next stage is another training process for clustering the IoT descriptions. The training involves a ML

technique known as the Latent Dirichlet Allocation. The aim of this technique is to discover a hidden

dimension behind the vector of concepts. This dimension essentially reflects a "topic" or "theme", which

is shared by certain IoT Descriptions. The topic itself is modelled as Latent Factors, which allows any type

of Descriptions to be represented in vector form, and hence making the search much more efficient

compared to keeping them in text format. The model then associates the Latent Factors with the

distribution of Concepts.

http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation



Figure 27: IoT Discovery – Latent Factor Extraction

Once the second training process is done, the clusters are then created. The number of clusters created

is based on the number of Latent Factors generated. The vector of Latent Factors that corresponds to an

IoT Description is used to determine which Latent Factor best describes it.



Figure 28: IoT Discovery – Folding in New Descriptions

After the training process, the Probabilistic Search Engine becomes online and is ready to receive

requests. During runtime, the repository is expected to receive more registrations of IoT Descriptions.

When this occurs the IoT Search Engine will read the new IoT Description and "fold" it in to one of the

clusters based in its probability of similarity with one of the Latent Factors in the vector.

4.6.2.2 Querying the Engine

For discovery, a user can query the IoT Search Engine by providing a description template or a SPARQL

query as the input to the search engine. The template should include concepts that are relevant to what

the user is searching for. The search engine will then convert the query into Latent Factor vector- as



done previously in the second training stage - and then match the query vector to the vector of Latent

Factors that represents the IoT Descriptions stored in the repository. A query example is shown below.

Figure 29: IoT Discovery – Query Match

A query about an Advanced Description is given in the form:

...whereby the query is asking for an Entity - in this case a room (38) - that has Humidity information

about it.

The Search Engine does as follows:



1. The query is converted to a vector of latent factors: {0.038, 0.916, 0.006, 0.006, 0.0, 0.006, 0.01,

0.008, 0.003, 0.0}

2. After analysing the vector, the query belongs to latent factor #2 with the highest probability.

3. Query is forwarded to the registry responsible for cluster #2 and query is matched within that

cluster.

4. The first five results obtained are:

Rank #1: 38_BA_01.owl with a score of 0.998479530076855





Once this is done, the respective IoT Descriptions are retrieved from the repository and returned to the

user.

4.6.3 Semantic Discovery

4.6.3.1 Overview

NGSI provides a simple, generic and effective means of exchanging information about context entities.

However, it does not specify a global, interoperable naming convention for them. Hence an entity type,

attribute, or attribute metadata can be syntactically declared differently from different context

producers, even though they might point to the same phenomenon. Or the case could be the other way

round, whereby the same syntax has different semantics among different context producers. Therefore

there is a need to enable the semantic interoperability of context information among context producers

and consumers, so that they are "talking" about the same thing. Here Linked Open Data can play a role

in increasing interoperability, whereby fields within an NGSI description can be linked to other well-

known descriptions that provide more information about it. Since this GE is designed for IoT entities,

then an IoT reference information model is needed to define the linking of an IoT related attribute

declared in NGSI to one that is.

This optional functionality within the IoT Discovery GE is targeted for:



Context Producers wanting to register their IoT entities using semantic IoT descriptions.

Context Consumers using SPARQL for discovering IoT Context Entities.

Applications requiring Semantic Discovery of IoT Context Entities.

Semantically-enabled GEs;

as a reference, semantic IoT Descriptions in this specification are based on the IoT-A ontology

models.

4.6.3.2 Linking NGSI Entities to Semantic Descriptions

Within the FIWARE architecture, NGSI-9 is the main protocol for the registration and discovery of

sources of context information. The NGSI-9 interface for discovering IoT entities is efficient due to its

simplicity and generality. But it is also regarded as a loose binding insofar as it does not provide a global

naming convention for entities and their attributes, which will essentially affect interoperability

between descriptions that might match in terms of syntax, and not in semantics. Therefore, to increase

interoperability, descriptions annotated using NGSI Context Information model should be exposed as

linked open data, so that its attributes can be compared with other entities on the web, whether it be

spatial, temporal or thematic. In order to do this, a mapping must be done between the information

annotated in NGSI to an semantic information model that reflects IoT entities and their properties. As an

initial reference, the ontology models defined by the EU FP7 IoT-A project can be used.

A Context Producer can link an NGSI Entity Description to an Advanced Description by adding a Context

Attribute that provides the URI of the corresponding Semantic Description, when registering the NGSI

Entity via the registerContextRequest operation. This requires the Semantic Description to be available

beforehand.

4.6.3.3 Associations

The Association Engine (AE) is a component that establishes and maintains associations between IoT

Entities and Services that expose Resources, which acts as the representative for interacting with a

Resource. The associations are made based on spatial, temporal and thematic relativity:

http://www.iot-a.eu/public

http://www.iot-a.eu/



Thematic (feature) match - using domain ontologies that capture an Entity's attributes and IoT

Service's input/output parameter.

Spatial match - using either co-ordinate comparison between IoT Entities and Services for

determining close proximity, or location ontologies that model logical locations with properties

such as 'contains'

Temporal match - utilising temporal aspects of entities which have a temporal aspect, such as

meeting rooms with the IoT Service's observation_schedule.

Figure 30: IoT Discovery – Association Process Overview



Creating the Associations

Formulation of associations utilises the IoT-A Entity and Service models (i.e. Advanced Descriptions).

Dynamic association inference is maintained through rules that incrementally reason on theme, spatial

aspects and time. At start-up, the AE will retrieve from the database all the different types of Advanced

Descriptions - Entities, and Services (N.B. Services represent Resources) - and processes them through

an Association Engine for matching. The matching process first looks for similar themes between Entity

Attributes and Service observation types, e.g. temperature, light intensity, acoustic level. It will then

look for similar locations. Lastly in the matching process it will look for temporal availability of a Service.

Once the matching process is done, the associations generated will then be stored, which can then be

queried via SPARQL.

Figure 31: IoT Discovery – Association Mechanism

Querying for Associations



The association can then be queried by users via the SPARQL interface. In the query, the predicate used

to retrieve the association is "isAssociatedToService". This will return the URI of the IoT-A service

description, which can then be retrieved by querying the IoT-A Service repository. It is this description

that will provide access to contextual information (e.g. temperature reading) produced by the IoT-A

Resource (e.g. temperature sensor).



5 FIWARE.ArchitectureDescription.IoT.Gateway.DataHandli

ng

5.1 Copyright

Copyright © 2012-2015 by Orange, Atos

5.2 Overview

The IoT world mainly consists of a huge amount of resources, creating a lot of events to be processed.

The Data Handling GE addresses the need of filtering, aggregating and merging real-time data from

different sources. A lot of applications expect to receive value-added data that are relevant to their

needs, and this can be easily achieved in a decoupled manner thanks to the Complex Event Processing

technology (CEP). This is also referred to as event stream analysis, or real time event correlation. Events

are published by Event Producers and subscribed to by Event Consumers. One can only subscribe to

events that are allowed by its granted access rights.

Typical applications that require Data Handling GE are sensor network applications, RFID readings,

supply chains, scheduling and control of fabrication lines, air traffic, smart buildings, home automation,

and so on. These applications have in common the requirement to process events (or messages) in real

time, or near real time. Key considerations for these types of applications are throughput, latency and

the complexity of the logic required. The CEP Engine component of the Data Handling GE typically

processes input data, in order to generate a smaller set of output data, which avoids upper-level

software overloading and network flooding.

The Data Handling GE handles data streams from IoT devices that cannot continuously be online, for

various reasons. Among them are possible network disruptions and power resource constraints. The

latter typically requires optimizing and limiting the communications with the IoT resources. For this

purpose, local storage is desirable, in order to cache the last available processed data and contexts.

The natural surrounding GEs for Data Handling are the IoT Broker, theIoT Discovery GE, and the Protocol

Adapter GE.

Although it can be operated standalone, the Data Handling GE typically receives events from the

Gateway Protocol Adapter GE, and propagates the processed data towards the IoT Broker GE. Prior to

propagating any data, the Data Handling GE must be registered to the IoT Discovery GE, in order to be

identified as a context producer.

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FT

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/ATOS



It should be noted that Publish-Subscribe and CEP features are also available in other GEs. This should

not be seen as useless redundancy: some features are deliberately distributed at different levels of

granularity, in order to provide reliability to the whole architecture, and also to appropriately dispatch

the load at different levels.

The Data Handling GE comes in two configurations: EspR4FastData and SOL/CEP. EspR4FastData targets

both mobile and fixed IoT gateways: it supports both local storage and data transformation. Despite its

gateway orientation, EspR4FastData could also be used server-side, whereas SOL/CEP is aimed at more

constrained environments, which lack both the CPU resources and memory size for storage, or data

transformation mechanisms. It targets small gateways, various devices and sensors that still have

enough CPU power to run it.

Both configurations are equipped with a CEP Engine component, but they operate in distinct ways. The

differences reflect the requirements for the underlying host hardware, on which the Data Handling GE is

deployed and executed.

The next sections discuss the overall architecture reflecting the different configuration options. Then

each component is described, if necessary in subsections according to the differences.

5.2.1 Internal architecture components diagram



Figure 32: IoT gateway Data Handling – Architecture

The interfaces for this GE are compliant with the FIWARE implementation of the Next Generation

Services Interface (NGSI), and more precisely, NGSI-9 and NGSI-10 subsections. It should be noted that

the official OMA NGSI Context Management standard neither specifies nor imposes any particular

implementation regarding the exchange format, but simply describes NGSI data types as belonging to

the "XML Schema Part 2" W3C recommendation. For the time being, FIWARE has chosen to support an

XML implementation of the NGSI standard: more information can be found here.


https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FI-WARE_NGSI_Open_RESTful_API_Specification_%28PRELIMINARY%29



5.2.2 Main interfaces

5.2.2.1 NGSI-9/10 API

The Data Handling API is NGSI-9/10 compliant. It accepts events from any NGSI compliant Event

Producer. In the IoT Service Enablement architecture, the Gateway Device Management GE publishes

device events and data towards the Data Handling GE.

In this setup, the Gateway Protocol Adapter GE registers as an NGSI context provider, by calling the

NGSI-9 registerContext method exposed by the Data Handling GE.

After context registration, the Gateway Protocol Adapter GE can send events by calling the NGSI-10

updateContext method of the Data Handling GE.

The Data Handling GE also registers as an NGSI context provider, by calling the NGSI-9 registerContext

method of the upper-level GE that acts as an NGSI registrar. In the context of the IoT Services

Enablement architecture, this GE is the IoT Backend Configuration Management. After registration, Data

Handling GE calls the NGSI-10 updateContext method of the IoT Broker GE, for each propagated event.

The IoT broker acts as an event consumer.

Third party software and other external GEs can subscribe to Data Handling GE output events by calling

its NGSI-10 subscribeContext methods. For every subscriber, events are propagated by calling their NGSI-

10 notifyContext method, which means subscribers must implement this method, in order for them to

receive events.

5.2.2.2 CEP Management API

This API allows the configuration and operation of the Complex Event Processor: it provides features

such as setting rules, defining event types, and configuring output event recipients.

In the standard configuration, it takes place as follows.

For more hardware-constrained configurations, it takes place as follows.

However, the meaning behind "standard" and "constrained" is left up to common sense and to the one

who is in charge of implementing a solution. There are no strong guidelines regarding how much a

system is embedded-oriented, hardware constrained, or server-oriented. For example, a PC-style

gateway might run EspR4FastData, whereas a high-end sensor might better run SOL/CEP.

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FIWARE.ArchitectureDescription.IoT.Gateway.DataHandling#EspR4FastData_implementation

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FIWARE.ArchitectureDescription.IoT.Gateway.DataHandling#SOL.2FCEP_Complex_Event_Processing_engine



It should be noted that the CEP management API methods should ideally be consistent, whether the

implementation concerns a constrained device, a mobile phone, or a powerful gateway.

5.2.2.3 Local Storage API

This API provides features to manage the local database at a low-level, and to query historical data.

This is not available in the SOL/CEP constrained configuration.

5.2.2.4 Data Transformation Management API

This API allows configuring transformers on both CEP input and output. Transformers define how to

update event properties and structure on the fly, before and after event processing.

5.2.3 Main components

5.2.3.1 NGSI-9/10 Publish-Subscribe component

This component is the core of the Data Handling GE NGSI functionality. It allows the Generic Enabler to

communicate with the outside world and the other GEs, in an ingoing and outgoing way, and also

features publish-subscribe functions. It implements the necessary subset of NGSI-9 and NGSI-10

methods for the purpose. It also allows the GE to register itself to the Context Broker GE, which is a

prerequisite to the propagation of events towards the IoT Broker GE.

This organization enables the creation of a Publish/Subscribe network of NGSI compliant GEs that

expose their services to each others.

It should be noted that NGSI compliance does not imply the support of the full set of NGSI methods.

5.2.3.2 CEP Engine component

This is the core functionality of the Data Handling GE.

The role of the CEP Engine component is to process events in real time, which mainly involves filtering,

aggregating and merging data. This processing is based on a set of rules that can be defined by the CEP

administrator.




The CEP Engine component can process XML data structures on its input. It is fed by the Data

Transformation component, which is in charge of XSLT event transformations before processing. Events

do not directly reach the CEP Engine component from outside.

Once processed, events pass through Data Transformation for optional output XSLT transformation

purpose, before being finally sent to their recipients, through the Publish-Subscribe component.

5.2.3.3 Data Transformation component

This component performs XSLT transformations on both input and output XML events.

The Data Handling GE supports XML events. It might also support other lower-level formats, when

configured for constrained devices with SOL/CEP.

When a NGSI-compliant event arrives, it first flows through the NGSI 9-10 Publish-Subscribe component

before being pushed towards Data Transformation and thus transformed or not, depending on

configured transformations. Other XML non-NGSI events come directly to the input of Data

Transformation.

In all cases, NGSI input events are finally translated into a simplified CEP-friendly XML format, that is

easily understood by the CEP Engine component, and easier to address with CEP rules scripting

language. Events are then pushed towards CEP Engine.

Data Transformation also receives processed events from CEP Engine output, for optional XSLT

transformation purpose, before the processed and transformed events are finally propagated towards

external configured recipients.

It should be noted that in the case of the SOL/CEP implementation for constrained environments, the

Data Transformation component is not available.

5.2.3.4 Local Storage component



This component stores output events from the Publish-Subscribe component and other XML output

events. For example, it could retain historical records of sensor readings. Compulsory metadata are

locally added to the actual stored data, such as source identification and local time stamps.

The amount of stored events can be configured, but of course depends on the available memory, which

may vary greatly among devices and gateways.

The Local Storage component provides methods to access and manage historical event data.

SOL/CEP implementation does not feature a Local Storage component.

5.2.3.5 Graphical rules editor

This optional component allows to manage CEP rules graphically, through the CEP Management API.

It allows graphical CEP rules building, by placing and biding blocks together. The resulting diagram is

then converted into EPL syntax, which is understood by EspR4FastData.

The Graphical rules editor is useful to manage CEP rules in a user-friendly environment. It is not available

for the SOL/CEP configuration.




Figure 33: IoT gateway Data Handling – Graphical Rules Editor

5.3 Basic Concepts

5.3.1 Event concepts

Similar to many topics in science and engineering, the term "event" has different meanings based on

who is observing the event and the context of the observation. Generally speaking, an event is

something notable that happens in the real world or in a system. But in the computer world, an event is

understood as a computer representation of such an occurrence, usually for the purpose of computer

processing. Events can contain data, are immutable, but more than one event may record the same

activity.

Before the actual event processing occurs, optional event pre-processing can be implemented: it is often

referred to as data normalization, validation, and pre-filtering and basic feature extraction. When

required, event pre-processing is often the first step in a distributed, heterogeneous CEP solution.

Heterogeneous, distributed event processing applications normally require some type of event pre-

processing for data normalization, validation and transformation.



An "event object" is a computer representation of an actual event in the real world or in any system. An

event object carries data or properties. But the term “event” is actually more often used than “event

object”, as a simpler, confusing, but yet improper replacement. However, we can consider the terms

“event” and “event object” to be somewhat equivalent when talking about CEP topics.

5.3.2 Complex Event Processing concepts

Complex Event Processing (CEP) is computing that performs operations on events. Common event

processing operations include reading, creating, transforming, and deleting events (ETZION & NIBLETT,

2011). It can also be considered as a method of analyzing real-time streams of data about things that

have happened and deriving a conclusion from them (LUCKHAM, 2011). It typically processes input

events order to produce a smaller set of output events.

It is also a distributed technology that creates actionable, situational knowledge from distributed

message-based systems, databases and applications in real time or near real time. It can provide an

organization with the capability to define, manage and sometimes predict events, situations,

exceptional conditions, opportunities and threats in complex, heterogeneous networks. (BASS, 2007)

CEP processes many events happening across all the layers of an organization or system, identifying the

most meaningful events within the event cloud, analyzing their impact, and taking subsequent action in

real time.

CEP also refers to state processing, which is the change of state occurring when a defined threshold is

exceeded. It requires event monitoring, event reporting, event recording, and event filtering. Any event

may be observed as a change of state with any physical or logical, or otherwise discriminated condition,

of and in a technical or economical system. Each state change usually comes with an attached time

stamp defining the order of occurrence, and a topology mark that can define the location of the

occurrence.

5.3.3 Choosing the right implementation for you

Regarding the Data Handling GE, the choice among the two implementations of the Complex Event

Processor highly depends on the constraint level of the hardware that hosts and runs the enabler.

For very constrained environment, SOL/CEP is the right choice, whereas for mid-end to high-end

gateways, EspR4FastData is appropriate.


https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FIWARE.ArchitectureDescription.IoT.Gateway.DataHandling#EspR4FastData_implementation



5.3.4 EspR4FastData implementation

EspR4FastData highly relies on the Esper (by EsperTech) java library. The latter is available for both Java

and .Net. It features a dedicated CEP rule (EPL) language. The Java version has been chosen for the

FIWARE project.

It is an open-source software under the GNU General Public License GPL.

Esper (by EsperTech) CEP library is mainly used for:

Complex computations: applications that detect event correlation, filter events, merge event

streams, trigger specific actions when particular conditions are satisfied, etc...

High throughput: applications that process large volumes of messages (between 1,000 to 100k

messages per second)

Low latency: applications that react in real-time to conditions that occur

Native supported event formats are plain Java objects, XML, and map objects. XML is the one

implemented in FIWARE.

Esper (by EsperTech) provides an dedicated Event Processing Language (EPL) which allows rules

definition. EPL is an SQL-like language, but is specifically optimized for dealing with high frequency event

data.

The EPL language allows grouping, aggregating, sorting, filtering, merging, splitting and duplicating event

streams, defining time and length sliding windows, matching event pattern, and so on...

Some important concepts drive the CEP Engine component structure:

The event types describe the internal structure of incoming events

The rules define the way event are processed. Rule triggering occurs when event data match the

rule criteria.

Sliding windows store event data in a First-In First-Out (FIFO) manner. They can be sized in

length or duration.

A listener is executed when a rule is triggered. A listener is always associated to a rule.

5.3.4.1 EPL statement structure

A typical statement looks like the following. The brackets indicate optional features.

http://esper.codehaus.org/



[INSERT into insert_into_def] SELECT select_list FROM stream_def [AS name] [, stream_def [AS name]]

[,...] [WHERE search_conditions] [GROUP BY grouping_expression_list] [HAVING

grouping_search_conditions] [OUTPUT output_specification] [ORDER BY order_by_expression_list]

5.3.4.2 Examples of rules statement:

Simple fields "select" from a single event source:

Merge event sources:

5.3.5 SOL/CEP implementation



SOL/CEP is a Complex Event Processor characterised by scalability and performance. It accepts complex

events that are described in the domain specific Dolce Language

With Dolce it is possible to specify what individual events need to be detected, and how and which of

these events are composed into Complex Events.

Figure 34: IoT gateway Data Handling – High-level Architecture

The CEP Engine reads events from the Event Collector, analyses them in the Complex Event Detector and

finally emits them through the Complex Event Generator. Events are accepted from a variety of sources

and protocols (Database, file system, TCP/IP), translated to an internal format by means of adapters and

processed. Likewise, Complex Events can be published towards different channels in a variety of

formats.

For the FIWARE Gateway Data Handling GE it will interact using NGSI.

5.3.5.1 Characteristics

SOL/CEP is designed for the following characteristics:

https://forge.fi-ware.eu/docman/view.php/11/1187/Dolce-Language-Spec-Release-v1.pdf



Sliding Time Window

This applies a time window over the evaluation of the complex event.

Example: all transactions during the last 10 hours. The “last 10 hours” is called the sliding time window

and always refers to the events that happened in this time frame.

Correlation between Event attributes and Complex Events

This means that the values of the attributes that are specified in the Event can be reused and evaluated

in the Complex Event.

Example: the customer name from an incoming event can be copied to the Complex Event.

Complex Event Functions

Aggregation allows totalizing the values of the incoming events, Averaging calculates the average of all

values of a time windows.

Example: the sum of all the money that was withdrawn in the last 5 days.

Example: The average temperature during the last 20 minutes.

Temporal Awareness

A complex event is a combination of simple, individual events. The temporal order in which these events

should happen for the complex event to be raised can be specified using the relative “after” or “during”

keywords.

Example: WarningLightsOff before LandingGearOut LandingGearUp during Approach

Evaluation of attributes

Attributes in the Complex Event can be evaluated using a free formula.

Example: Average(temperature) >= 20

Event Filtering

Incoming Events can also be filtered based on a formula.

Example: TransactionAmount > 1000

5.3.5.2 Technical aspects



The CEP runs as a process and is language agnostic. Accepts input and generates output through Format

Adapters (explained before)

Application-level language binding is C++.

Low footprint: designed for performance and small resource usage, but scalable to high end

servers.

Operating system support: BSD, Linux (Debian).

5.3.5.3 Dolce Language

In Dolce, a user can specify Events that need to be detected as follows:

This tells the CEP to detect RainfallMeasurement events, with the attributes amount and sensorId. The

event is only accepted for Sensor number 5. Other sensors are ignored. The amount represents the mm

of rain that fell since the last measurement.

NOTE: the Format Adapter needs to map the event type and assign a number to the different incoming

events. This is a convention that is decided at design time by the developer – in this case, the number 5

was chosen to identify RainfallMeasurement events.

A Complex Event could be specified as follows:



This tells the CEP to raise a complex event when the total amount of rain exceeds 100 mm in 10

minutes. The event will be of type 1600 and contains a field called “Total” which is an integer and

contains the amount of rain that fell.

5.3.6 NGSI

[1] Ref. NGSI Context Management - Open Mobile Alliance V1.0 May 29, 2012

NGSI 9/10 information model for FIWARE

FIWARE NGSI Open RESTful API Specification (PRELIMINARY)

5.3.7 ETSI M2M

[2] ETSI M2M communications


The NGSI 9-10 Publish-Subscribe component registers the Data Handling GE to the Configuration

Management GE by calling its registerContext method.

http://technical.openmobilealliance.org/Technical/release_program/docs/CopyrightClick.aspx?pck=NGSI&file=V1_0-20120529-A/OMA-TS-NGSI_Context_Management-V1_0-20120529-A.pdf



http://forge.fi-ware.eu/plugins/mediawiki/wiki/fiware/index.php/Internet_of_Things_%28IoT%29_Services_Enablement_Architecture#ETSI_Machine-to-Machine_communications



The NGSI 9-10 Publish-Subscribe component sends events to the IoT Broker by calling its

updateContext NGSI method.

When needed, the IoT broker can call the queryContext method to synchronously retrieve

context data from the Data Handling NGSI Publish-Subscribe component.

The Protocol Adapter GE registers itself to the Data Handling GE by calling its registerContext

method.

The Protocol Adapter GE sends events towards the Data Handling GE by calling its

updateContext method.

Figure 35: IoT gateway Data Handling – Main Interactions




The Complex Event Processing API was designed with the main CEP concepts in mind: CEP

component engine state, CEP rules management, event type management, and configuration of

the actions that are executed on rule triggering.

The API was designed in a resource-oriented manner.

NGSI interface is also part of the API and it was designed in a restful manner. The Data Handling

GE has to keep technical consistency on its whole API.

5.6 Re-utilised Technologies/Specifications

The Gateway Data Handling GE is based on RESTful Design Principles. The technologies and

specifications used in this GE are:

RESTful web services

HTTP/1.1 (RFC2616)

XML data serialization format.

Other technology which is re-used

Esper (By EsperTech)

http://www.ietf.org/rfc/rfc2616.txt

http://esper.codehaus.org/



6 IoT Gateway – Protocol Adapter

6.1 Copyright

Copyright © 2014-2015 by Orange, Ericsson, Telecom Italia

6.2 Overview

The Protocol Adapter GE deals with the incoming and outgoing traffic and messages between the IoT

Gateway and registered devices, to be served by either the Gateway Device Management GE or the Data

Handling GE. There may be multiple instances of Protocol Adapter GEs capable of serving partially IoT-

compliant devices, i.e. devices that do not support ETSI M2M (the specifications may be found at the

following link ETSI M2M Latest Drafts). These devices can be IP-based devices, that communicates using

the IP stack (IPv4 or IPv6), or "legacy devices", meaning devices communicating using non-IP based

protocols, for instance ZigBee, or Z-Wave.

The Protocol Adapter GE receives these device-specific protocols and translates them to a uniform

internal API. The exposed API handles capabilities to read and write to the resources, as well as IoT

specific management and configuration services such as resource discovery consisting of both look-up

and publication.

In particular, the ZigBee Protocol Adapter provides a communication conduit into a ZigBee PAN(s)

(Personal Area Network). It supports a mechanism whereby a gateway can interact with individual

ZigBee nodes to exert control over or to obtain data from those nodes, or conversely a mechanism

whereby the nodes can communicate some information to the gateway.

6.3 Basic Concepts

An overview of the Protocol Adapter GE is provided below, and is followed by an identification of the

interfaces.

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FT

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/EAB

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/TI

http://docbox.etsi.org/M2M/Open/Latest_Drafts/



Figure 36: IoT Gateway Protocol Adapter – Architecture

There are three interfaces to the Protocol Adapter GE. The southbound APIs provide the gateway

external interface to non-ETSI M2M devices hosting sensor and actuator resources. The currently

supported interfaces are IETF CoRE (provided by Ericsson and no more supported) and ZigBee (the

specifications may be found at the following link ZigBee Network Devices Standard Overview).

On the northbound side there are two interfaces:

the first interface is the Protocol Adapter Interface that is a communications protocol to the

Gateway Device Management GE, based on the Generic Device Access API. This interface can be

http://www.zigbee.org/Standards/ZigBeeNetworkDevices/Overview.aspx



used for initiating subscriptions to resources and receiving notifications from resources that

have been tasked with subscriptions, read or write resources that are determined to be online,

publishing resource capabilities in the Resource Directory, or querying devices for their

resources.

the second one is the NGSI compliant Gateway Data Handling API that allows to interact with

the Gateway Data Handling GE. Through this interface, device events and data are published

towards the Gateway Data Handling GE.

Figure 2 introduces the architecture of the FIWARE Protocol Adapter GE.



Figure 37: IoT Gateway Protocol Adapter – Internal Architecture



6.3.1 Base Driver

The Base Driver is the low-level API for legacy devices (i.e. an implementation of the device specific

protocol stack). Base Drivers handle device discovery and access to sensor and actuator resources in a

protocol-specific way.

For instance, the ZigBee Base Driver is based on the Network Device Gateway Specification defined by

the ZigBee Alliance (ZigBee document 075468r35 may be found at this link ZigBee Network Devices

Standard Overview). The included operations in this specification are:

operations to read and write attributes, and configure and report events;

macro operations for network and service discovery;

endpoint management;

flexible start-up and network join operations;

bi-directional communication mechanisms between ZigBee Base Driver and gateway

6.3.2 Protocol Adaptation

A Protocol Adapter is the glue between a base driver and the Generic Device Access API or the Gateway

Data Handling API. Via the base driver, it discovers devices, tracks events occurred on them and

executes commands to actuate them. Therefore, a Protocol Adapter is necessary for each protocol that

the Base Drivers support. Whether the protocol is standardized or proprietary does not matter. Provided

that the Base Driver is available and the Protocol Adapter is implemented on top, the Generic Device

Access API or the Gateway Data Handling API is able to support the protocol and provide a unified way

to access devices with the protocol. The Protocol Adapter is able to support:

Device discovery. When a new device is discovered and gets available, it shall create and register

this device as a service within the Protocol Adapter framework. When a previously discovered

device gets unavailable, it shall unregister the corresponding service.

Device measurement update. When a service parameter update occurs on a device, it shall

update the corresponding variable on the device in the Protocol Adapter framework and trigger

update for the corresponding device service.

Device actuation. For each action in a service that a device supports, it should implement a

protocol-specific logic and put it as an action in the device service that is registered in the

Protocol Adapter framework.





Device event and data. Its behavior is like an Event Producer and for each device connected it

shall send events and data

6.3.3 Generic Device Access API

The Generic Device Access API (GDA) exposes a high-level, protocol agnostic API towards the Gateway

Data Handling and Gateway Device Management. GDA uses service schemas which are XML-files that

describe the supported resources, i.e. the application profiles. This schema based approach makes it

possible to cover a wide range of applications spanning from home automation, to media, to health

care.

The GDA defines two main data structures:

Device: it represents the sensor/actuator,

Service: it represents a set of functionalities provided by the Device.

The GDA main methods, which have to be used by the Gateway Data Handling and Gateway Device

Management are:

device.getService(<name of service>) – used to get the services associated to a specific device,

service.getProperties() – used to get the list of properties (i.e. attributes) implemented by a

specific service implementation,

service.getAction(<name of action>) – used to get a single action (i.e. command) implemented

by a specific service implementation.

Gateway Data Handling and Gateway Device Management can get sensor data or configure a device by

reading/writing a Service property, and can command an actuator by calling a Service action.


Figure 3 shows an example of device and resource discovery, and how to subscribe to resources using

the Protocol Adapter GE. The Device Management GE and Protocol Adapter register listeners for new

devices with the gateway framework (OSGi service bus). When the basedriver finds a new device in the



network it will register this device in the framework which in turn notifies the listening Protocol Adapter.

The Protocol Adapter can now query the device for resources which are then mapped to a service

schema. These resources are then made available to the Generic Device Management GE.

Subscriptions are also registered in the framework which then triggers the Protocol Adapter to start a

subscription to the requested resource. A new update (e.g. change in sensor value) will send an update

to the Generic Device Management GE.

Figure 38: IoT Gateway Protocol Adapter – Device discovery and subscribe

Moreover the Protocol Adapter GE is an NGSI compliant Event Producer towards the Data Handling GE.

In the IoT Service Enablement architecture, the Protocol Adapter GE publishes device events and data

towards the Data Handling GE. In this setup, the Protocol Adapter GE registers itself as an NGSI Context

Provider, by calling the NGSI-9 registerContext method exposed by the Data Handling GE. After this

context registration, the Protocol Adapter GE sends events, related to the connected devices, by calling

the NGSI-10 updateContext method of the Data Handling GE.




Projects deciding to support additional protocols should make sure that their implementations provide

the following functionality:

Device discovery

Device measurement update

Device actuation

Device events and data support



7 IoT Gateway Device Management

This section refers to the previous version of this GE and thus obsolete. It will be replaced soon with

the new functionalities including the Gateway Logic, currently under definition.

At that time it will be included as well in the IoT Architecture deliverable.



8 Glossary

GW Gateway. A hub device that intermediates IoT devices communications among

themselves and an IoT Cloud and/or IoT Application.

GW2GW Gateway to Gateway. Refers to Gateway to Gateway communications/APIs.

IoT Internet of the Things.

MQTT MQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol.

http://mqtt.org

NGSI9/10 OMA Context Management Protocol.

http://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/NGSI-9/NGSI-

10_information_model

UL2.0 Ultralight2.0. A lightweight version of the SensorML protocol.

https://forge.fiware.org/docman/admin/index.php?editdoc=1&docid=5084&group_id=

11

http://mqtt.org/



https://forge.fiware.org/docman/admin/index.php?editdoc=1&docid=5084&group_id

Documents

Private Public Partnership Project (PPP) - FIWARE · Private Public Partnership Project (PPP) Large-scale Integrated Project (IP) D.14.1.1: Contribution to FIWARE Reference Architecture