When IoT Meets Virtual

3D World

Liane Tarouco

Universidade Federal do Rio Grande do Sul

Brazil

Team

� Liane Tarouco - PhD professor - chairman

� Lisandro Granville – PhD professor

� Leandro Bertholdo – Network engineer

� Cesar Loureiro – Software engineer

� Leandro Tibola – PhD student

� Fabiana Sgobbi – PhD student

� Fabricio Harpich – PhD student

� Lucas Arbiza – MsC student

IoT and Virtual World

� Internet of Things are becoming ubiquitous and able to provide data to allow better contextualization in virtual 3D world

� This presentation report alternatives for integrating IoT and VW

� Using special designed midleware

� Using derailleur based on SDN

� Using direct access to web server

� Using the silo approach provided by default in

IoT devices available in the market

Gardner – Hype Cycle 2015

Realism and engagement

� Realistic environment promotes better engagement

� Projects

� AVATAR – Ambiente Virtual de Aprendizagem e

Trabalho Acadêmico Remoto

� HIGIA - Habitat Individual e Guia Interativa de Atitudes

Virtual environment

� AVATAR - Effectiveness of digital laboratories as a learning resource.

� HIGIA - Virtual 3D world to improve motivation and self determination

Communication in and outside

virtual worlds

� Communication with other objects in virtual and/or physical worlds.

Virtual World being used

� Servidor Open Simulator

� Standalone

� Server / grid

� Clients

� Firestorm

� Singularity

Authoring in VW

1. Visual blocks

� ScripTastic

� Scratch for Open Sim

� Flash Scratch to Linden Scripting Language

2. LSL & OSSL

• Notes

• Notepad++

• LSL Editor

Multmedia resources in VW

� Presenting information� Slide presentation

� Videos

� Animations

� Web pages

LSL comands for

interopreration

� llHTTPRequest� Send data from Open Sim to na URL address

� llOpenRemoteDataChannel();

� llOwnerSay("Channel : " + (string) channel + "");

� llRemoteDataReply(channel,NULL_KEY,"OK",1);

OpenSim sending SMS

12

LSL code for sending SMSkey myRequest;integer CHANNEL = 0;default{touch_start(integer number){

llSay(CHANNEL,“Type your phone number with area code 555191494040:");llListen(CHANNEL, "", NULL_KEY, "");

}listen(integer channel, string name, key id,string message){llSay(CHANNEL,“Sendin name: " +name + "=> test message sent to :" +message);

myRequest = llHTTPRequest("http://www.ieduca.com.br/enviasms.php",[HTTP_METHOD, "POST",HTTP_MIMETYPE, "application/x-www-form-urlencoded"],"name="+name+"&celular="+message);

}http_response(key request_id, integer status, list metadata, string body){

if (request_id == myRequest){

llSay(0,body);}

}}

Exchanging data with external

world� Virtual worlds can exchange data with learning

management systems, enterprise resource planning, legacy systems, databases and Web systems, including chatterbot.

� These exchanges allow information to be sent and received regarding simulations and the its users

� User data can be collected to monitor user performance in learning activities

Moodle

PHP/My SQL

Chatterbot

Virtual world Real World

� Applications to communicate with a virtual world� an HTTP request to a Web service, where a prim

sends a request to the Web service, which processes it and sends back an answer to OpenSim,

� an HTTP script call that runs a remote script program to manipulate data in a database

� XML-RPC: enable two-way communication between the virtual and physical world through remote procedure calls and XML data transfer

Interoperation problems with IoT

� Interoperation problems derived from limited communication capabilities and low computational power of IoT devices. � Most of smart devices, used in the context of Internet

of Things is not using generic standards interfaces.

� One of the challenges in this context derives from the need to integrate the various devices and protocols making it possible to obtain information about the scenario being monitored

REMOA project

17

� Environment Monitoring Network - project

targeting home solutions for homecare or/and

tele monitoring of patients with chronic

illnesses.

� Health in family public program

Initial approach

18

ArchitectureData Collection

Agents

Middleware

Middleware

� Wireless Access Point (AP) that supports OpenWRT or DD-WRT.

� Enables the development and deployment of additional software, and supports protocols

� IPv4, IPv6, NAT

� SNMP proxy

� Gateway to other devices19

Middleware issues perceived

� Development of services features limited by APs constraints

� Complex deployment and maintenance

� Complex mechanism to manage smart devices

20

Refactoring

� A solution to circumvent this problem may derives from using a SDN based approach, which turn an access point in a flow switch.

� This solution offers a possibility to divert traffic to a remote server in order to handle, transform and redirect data as required to use it as input in virtual 3D world.

SDN-based middleware refactoring

22

Architecture overview

23

VW integration with external

objects

� Internet of Things� Sensors

� Monitors

� WIFI Camera with movement Sensor - WIFI

� Pedometer - Bluetooth

IoT & Virtual Word

� Camera captured images may bepresented as web pages used as texture in 3D Objects

Internet of Things

� Pedometer data recovered from FitBitserver and presented in VW

Conection with external system

link_message(integer sender_num, integer num, string msg,

key id)

{requestid=llHTTPRequest(“http://www.anyexternalsystem.com

/talk-

xml?id=”+id+&input=”+llEscapeURL(msg)+”&custid=”+cust,

[HTTP_METHOD ,"POST"], ””); }

http_response(key request_id, integer status, list metadata,

string body)

{ if (request_id == requestid)

if (msg == “/create”){

listen_to (id); }

HIGIA - academy

Prof. PhD. Liane Margarida Rockenbach Tarouco

liane@penta.ufrgs.br

Federal Federal Federal Federal UniversityUniversityUniversityUniversity ofofofof Rio Grande do Rio Grande do Rio Grande do Rio Grande do SulSulSulSul

Porto Alegre, Brazil